Add calc_fdr_hurdle, the Harvey and Liu (2020) false discovery hurdle - #321
Open
ipezygj wants to merge 1 commit into
Open
Add calc_fdr_hurdle, the Harvey and Liu (2020) false discovery hurdle#321ipezygj wants to merge 1 commit into
ipezygj wants to merge 1 commit into
Conversation
There is no universal t > 3. The bar a candidate must clear depends on how many candidates were tried and how correlated they were, so it has to be derived from the search rather than read off a table. calc_fdr_hurdle takes the return series of every trial that was evaluated, one per column, and returns the |t| a trial must reach at a chosen false discovery rate. The null is built by demeaning each column and resampling the time index, drawing the same periods for every column so cross-trial correlation survives into the null. Every trial is treated as null when counting expected false discoveries, conservative in the same way Benjamini-Hochberg is with m0 = m. The hurdle is the smallest one whose whole tail still meets the target, so a dip in a noisy curve cannot pass. Columns with no dispersion are dropped and the docstring says so, because a constant column has no t-statistic but floating-point residue gives it an enormous finite one; left in, it would be the strongest trial in the search. That is the same failure this repo fixed for the deflated Sharpe ratio in pmorissettegh-317, so the guard is here from the start rather than as a follow-up, and its test sweeps scales 1e-12 to 1e3 across lengths 50 to 1000 instead of resting on the single point the floor was calibrated on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
calc_fdr_hurdle: the t-statistic hurdle a strategy search implies at a chosen false discovery rate, following Harvey and Liu, False (and Missed) Discoveries in Financial Economics, Journal of Finance 75(5), 2503-2553.It sits next to
calc_deflated_sharpe_ratioand answers the neighbouring question. The deflated Sharpe ratio asks whether the winner survives its own selection. This asks what bar the search sets in the first place, and it needs the whole trial panel rather than a summary: one column per trial that was evaluated.There is no universal t > 3
The bar depends on how many candidates were tried and on how correlated they were, so it is derived from the search rather than read off a table. On a panel of 60 trials with no skill whatsoever (500 daily observations, seeded):
The best of those 60 trials reaches |t| = 2.18, which reads as significant on its own and is nothing of the kind: the search that produced it sets a bar of 2.20, and nothing clears it.
Plant three genuinely skilled trials in the same panel and widen the search around them:
The same three strategies, the same data, a rising bar. That is the whole content of the method, and it is why the trials that were discarded belong in the panel.
How the null is built
Each column is demeaned and the time index is resampled, drawing the same periods for every column so that cross-trial correlation survives into the null rather than being assumed away. Every trial is treated as null when counting expected false discoveries, which is conservative in the same way Benjamini-Hochberg is with
m0 = m. The hurdle returned is the smallest one for which the target also holds at every stricter hurdle, so it cannot land on a dip in a noisy curve.Because the alternative is never represented, the function bounds false discoveries only and says nothing about missed ones. The docstring states that rather than leaving it to be inferred.
Zero-dispersion guard, from the start
A constant column has no t-statistic, but it does not arrive as a
nan: the standard deviation of a constant series is floating-point residue rather than an exact zero, so a flat column divides out to something enormous but finite and would be counted as the strongest trial in the search. Such columns are dropped, and the docstring says so, since dropping changes the trial count that sets the hurdle.This is the same failure mode this repo fixed for the deflated Sharpe ratio in #317, so the guard ships with the feature instead of arriving as a follow-up. Its test sweeps scales 1e-12 to 1e3 across lengths 50 to 1000, rather than resting on the single point the floor was calibrated on — a genuinely low-volatility panel still gets a number, only the degenerate one does not.
Verification
ruff checkandruff formatintroduce nothing new: verified by running both againstmasterand diffing the findings, since the repo carries some pre-existing ones.