Skip to content

Add calc_fdr_hurdle, the Harvey and Liu (2020) false discovery hurdle - #321

Open
ipezygj wants to merge 1 commit into
pmorissette:masterfrom
ipezygj:add-fdr-hurdle
Open

Add calc_fdr_hurdle, the Harvey and Liu (2020) false discovery hurdle#321
ipezygj wants to merge 1 commit into
pmorissette:masterfrom
ipezygj:add-fdr-hurdle

Conversation

@ipezygj

@ipezygj ipezygj commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Adds calc_fdr_hurdle: the t-statistic hurdle a strategy search implies at a chosen false discovery rate, following Harvey and Liu, False (and Missed) Discoveries in Financial Economics, Journal of Finance 75(5), 2503-2553.

It sits next to calc_deflated_sharpe_ratio and answers the neighbouring question. The deflated Sharpe ratio asks whether the winner survives its own selection. This asks what bar the search sets in the first place, and it needs the whole trial panel rather than a summary: one column per trial that was evaluated.

There is no universal t > 3

The bar depends on how many candidates were tried and on how correlated they were, so it is derived from the search rather than read off a table. On a panel of 60 trials with no skill whatsoever (500 daily observations, seeded):

hurdle = trials.calc_fdr_hurdle()          # 2.20
trials.apply(lambda c: c.mean() / (c.std() / len(c) ** 0.5)).abs().max()   # 2.18

The best of those 60 trials reaches |t| = 2.18, which reads as significant on its own and is nothing of the kind: the search that produced it sets a bar of 2.20, and nothing clears it.

Plant three genuinely skilled trials in the same panel and widen the search around them:

trials evaluated hurdle discoveries
5 2.20 3
20 2.60 4
60 2.95 3

The same three strategies, the same data, a rising bar. That is the whole content of the method, and it is why the trials that were discarded belong in the panel.

How the null is built

Each column is demeaned and the time index is resampled, drawing the same periods for every column so that cross-trial correlation survives into the null rather than being assumed away. Every trial is treated as null when counting expected false discoveries, which is conservative in the same way Benjamini-Hochberg is with m0 = m. The hurdle returned is the smallest one for which the target also holds at every stricter hurdle, so it cannot land on a dip in a noisy curve.

Because the alternative is never represented, the function bounds false discoveries only and says nothing about missed ones. The docstring states that rather than leaving it to be inferred.

Zero-dispersion guard, from the start

A constant column has no t-statistic, but it does not arrive as a nan: the standard deviation of a constant series is floating-point residue rather than an exact zero, so a flat column divides out to something enormous but finite and would be counted as the strongest trial in the search. Such columns are dropped, and the docstring says so, since dropping changes the trial count that sets the hurdle.

This is the same failure mode this repo fixed for the deflated Sharpe ratio in #317, so the guard ships with the feature instead of arriving as a follow-up. Its test sweeps scales 1e-12 to 1e3 across lengths 50 to 1000, rather than resting on the single point the floor was calibrated on — a genuinely low-volatility panel still gets a number, only the degenerate one does not.

Verification

  • Agreement with an independent implementation of the same method to the grid resolution on both a null panel and one with planted signal (2.20 vs 2.20, 2.85 vs 2.85).
  • Seeded, so the hurdle is reproducible; equality across repeated calls asserted in the tests.
  • 79/79 tests pass (5 new). ruff check and ruff format introduce nothing new: verified by running both against master and diffing the findings, since the repo carries some pre-existing ones.
  • No new dependency; numpy and pandas only.

There is no universal t > 3. The bar a candidate must clear depends on how
many candidates were tried and how correlated they were, so it has to be
derived from the search rather than read off a table.

calc_fdr_hurdle takes the return series of every trial that was evaluated,
one per column, and returns the |t| a trial must reach at a chosen false
discovery rate. The null is built by demeaning each column and resampling
the time index, drawing the same periods for every column so cross-trial
correlation survives into the null. Every trial is treated as null when
counting expected false discoveries, conservative in the same way
Benjamini-Hochberg is with m0 = m. The hurdle is the smallest one whose
whole tail still meets the target, so a dip in a noisy curve cannot pass.

Columns with no dispersion are dropped and the docstring says so, because
a constant column has no t-statistic but floating-point residue gives it an
enormous finite one; left in, it would be the strongest trial in the search.
That is the same failure this repo fixed for the deflated Sharpe ratio in
pmorissettegh-317, so the guard is here from the start rather than as a follow-up, and
its test sweeps scales 1e-12 to 1e3 across lengths 50 to 1000 instead of
resting on the single point the floor was calibrated on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant