honest-backtest

Honest-inference statistics for backtests. A small, dependency-light toolkit for the questions that separate a real edge from a data-snooped one:

• Is this Sharpe real, or the best of N noisy trials? — Deflated Sharpe Ratio (DSR), Probabilistic Sharpe Ratio (PSR), expected-maximum-Sharpe haircut.
• How likely is my selection procedure to be overfit? — Probability of Backtest Overfitting (PBO) via Combinatorially-Symmetric Cross-Validation (CSCV).
• Is forecast A actually better than forecast B? — the Diebold-Mariano test (HAC-corrected), with QLIKE for volatility forecasts.
• Do my many tests survive multiple-comparison control under dependence? — Romano-Wolf stepdown (FWER), White's Reality Check (data-snooping), Benjamini-Hochberg (FDR).
• Is my run reproducible, and did the upstream data change under me? — a run-provenance manifest (git commit, environment, config hash, seed) and deterministic DataFrame fingerprints with a data-drift gate.

Pure numpy / scipy / pandas, no other dependencies. Extracted and generalized from a private quant-research program whose main empirical result was a stack of disciplined negatives — these are the gates that made those negatives trustworthy.

Install

pip install -e .

Examples

Deflated Sharpe + PBO — is the best of 20 strategies real?

import numpy as np
from honest_backtest import deflated_sharpe_ratio, pbo

rng = np.random.default_rng(0)
M = rng.normal(0, 0.01, size=(500, 20))          # 20 pure-noise strategies, 500 periods
best = M[:, M.mean(0).argmax()]                  # the "winner" you'd be tempted to trade
sr_var = float((M.mean(0) / M.std(0, ddof=1)).var(ddof=1))
print("DSR:", round(deflated_sharpe_ratio(best, n_trials=20, sr_variance=sr_var)["dsr"], 3))  # ~low
print("PBO:", round(pbo(M)["pbo"], 3))           # ~0.5 -> no real skill

Diebold-Mariano — is model A's forecast better?

import numpy as np
from honest_backtest import diebold_mariano

rng = np.random.default_rng(1)
loss_a = np.abs(rng.normal(0.5, 0.2, 400))       # smaller loss
loss_b = np.abs(rng.normal(0.8, 0.2, 400))       # larger loss
print("p(A better):", round(diebold_mariano(loss_a, loss_b, h=1)["p_value_a_better"], 4))

Romano-Wolf — which of K statistics survive FWER control under dependence?

import numpy as np
from honest_backtest import romano_wolf

rng = np.random.default_rng(2)
K = 10
stats = rng.normal(0, 1, K); stats[0] = 3.5      # one genuinely large statistic
boot = rng.normal(0, 1, size=(2000, K))          # centred bootstrap draws under the null
print("reject:", romano_wolf(stats, boot, alpha=0.05)["reject"])

End to end — backtest → walk-forward → deflate: the package's two halves compose, from raw weights to an honesty verdict.

import numpy as np, pandas as pd
from honest_backtest import run_backtest, run_walk_forward, summary, deflated_sharpe_ratio

rng = np.random.default_rng(0)
idx = pd.date_range("2020-01-01", periods=400, freq="W")
returns = pd.DataFrame({"a": rng.normal(0.001, 0.02, 400), "b": rng.normal(0.001, 0.02, 400)}, index=idx)

# a trivial "always long a" strategy: target weights decided at t, held over t+1 (no look-ahead)
weights = pd.DataFrame({"a": 1.0, "b": 0.0}, index=idx)
res = run_backtest(weights, returns, cost_bps=5.0, freq="W")
print(summary(res)["sharpe"])                       # full-sample metrics

# walk-forward OOS: refit weights on each train window, stitch the out-of-sample returns
def make_weights(train_returns, test_index):
    best = train_returns.mean().idxmax()            # pick the better asset in-sample
    return pd.DataFrame({c: (1.0 if c == best else 0.0) for c in train_returns.columns}, index=test_index)

splits, results = run_walk_forward(              # returns (folds, per-fold BacktestResults)
    make_weights, returns, train=104, test=26, mode="expanding", cost_bps=5.0, freq="W"
)
oos_returns = pd.concat([r.returns for r in results])   # stitch the out-of-sample returns
dsr = deflated_sharpe_ratio(oos_returns.to_numpy(), n_trials=2, sr_variance=0.01)
print("OOS DSR:", round(dsr["dsr"], 3))             # is the walk-forward Sharpe real after deflation?

Reproducibility — fingerprint the inputs, catch upstream drift: an honest backtest is a reproducible one. Pin a run's provenance with a manifest, and fail loudly if your data source silently changes.

import io, json, tempfile, os
from honest_backtest import run_drift_check, synthetic_fetcher, data_fingerprint

# 1. fingerprint any DataFrame deterministically (same index+columns+values -> same sha256)
fp = synthetic_fetcher("synthetic:7", "1h", "", "")
print("rows:", data_fingerprint(fp, name="demo")["n_rows"])

# 2. seed a reference of expected fingerprints, then gate against drift in CI
ref = os.path.join(tempfile.mkdtemp(), "reference_fingerprints.json")
json.dump({"targets": {"demo_btc": {"symbol": "synthetic:7", "timeframe": "1h",
                                    "start": "", "end": ""}}}, open(ref, "w"))
run_drift_check(ref, fetcher=synthetic_fetcher, seed=True, out=io.StringIO())   # seed once
print("same source -> rc", run_drift_check(ref, fetcher=synthetic_fetcher, out=io.StringIO()))   # 0
bad = lambda *a: synthetic_fetcher("synthetic:8", "1h", "", "")                 # data changed
print("drifted source -> rc", run_drift_check(ref, fetcher=bad, out=io.StringIO()))              # 1

Supply your own fetcher — any (symbol, timeframe, start, end) -> DataFrame — for real data sources; synthetic_fetcher is the bundled deterministic one for tests and examples.

API

Backtest harness: run_backtest, BacktestResult, run_walk_forward, walk_forward_splits, summary, sharpe, sortino, max_drawdown, calmar, ann_return, ann_vol. Honest-inference gates: deflated_sharpe_ratio, probabilistic_sharpe_ratio, expected_max_sharpe, gaussian_dsr, pbo, build_return_matrix · qlike, diebold_mariano · romano_wolf, reality_check, benjamini_hochberg · block_resample_idx, ci_from_bootstrap. Reproducibility: data_fingerprint, hash_config, git_info, package_versions, build_manifest, write_manifest · run_drift_check, synthetic_fetcher.

References

• Bailey & López de Prado (2014), The Deflated Sharpe Ratio, J. Portfolio Management.
• Bailey, Borwein, López de Prado & Zhu (2015), The Probability of Backtest Overfitting, J. Computational Finance.
• Diebold & Mariano (1995), Comparing Predictive Accuracy, J. Business & Economic Statistics.
• White (2000), A Reality Check for Data Snooping, Econometrica.
• Romano & Wolf (2005), Stepwise Multiple Testing as Formalized Data Snooping, Econometrica.

License

MIT © 2026 Robin Denis. Research software — not investment advice.