A quantitative research case study. I took the published trade history of a third-party crypto trading agent ("BTC Futures Sentiment – Daily"), tested whether its trades contain a real, tradable edge, rebuilt the backtest with realistic execution, and tried to reproduce the agent's signal from public data.
Result in one line: the agent's trade timing did carry a real edge, but it could not be reproduced from public price, volume, funding or sentiment data, and the agent was later delisted. No tradable strategy came out of this project. The value is in the method: a realistic perpetual-futures backtester and a full validation process that caught several bugs, look-ahead leaks and a lot of overfitting.
Research only. Nothing here is investment advice, and none of the results are a forecast.
| Stage | What | Key finding |
|---|---|---|
| Data | Binance USDT-M perp 1-minute klines for 13 coins, real 8-hour funding, futures metrics (open interest, long/short ratios), premium index, Fear & Greed; 6 FX pairs and gold from HistData | All data quality-checked. HistData timestamps follow EU daylight saving (raw + 6 h = Berlin time), not the documented fixed EST; verified minute-by-minute against Dukascopy UTC |
| Signal timing | Reconciled the agent's logged P&L against candles | A trade labelled "day D 12:00 AM" actually fills at the end of D, ±30 min |
| Backtest v1 | Short ETH on each agent Short flip, cover on Long flip, 3% stop | Early reports claimed +27,000,000% and Sharpe 6.8; caused by compounding at leverage, a per-trade Sharpe annualised by √(252/hold), same-bar fills and close-only stops |
| Audit | Deflated Sharpe, probability of backtest overfitting (CSCV), matched random entries, parameter perturbation, cost stress, latency, event study | Statistics strong in-sample (t = 4.5, PBO 7.8%), but the rule to follow the agent was chosen with hindsight |
| Engine v2 | All leaks and bugs fixed (see below), revalidated | Design period +821% at 1× equity. Out-of-sample edge ≈ +0.68% per trade, about ⅓ of the design period, t = 1.1 |
| Own signal | Rebuilt the agent's signal from public price, funding and sentiment data (walk-forward, pre-registered pass/fail rules) | Imitation matched the agent's position on 67–69% of days but lost money on unseen data. Direct prediction was no better than a coin flip (AUC 0.52) |
| Other markets | Same signal on 12 other coins, 6 FX pairs, gold | Worked across crypto (one shared bet), no edge in FX or gold |
| ID | Problem | Fix |
|---|---|---|
| L1 | Entry at exactly 00:00 UTC although the agent's fill is only known to ±30 min, a possible look-ahead | Act at 00:30 UTC (cost: −18% of design-period return) |
| L2 | Stop and hold cap chosen on the full sample | Monthly walk-forward re-selection |
| L3 | Coin universe picked from coins that exist today (survivorship) | Final strategy trades ETH only |
| L5 | Spot candles used for a futures strategy | Binance perp candles + real funding |
| B1 | Funding at the entry-instant settlement was credited | Only settlements strictly inside the holding period |
| B2 | Open-trade equity ignored the entry fee and accrued funding | Marked with costs to date |
| B3 | Drawdown from daily closes only | Intraday-worst drawdown reported too |
| B4 | Sortino divided by the std of losing days | Standard downside deviation |
| B5 | Time-cap exits labelled as signal exits | Separate exit reasons |
src/engine_v2.py self-checks on every run: no entry before the signal is knowable, no overlapping positions, and net = gross − spread − fees + funding to the cent.
| Period | Trades | Return | Avg per trade | Notes |
|---|---|---|---|---|
| Out-of-sample Oct–Dec 2023 | 16 | +9.8% | +0.64% | never used in design |
| Design Jan 2024 – Mar 2026 | 123 | +821% | +1.97% | chosen with hindsight |
| Holdout Apr–May 2026 | 8 | +6.0% | +0.76% | last signals before the agent stopped |
| Jun–Sep 2026 | 0 | 0% | — | agent delisted, no signals |
Monte Carlo at 1× using the out-of-sample edge: median +35% in a year, 19% chance of a losing year, 57% chance of a drawdown beyond 20%.
src/ reusable code
paths.py all paths (data via $ETHFS_DATA, private logs, outputs)
engine_v2.py final ETH perp backtest engine (fixed, self-checking)
portfolio.py multi-coin perp engine (fees, spread, impact, funding, liquidation)
scanner.py asset scanner (liquidity, BTC beta/corr, volatility, recent edge, funding)
load.py, load_fx.py loaders for spot candles, agent logs, cleaned FX data
scripts/data/ download and build the datasets
scripts/research/ numbered research steps, 01 to 21, in the order they were done
private/ (git-ignored) third-party agent trade logs, not included
outputs/ (git-ignored) all results, reports and intermediate files
data -> $ETHFS_DATA (git-ignored) market data, about 2 GB
Research scripts in order:
| # | Script | Purpose |
|---|---|---|
| 01–02 | fill-time reconciliation, timeframe cross-check | when do the agent's fills really happen; are the candles consistent |
| 03–04 | event study, latency | is the edge real or leaked; how much does signal delay cost |
| 05–07 | replica data / explore / test | can public price data reproduce the signal |
| 08–09 | cross-asset, gold | does the signal work on other coins, FX, gold |
| 10–14 | portfolio scenarios, train/test, scanner, Monte Carlo | multi-coin compounding results and validation |
| 15a–d | audit | t-stats, Sharpe CI, Deflated Sharpe, PBO, matched random entries, reconciliation |
| 16 | improvements | maker execution, entry timing, agreement split |
| 17–19 | v2 attribution, validation, walk-forward + Monte Carlo | final corrected results and reports |
| 20–21 | own signal | rebuild the signal from sentiment data (pre-registered test) |
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
export ETHFS_DATA=/path/to/market-data # or create a "data" folder/symlink in the repo
python scripts/data/download_binance.py # perp klines, funding, metrics, premium, fear & greed
python scripts/data/build_binance_perp.py # clean per-coin parquet + quality report
# optional FX / gold
python scripts/data/download_histdata_fx.py && python scripts/data/build_histdata_fx.pyThe agent trade logs are not included: they belong to a third-party provider. Scripts that need them expect private/btc_trades_raw.csv (columns: entry, exit, side, size, pnl_usd, pnl_pct). Without them, the data pipeline, engines, scanner and validation tools still work with any signal you supply.
Run the final validation:
python scripts/research/18_v2_validate.py
python scripts/research/19_v2_walkforward_montecarlo.py- A strategy built on someone else's signal is only as durable as that signal. This one disappeared when the agent was delisted.
- Most of the headline return was artefacts: leverage compounding, inflated Sharpe, same-bar fills and close-only stops took an early "+27,000,000%" down to a realistic ~+35% a year expectation.
- Look-ahead can hide in timestamps: a ±30 minute ambiguity in when a signal is knowable was worth 18% of the return.
- Out-of-sample data is the only honest judge: the average out-of-sample trade was a third of the design period's.
- Pre-register pass/fail rules before testing a new idea, or the testing itself becomes overfitting.