Skip to content

SPEC.md

Latest commit

 

History

History
291 lines (201 loc) · 23.8 KB

SPEC.md

File metadata and controls

291 lines (201 loc) · 23.8 KB
Raw

PCA-Based Market-Neutral Statistical Arbitrage

An open-source research framework for extracting systematic risk factors from the S&P 500 via Principal Component Analysis and constructing dollar- and beta-neutral portfolios from the idiosyncratic residuals.

Table of Contents

  1. Motivation
  2. Project Goals & Non-Goals
  3. Theoretical Background
  4. Methodology
  5. System Architecture
  6. Data Layer
  7. Backtesting Engine
  8. Evaluation & Statistics
  9. Reproducibility & Engineering Standards
  10. Roadmap & Milestones
  11. Risks, Caveats & Known Limitations
  12. References
  13. Glossary

1. Motivation

The S&P 500 is commonly treated as "the market," but a cap-weighted index is a single composite signal that bundles together many distinct, overlapping sources of variation: a broad market factor, sector rotations, style tilts (value/growth, size, momentum), and stock-specific idiosyncratic moves. For a trader the index level is mostly noise relative to the structure that actually drives co-movement among constituents.

The thesis of this project is that the dominant axes of co-movement among S&P 500 constituents can be recovered statistically — without ever specifying sectors or fundamental factors by hand — using Principal Component Analysis (PCA) on the cross-section of stock returns. The leading principal components behave like tradable risk factors (the first typically loads positively on nearly every name and is a near-perfect proxy for "the market"). Once a stock's return is decomposed into its exposure to these statistical factors plus an idiosyncratic residual, the residual is, by construction, market-neutral. If those residuals are mean-reverting, there is a tradable signal: short stocks whose residual has run up, long stocks whose residual has fallen, and hold a portfolio whose net exposure to the recovered factors — and therefore to the broad market — is approximately zero.

This is a faithful, reproducible, open-source re-implementation and extension of the framework in Avellaneda & Lee (2010), built entirely on free data sources so that anyone can clone, run, and audit it end to end.

2. Project Goals & Non-Goals

Goals

  • Provide a clean, well-tested, well-documented Python library for PCA-based statistical arbitrage that others can fork, extend, and contribute to.
  • Recover statistical risk factors (eigenportfolios) from S&P 500 constituent returns and quantify each stock's correlation/loading with the market and with secondary factors.
  • Construct dollar-neutral and beta-neutral portfolios from the idiosyncratic residuals and evaluate them with institution-grade backtesting (transaction costs, borrow costs, realistic execution assumptions, survivorship-bias controls).
  • Ship a reproducible research pipeline: pinned environment, deterministic seeds, cached data, and one-command reproduction of every figure and table.
  • Be pedagogically transparent — the math in this spec maps one-to-one onto named modules and functions, so the repo doubles as a teaching reference for statistical arbitrage.

Non-Goals

  • This is not a live trading system. There is no broker integration, order router, or real-money execution. (A paper-trading adapter is a stretch goal, see Roadmap.)
  • This is not investment advice and makes no claim of profitability out-of-sample. Published statistical-arbitrage edges have decayed substantially since the 2000s.
  • We do not use proprietary or paid data vendors (CRSP, Bloomberg, Refinitiv). Free sources have known limitations (notably survivorship bias) that we document and mitigate, not eliminate.
  • We do not implement intraday or high-frequency strategies; the core cadence is daily.

3. Theoretical Background

3.1 The factor model

Let $R_{i,t}$ be the return of stock $i$ over period $t$, and let $\tilde R_{i,t} = (R_{i,t} - \bar R_i)/\sigma_i$ be its standardized return. A linear factor model posits

$$R_{i,t} = \alpha_i + \sum_{k=1}^{m} \beta_{i,k}, F_{k,t} + \varepsilon_{i,t},$$

where $F_{k,t}$ are $m$ common factors, $\beta_{i,k}$ are factor loadings, and $\varepsilon_{i,t}$ is the idiosyncratic residual with $\mathbb{E}[\varepsilon_{i,t}]=0$ and $\varepsilon_{i,t}$ uncorrelated with the factors. Classical approaches specify the factors (CAPM market factor, Fama–French, sector ETFs). PCA instead derives them.

3.2 PCA as statistical factor extraction

Form the empirical correlation matrix $\rho$ of standardized returns across $N$ stocks over a trailing window of length $T$. Its eigendecomposition is

$$\rho = \sum_{k=1}^{N} \lambda_k, v_k v_k^{\top}, \qquad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_N \ge 0.$$

Each eigenvector $v_k$ defines an eigenportfolio with weights

$$Q_{k,i} = \frac{v_{k,i}}{\sigma_i},$$

and the corresponding factor return is the return of that portfolio,

$$F_{k,t} = \sum_{i=1}^{N} Q_{k,i}, R_{i,t}.$$

Empirically the largest eigenvalue $\lambda_1$ is detached from the rest and its eigenvector has all-positive entries — this is the "market" eigenportfolio, and a stock's loading on it is a clean, data-driven measure of how correlated that name is with the market. Subsequent eigenportfolios capture sector and style structure. The number of significant factors $m$ is chosen so that the retained components explain a target fraction of variance (e.g. 55–65%) or by separating signal eigenvalues from the noise bulk predicted by the Marčenko–Pastur law for a random correlation matrix of aspect ratio $T/N$.

A note on the $T < N$ regime

With the default estimation window of $T = 252$ days and a universe approaching $N = 500$ names, the sample correlation matrix is rank-deficient (rank at most $T$) and badly conditioned: the Marčenko–Pastur ratio $q = N/T$ sits near 2, so a large block of eigenvalues is pure noise and a further block is exactly zero. The leading eigenportfolios (the market factor and the first few sector axes) are robust to this, but the higher-order factors and therefore the residuals that drive the signal inherit the estimation noise. We address this directly rather than ignoring it: before eigendecomposition the correlation matrix is cleaned, either by Ledoit–Wolf shrinkage toward a structured target or by Marčenko–Pastur eigenvalue clipping (replacing eigenvalues inside the noise bulk with their average, following Laloux et al. 1999). Cleaning is on by default and is itself a configurable, comparable choice; the variance-target and MP rules above then operate on the cleaned spectrum. Running with a longer window or a trimmed universe to push $q$ below 1 is supported as an alternative to cleaning.

3.3 From residuals to a tradable signal

Regress each stock's returns on the $m$ retained factor returns to obtain loadings $\beta_{i,k}$ and the residual series $\varepsilon_{i,t}$. Define the cumulative residual (the discrete analogue of an Ornstein–Uhlenbeck process)

$$X_{i,t} = \sum_{s=1}^{t} \varepsilon_{i,s}.$$

Fit an OU / AR(1) model $X_{i,t} = a_i + b_i X_{i,t-1} + \zeta_{i,t}$. The estimated parameters give a mean-reversion speed $\kappa_i = -\log(b_i)\cdot 252$ and an equilibrium level, from which Avellaneda–Lee define the dimensionless s-score

$$s_i = \frac{X_{i,t} - m_i}{\sigma_{\text{eq},i}},$$

the number of equilibrium standard deviations the residual sits from its mean. Trading rules act on the s-score: open a short when $s_i$ is large and positive (residual unusually rich), open a long when $s_i$ is large and negative, and close as it reverts toward zero. Stocks with insufficient mean reversion (low $\kappa_i$, i.e. long half-life) are excluded.

3.4 Why the resulting portfolio is market-neutral

Because positions are taken on residuals that have had the factor exposures projected out, the aggregate book carries approximately zero loading on the recovered factors. The construction step enforces this explicitly by neutralizing net dollar exposure and net beta to the first (market) eigenportfolio — and optionally to all $m$ retained factors — so the strategy's P&L is, by design, decoupled from the index level.

4. Methodology

The end-to-end pipeline, executed on a rolling basis at each rebalance date $t$:

  1. Universe construction. Resolve the S&P 500 membership as known on date $t$ (point-in-time, to mitigate survivorship and look-ahead bias). Apply liquidity and price filters.
  2. Return computation. Pull adjusted daily prices, compute log or simple returns over a trailing estimation window (default 252 trading days), winsorize extreme values, and standardize cross-sectionally.
  3. Factor extraction (PCA). Compute the correlation matrix, clean it (Ledoit–Wolf shrinkage or Marčenko–Pastur eigenvalue clipping) to control the $T < N$ noise discussed in 3.2, eigendecompose, select $m$ via explained-variance target and/or Marčenko–Pastur thresholding, and form eigenportfolio factor returns.
  4. Residual estimation. Regress each stock on the $m$ factors; collect betas and residuals.
  5. Signal generation. Fit the OU model per stock, compute s-scores, filter by mean-reversion speed, and map s-scores to target positions via entry/exit thresholds. The half-life and mean-reversion speed $\kappa_i$ are estimated from a short series and are noisy, so the engine carries the standard error of $\kappa_i$ and excludes names whose half-life filter decision is not robust to that estimation error, rather than acting on the point estimate alone.
  6. Portfolio construction. Size positions for dollar-neutrality and beta-neutrality, apply position/sector/turnover constraints, and produce target weights.
  7. Execution simulation. Trade from current to target weights subject to transaction-cost and borrow-cost models.
  8. Bookkeeping. Mark to market, accrue costs and financing, record P&L, exposures, and diagnostics.
  9. Roll forward to the next rebalance date and repeat.

Default parameters (all configurable)

Parameter Default Notes
Estimation window $T$ 252 days Trailing window for PCA & regression
Number of factors $m$ 15, or variance-target 55% Whichever rule is selected in config
Correlation-matrix cleaning Ledoit–Wolf shrinkage MP clipping or none also selectable
Rebalance cadence Daily Weekly/monthly supported
s-score entry (open short / long) $+1.25$ / $-1.25$ Symmetric thresholds
s-score exit (close short / long) $+0.50$ / $-0.75$ Asymmetric per Avellaneda–Lee
Max mean-reversion half-life 30 days Exclude slow reverters
Gross leverage 1.0–2.0× Per-side cap enforced
Transaction cost 5 bps/side Configurable cost model
Borrow cost 50 bps/yr Applied to short notional

5. System Architecture

The library is organized as a set of composable, single-responsibility modules behind clean interfaces. Each numbered methodology step maps to a module.

statarb/
├── data/
│   ├── universe.py        # point-in-time S&P 500 membership
│   ├── prices.py          # price/return ingestion, adjustment, caching
│   └── interfaces.py      # DataProvider ABC (pluggable sources)
├── factors/
│   ├── pca.py             # correlation matrix, eigendecomposition
│   ├── shrinkage.py       # Ledoit–Wolf / MP eigenvalue cleaning
│   ├── selection.py       # Marčenko–Pastur & variance-target rules
│   └── eigenportfolio.py  # eigenportfolio weights & factor returns
├── signals/
│   ├── residuals.py       # factor regression, residual series
│   ├── ou.py              # OU/AR(1) fit, half-life, s-score
│   └── rules.py           # s-score -> target positions
├── portfolio/
│   ├── construct.py       # dollar/beta neutralization, constraints
│   └── risk.py            # ex-ante exposures, factor risk decomposition
├── backtest/
│   ├── engine.py          # event loop, rolling estimation
│   ├── costs.py           # transaction & borrow cost models
│   └── ledger.py          # P&L, positions, marks
├── evaluation/
│   ├── metrics.py         # Sharpe, drawdown, turnover, etc.
│   ├── stats.py           # deflated Sharpe, bootstrap, DSR
│   └── report.py          # figures & tables
└── cli.py                 # one-command reproduction entrypoint

Design principles

  • Pluggable data via an abstract DataProvider. The default implementation wraps free APIs; swapping in a paid vendor is a single class.
  • Pure functions where possible. Factor extraction and signal generation are deterministic given inputs and a seed, which makes them trivially unit-testable.
  • No look-ahead by construction. The backtest engine only ever hands a module data strictly prior to the decision timestamp; this is enforced in code, not by convention.
  • Config-driven. Every default in the table above lives in a versioned YAML/TOML config so experiments are declarative and diffable.
  • Estimation hygiene, enforced. Three things that silently break this class of backtest are handled explicitly rather than left to convention: (1) the trading calendar and panel alignment, so a single stale or missing ticker cannot corrupt the cross-section on a given date; (2) a fixed PCA sign convention (the entry with the largest absolute loading is forced positive) so eigenportfolios are reproducible run to run; and (3) cross-sectional standardization and OU statistics computed from in-window data only, enforced by the same no-look-ahead guard as the rest of the engine.

6. Data Layer

6.1 Sources

Primary source is yfinance for adjusted daily OHLCV. Secondary/fallback sources behind the same DataProvider interface include Stooq and Alpha Vantage (free tier). The constituent list is bootstrapped from the Wikipedia S&P 500 table and, where available, historical change logs to approximate point-in-time membership.

6.2 Survivorship bias — the central data caveat

Free sources give the current index membership and only the prices of surviving tickers. Using today's 500 names to backtest the past silently discards delisted/acquired/bankrupt firms and inflates returns. Two facts about free data make this hard to fully fix, and we state them plainly rather than imply they are solved:

  • Point-in-time membership reconstructed from public S&P change announcements (and the Wikipedia change log) is accurate for recent years and progressively incomplete the further back you go. It is a best-effort approximation, not a clean historical index.
  • Even with correct historical membership, yfinance and the other free sources generally do not serve prices for delisted, acquired, or bankrupt tickers — which are exactly the names that matter most for a survivorship correction. The hole is largest precisely where it does the most damage.

Because of this, the realistic operating mode of the framework is a current-membership ("surviving names") backtest, and its results carry an upward survivorship bias by construction. The point-in-time path is implemented as best-effort and labelled as such wherever it is used. We mitigate, and quantify what we cannot mitigate, by:

  • Reconstructing historical membership where the change record allows, so backtest dates use the constituents known then, with explicit coverage flags where the record is thin.
  • Retaining price histories for delisted tickers where any free source still serves them, and flagging windows where coverage is incomplete.
  • Reporting a survivorship-sensitivity analysis as a first-class deliverable, so readers can see how much of the headline result is the surviving-names approximation rather than the strategy. This analysis, not a single headline Sharpe, is the honest product of the data layer.

This limitation is stated prominently in the README and every results table; honest treatment of it is a first-class requirement of the project.

6.3 Caching & integrity

All downloads are cached to a local parquet store keyed by ticker and date range, with checksums. A data verify command validates coverage, flags suspicious adjustments (e.g. unhandled splits), and reports gaps. Corporate-action handling relies on source-provided adjusted closes, with sanity checks for discontinuities.

7. Backtesting Engine

A single-threaded, event-driven loop over rebalance dates. At each date the engine:

  1. Slices the trailing estimation window (strictly historical).
  2. Calls the factor → signal → construction pipeline to get target weights.
  3. Computes trades against current holdings.
  4. Applies the cost models and updates the ledger.
  5. Advances and marks the book to the next date's prices.

Realism controls. Transaction costs (default 5 bps/side, with an optional notional- and volatility-scaled model), short borrow financing, gross- and per-name leverage caps, and a turnover penalty. Execution lag is on by default: the signal is computed on the close at $t$ and filled at the $t+1$ open, so the same close is never used for both signal and fill. Same-bar fills are the most common way a strat-arb backtest flatters itself, so the zero-lag mode is available only as an explicit, clearly labelled diagnostic. The engine is deterministic given a config and seed.

8. Evaluation & Statistics

Performance is never summarized by a single Sharpe ratio. The evaluation module reports:

  • Return/risk: annualized return, volatility, Sharpe, Sortino, Calmar; rolling Sharpe.
  • Drawdowns: max drawdown, average drawdown, recovery time, underwater curve.
  • Activity & cost: turnover, average holding period, gross/net leverage, total costs as a share of gross P&L, breakeven cost level.
  • Neutrality checks: realized beta to the market eigenportfolio and to SPY, net dollar exposure, and factor-risk decomposition over time — the strategy must demonstrably be market-neutral, not just claim it.
  • Overfitting controls: Deflated Sharpe Ratio and the probabilistic Sharpe ratio (Bailey–López de Prado) to penalize multiple-testing, stationary bootstrap confidence intervals on the Sharpe, and a strict walk-forward / out-of-sample split where all hyperparameters are frozen on a development period and evaluated once on held-out data.
  • Attribution: P&L by factor count $m$, by s-score bucket, and by GICS sector, to show where the edge comes from and whether it is concentrated or robust.

Every figure and table is regenerated by the CLI so results in the README are never stale relative to the code.

9. Reproducibility & Engineering Standards

  • Environment: pinned dependencies via pyproject.toml + lockfile; reproducible with uv/pip. Optional Docker image.
  • Determinism: global seed; PCA sign convention fixed; no nondeterministic parallelism in the critical path.
  • One-command repro: statarb run --config configs/baseline.yaml reproduces all artifacts; make figures rebuilds the README assets.
  • Testing: pytest unit tests for every math module (PCA against numpy.linalg ground truth, OU recovery on simulated paths with known $\kappa$, no-look-ahead assertions in the engine), plus property-based tests for the neutrality constraints. Target ≥85% coverage on factors/, signals/, portfolio/.
  • CI/CD: GitHub Actions running lint (ruff), format (black), type-check (mypy), and the test suite on every PR; a nightly job runs a small end-to-end smoke backtest.
  • Docs: module-level docstrings tied to the equations in this spec, a docs/ site, runnable example notebooks, CONTRIBUTING.md, CODE_OF_CONDUCT.md, issue/PR templates, and a permissive license (MIT or Apache-2.0).
  • Versioning: semantic versioning; a CHANGELOG.md; tagged releases.

10. Roadmap & Milestones

Phase Milestone Deliverable
M0 Scaffold & data layer Repo skeleton, DataProvider, cached S&P prices, data verify
M1 Factor engine PCA, eigenportfolios, factor selection, unit tests, eigenvalue/Marčenko–Pastur plots
M2 Signal engine Residual regression, OU fit, s-scores, signal diagnostics
M3 Portfolio & backtest Neutral construction, event loop, cost models, ledger
M4 Evaluation suite Full metrics, DSR/bootstrap, neutrality checks, walk-forward harness
M5 Reproducibility & docs CI, Docker, notebooks, docs site, baseline results in README
M6 (stretch) Extensions Paper-trading adapter, alternative factor models (sparse/robust PCA), regime conditioning, intraday cadence

11. Risks, Caveats & Known Limitations

  • Survivorship & data quality. Free data is the project's biggest threat to validity (Section 6.2). Results are best read as methodology demonstrations, not return forecasts.
  • Alpha decay. The Avellaneda–Lee edge weakened markedly after ~2007 as the trade became crowded; expect thin or negative net-of-cost performance in recent samples. This is an honest finding, not a bug.
  • Parameter sensitivity & overfitting. Many knobs ($m$, windows, thresholds) invite curve-fitting; the DSR, walk-forward split, and frozen-hyperparameter discipline exist specifically to keep this in check.
  • Estimation noise in the $T < N$ regime. With a 252-day window and a ~500-name universe the correlation matrix is rank-deficient and the higher-order factors are noise-dominated. Matrix cleaning (Section 3.2) reduces but does not remove this; the residuals that drive the signal are only as clean as the spectrum they are projected out of.
  • Non-stationarity. Correlation structure shifts across regimes (e.g. 2008, 2020); rolling re-estimation helps but cannot fully adapt to abrupt breaks.
  • Cost & execution realism. A 5 bps cost on daily rebalancing of a 500-name book is optimistic; the breakeven-cost report makes the fragility explicit.
  • Borrow availability. The model assumes shorts are always borrowable at a flat rate; hard-to-borrow names violate this.

12. References

  1. Avellaneda, M. & Lee, J. (2010). Statistical Arbitrage in the US Equities Market. Quantitative Finance, 10(7), 761–782.
  2. Bailey, D. & López de Prado, M. (2014). The Deflated Sharpe Ratio. Journal of Portfolio Management.
  3. Laloux, L., Cizeau, P., Bouchaud, J.-P. & Potters, M. (1999). Noise Dressing of Financial Correlation Matrices. Physical Review Letters.
  4. Marčenko, V. & Pastur, L. (1967). Distribution of Eigenvalues for Some Sets of Random Matrices.
  5. Tsay, R. (2010). Analysis of Financial Time Series. Wiley.

13. Glossary

  • Eigenportfolio — a portfolio whose weights are an eigenvector of the return correlation matrix (scaled by inverse volatility); its return is a statistical factor.
  • s-score — the standardized distance of a stock's cumulative residual from its OU equilibrium; the core entry/exit signal.
  • Market-neutral — a book with approximately zero net exposure (dollar and beta) to the broad market factor.
  • Marčenko–Pastur law — the limiting eigenvalue distribution of a random correlation matrix, used to separate signal factors from noise.
  • Deflated Sharpe Ratio — a Sharpe estimate adjusted for the number of trials and the higher moments of returns, to counter selection bias.
  • Correlation-matrix cleaning — shrinkage (Ledoit–Wolf) or Marčenko–Pastur eigenvalue clipping applied before eigendecomposition to suppress noise eigenvalues in the $T < N$ regime, so the recovered factors and residuals are less noise-contaminated.

This specification is the single source of truth for scope and methodology. Changes go through pull requests against this file.