Catalog Capture

Independent, community-maintained capture tool (catalog_capture) that writes live market data into Nautilus Trader Rust ParquetDataCatalog layouts for direct backtest use.

Not affiliated with or endorsed by Nautech Systems Pty Ltd. See TRADEMARK.md and NOTICE.

venues → catalog-capture-cli + TOML
      → {catalog}/data/{type}/{instrument_id}/…parquet
      → ParquetDataCatalog / Rust backtest

Status

0.1.x — early open source, research / operator oriented.

• Multi-venue mainnet capture (public market data) has been run unattended for multi-day windows with scheduled segment sealing and Parquet readback.
• APIs and TOML may still evolve; pin a commit for production jobs.
• Best-effort support — see SECURITY.md and ROADMAP.md.

Features

• One product binary — catalog-capture-cli (TOML configs only)
• Rust catalog layout only — rust_canonical_only (Nautilus ParquetDataCatalog)
• Venue adapters — Binance Futures/Spot SBE, Bybit, Deribit, OKX, Hyperliquid, Lighter/Robinhood Chain, Extended, Bitfinex Spot L3, and Predict (venue-* features)
• Lighter binary options — fixed markets and automatic 15m capture via nautilus-lighter-ext (venue-lighter-ext); operator guide and rolling profile
• Binance Prophet — anonymous campaign depth and latest-price recording for fixed markets with venue-prophet. See the Prophet guide.
• Dynamic products — option universes, Hyperliquid HIP-4, and Predict crypto Up/Down selectors discover and rotate instruments without restarting
• Order book checkpoints — managed Nautilus books checkpointed hourly and at segment boundaries
• Segment lifecycle — append *.parquet.part, then seal on the configured schedule; rolling Predict products seal on their own interval boundaries
• Mainnet-oriented examples — public data by default (no keys in TOML)

Quick start

Requires Rust 1.99.1 and sibling ../nautilus_trader_quantity_commission. Run from the repo root.

# Clone + checkout the Nautilus revision pinned by CI (recommended)
make bootstrap-deps

# Standard public multi-venue profile: link only the adapters it uses
make build-release-capture
# default CAPTURE_FEATURES: venue-binance,venue-deribit,venue-hyperliquid,venue-lighter

./bin/catalog-capture-cli validate \
  --config examples/capture.multi-venue-mainnet.toml

# Long-running mainnet capture (logs under ./logs/)
./scripts/run-mainnet-capture.sh
# default config: examples/capture.multi-venue-mainnet.toml

Short smoke:

CAPTURE_SECONDS=120 ./scripts/run-mainnet-capture.sh

Data: ./data/… (gitignored). Catalog layout: docs/concepts/catalog_layout.md.

Recommended configs

Config	Use
examples/capture.multi-venue-mainnet.toml	Multi-venue instruments, quotes, trades and order-book data
examples/capture.prophet.toml	Binance Prophet campaign depth and latest price
examples/capture.predict-native.toml	Predict BTC 5m native YES depth and trades
examples/capture.predict-crypto-updown.toml	BTC/ETH/BNB 5m/15m Predict depth/trades plus Binance Spot SBE
examples/capture.extended-btc-perp.toml	Extended BTC perpetual public market data
examples/capture.bitfinex-spot-l3.toml	Bitfinex BTC/USD and ETH/USD public R0 order books
examples/capture.hyperliquid-hip4-btc-daily.toml	Hyperliquid rolling-universe example
examples/capture.deribit-btc-book-summary.toml	Deribit BookSummary only (1s poll)

More: examples/README.md.

Build options

All builds use the pinned huacnlee/cargo through scripts/cargo.sh. Run make install-cargo once; make bootstrap-deps also installs it. Rust 1.99.1 remains the compiler.

Command	When
make build-release-capture	Standard mainnet feature set; override CAPTURE_FEATURES for another profile
make build-release	All venues (all-venues)
make build-slim FEATURES=venue-deribit	Single-venue debug
make build-release-small	Smaller binary, slower compile
make cache-size / make clean	Inspect caches / clear this project's local and central build cache

Build size notes: docs/how_to/build_size.md.

Cloud / unattended

# Full runbook
# docs/how_to/cloud_capture.md

mkdir -p logs
nohup env NAUTILUS_LOG='stdout=Info;is_colored=false' \
  ./bin/catalog-capture-cli run \
  --config examples/capture.multi-venue-mainnet.toml \
  > logs/capture.log 2>&1 </dev/null &
echo $! > logs/capture.pid

# Metrics (if enabled in TOML): http://127.0.0.1:9108/metrics
# Stop gracefully: kill -TERM "$(cat logs/capture.pid)"

Start the binary directly so logs/capture.pid is the recorder PID. Do not background scripts/run-capture-service.sh; it is a foreground build/logging helper. See docs/how_to/unattended_capture.md for the stale-PID guard and shutdown wait.

Documentation

Section	Path
Doc map	docs/index.md
Install	docs/getting_started/installation.md
Cloud capture	docs/how_to/cloud_capture.md
Catalog layout	docs/concepts/catalog_layout.md
Dynamic universe / subscription lifecycle	docs/how_to/hip4_capture.md
Build size	docs/how_to/build_size.md
CLI	docs/reference/cli.md

Credentials

Most venue profiles use public market data. Predict and Binance Spot SBE require dedicated local credentials; secrets never belong in capture TOML. See credentials.

Development

make bootstrap-deps
make install-tools
pip install pre-commit && pre-commit install
make test && make clippy && make cargo-deny

See CONTRIBUTING.md, SECURITY.md, ROADMAP.md.

License

LGPL-3.0-or-later. Links against Nautilus Trader (LGPL). NOTICE.