ARCHITECTURE.md

Architecture

How the code is laid out and how data moves through it. The source of truth is the code; this document tracks it.

Project layout

arb_bot/
├── bot/          # runtime: exchange connectors, strategy, execution, risk, storage, signing, web
├── common/       # shared: types (Tick, ArbDirection), PnL/SL formulas, signal detection, config
├── optimizer/    # offline: grid/tpe/cmaes/funnel search over historical ticks (rayon-parallel)
├── config/       # default.toml, optimizer.toml, connections.toml, local.toml (overlay)
├── db/           # SQLite (arb_bot.db)
├── data/         # last_prices.json + Parquet ticks (data/ticks/) + funding/
├── logs/         # bot/, optimizer/
├── tools/        # bot_start.sh, tick_dedup.py, tick_recorder helpers
└── docs/         # DESIGN.md, ARCHITECTURE.md, PATTERNS.md, EXCHANGE_ADD_GUIDE.md,
                  # EXCHANGE_NOTES.md, EXCHANGE_CANDIDATES.md, OPTIMIZER_VS_BOT.md,
                  # LOCAL_SETUP.md, SERVER_SETUP.md

Three crates: common holds the shared logic (PnL, SL, signal detection, types); bot is the runtime; optimizer is the backtester.

Modules (bot/src/)

bot/src/
├── main.rs                     # entry point, async_main: startup, factories, WS tasks, event loop
├── lib.rs                      # module declarations + re-exports for integration tests
├── config.rs                   # Config struct + ExchangeConfig + TOML deserialization
├── constants.rs                # `pub use arb_common::constants::*;` — source of truth lives in common
├── types.rs                    # core types: Side, SymbolId, Position, Fill, Signals, ExchangeId, ...
├── symbols.rs                  # symbol discovery via each exchange's factory.fetch_symbols()
├── halt.rs                     # trading-halt manager (kill switch, API errors, manual halts)
├── tick_writer.rs              # Parquet writer (shared sink for all exchanges)
├── tick_writer_task.rs         # dedicated tokio task: tick_rx → TickParquetWriter
│
├── exchange/
│   ├── mod.rs                  # OrderRequest types (Market/Ioc/Gtt/StopLoss/StopLossLimit)
│   ├── net.rs                  # shared HTTP/WS/TLS: http_get, connect_wss_split, build_tls_connector
│   ├── traits.rs               # ExchangeClient trait (runtime API), MarketInfo, ConfigField
│   ├── factory.rs              # ExchangeFactory trait, BuildContext, BuiltExchange, FundingSource, build_factories()
│   ├── registry.rs             # ExchangeRegistry: runtime id↔name mapping from DB
│   ├── health.rs               # ExchangeHealth: per-channel state + two-tier WS freshness watchdog
│   ├── ws_reconnect.rs         # WsReconnect: supervisor loop + exponential backoff + health tracking
│   ├── ws_session.rs           # shared WS helpers: parse_close_frame, join_read_task, make_send_fn,
│   │                           #   spawn_write_task (RelayOnly/TextPingRelay/WsPingRelay), PendingOrders<K,M>
│   │
│   ├── lighter/                # ZK-proof signed, Sonic L2, per-market or single-stream WS
│   │   ├── client.rs, ws.rs, symbols.rs, mod.rs
│   │
│   ├── extended/               # STARK-signed, all-markets orderbook WS
│   │   ├── client.rs, ws.rs, symbols.rs, mod.rs
│   │
│   ├── zero1/                  # StarkNet perps, synchronous REST + account WS
│   │   ├── client.rs, factory.rs, rest.rs, ws.rs, session.rs, nonce.rs,
│   │   ├── liq_price.rs, market_info.rs, wire.rs, symbols.rs, funding_source.rs,
│   │   ├── zero1_schema.proto, mod.rs
│   │
│   ├── hyperliquid/            # DEX — EIP-712 msgpack signing, block-gated BBO feed, shared-connection WS
│   │   ├── client.rs, factory.rs, rest.rs, ws.rs, symbols.rs, market_info.rs, funding_source.rs, mod.rs
│   │
│   └── bybit/                  # the one CEX — HMAC auth, orderbook.1 BBO (AbsoluteReplace)
│       ├── client.rs, factory.rs, rest.rs, ws.rs, symbols.rs, market_info.rs, funding_source.rs, mod.rs
│
├── execution/
│   ├── mod.rs, helpers.rs
│   ├── executor.rs             # ArbEngine: event loop, position tracking, SL drift check, funding persistence, dashboard snapshots
│   ├── open.rs                 # execute_open_parallel: parallel entry, leg-risk recovery
│   ├── close.rs                # execute_close_parallel: parallel exit, residual close
│   ├── sl_monitor.rs           # SL movement: execute_sl_move, InFlightGuard (panic-safe RAII)
│   ├── orphan.rs               # orphan-fill handler (unmatched fills / single-leg stragglers)
│   ├── position.rs             # PositionTracker (Vec<Position>, no Arc/Mutex — single-threaded)
│   ├── position_events.rs      # PositionEventSink — façade for external leg closes (SL fire / liquidation / manual)
│   └── signal_log.rs           # signal audit log (SQLite `signals` table) helpers
│
├── strategy/
│   └── spread_detector.rs      # SpreadDetector: entry/exit signals, filters, per-symbol overrides
│
├── orderbook.rs                # L2OrderBook: bid/ask tracking, delta updates (absolute / incremental)
│
├── risk/
│   └── manager.rs              # RiskManager: capital reservation, daily loss limits, per-exchange balance
│
├── signing/
│   ├── mod.rs
│   ├── lighter.rs              # LighterSigner: sign_order, sign_update_leverage, auth_token
│   ├── ecgfp5.rs               # ECgFp5 crypto: Goldilocks GF, GFp5, Scalar, Poseidon2, Schnorr
│   ├── stark.rs                # Extended STARK curve (Pedersen hash, rfc6979)
│   ├── zero1.rs                # Zero1 session/derivation signing helpers
│   ├── hyperliquid.rs          # Hyperliquid EIP-712 msgpack action signing
│   └── bybit.rs                # Bybit HMAC-SHA256 request signing
│
├── storage/
│   ├── mod.rs
│   ├── types.rs                # row types: BlockType, SymbolBlock, OrderRecord, TradeRecord, ...
│   └── sqlite.rs               # storage impl: CRUD, schema, idempotent migrations
│
├── funding/
│   ├── rates_poll.rs           # public funding-rate poll task (iterates Vec<Box<dyn FundingSource>>)
│   ├── fetchers.rs             # per-exchange REST fetchers (Lighter, Extended, Zero1)
│   └── extended_payments.rs    # Extended-only authenticated cum_funding REST poll
│   # FundingRow + FundingWriter live in arb_common::funding (shared with the optimizer)
│
├── telegram/
│   ├── mod.rs                  # spawn() entry, NotifyEvent enum
│   ├── client.rs               # TelegramClient: send_message, Telegram Bot API
│   ├── format.rs               # message formatting (open/close/halt/error events)
│   └── worker.rs               # background worker: event queue, interactive commands (Status, Trades)
│
├── web/                        # Axum HTTP + SSE dashboard + auth + backtest/optimizer UI
│   ├── server.rs               # route definitions (50+ endpoints)
│   ├── state.rs                # DashState: live snapshots, TradeSnap, HotConfig
│   ├── auth.rs, auth_routes.rs # Argon2 password hashing, sessions, roles/permissions
│   ├── extractors.rs           # Axum extractors: session validation, permission checks
│   ├── credentials.rs          # UI for exchange API keys (written to .env, chmod 600)
│   ├── extended_api.rs         # Extended API proxy (rewards, leaderboard)
│   ├── backtest.rs             # web backtest API
│   ├── optimizer/              # web optimizer API (handlers.rs + state)
│   └── app.html / app.js / login.html  # SPA (hot-reloadable from disk)
│
└── utils/
    └── time.rs                 # now_ns(), current_unix_secs(), current_unix_ms()

Architectural decisions

Load-bearing choices — the reasoning behind each is worth reading before changing it.

• fastwebsockets 0.10 (no deflate) for every WS client, across all five exchanges.
• Five signing schemes, one per venue: Lighter's own ECgFp5 Schnorr, Extended's STARK curve (starknet-crypto, rfc6979), Zero1's Ed25519 session keys, Hyperliquid's EIP-712 msgpack, and Bybit's HMAC-SHA256.
• fastwebsockets::read_frame() is not cancel-safe — never pair it with tokio::select! or timeout. WS reads live in a dedicated task.
• Lighter WS is dual-phase: phase 1 uses a unified FragmentCollector at startup; phase 2 splits read and write for the main loop.
• Execution is parallel on both entry and exit — both legs fire at once.
• Two exit modes, pnl (percent of notional) and spread (captured PnL in bps), are roughly equivalent but not identical. Both ask "have we crossed the take-profit threshold?" but normalize differently: spread_bps_decimal divides close_spread by the close mid, while compute_pnl_decimal divides gross_pnl by the entry notional. The gap is O(Δmid / entry_mid) — about 0.1–0.2 bps at 1–2% intra-position drift, but ~48 bps at 10% (counterexample: is_long_1=true, entry 100/100, exit 110/100 → spread mode gives 952.4 bps, pnl mode 1000 bps). For typical scalps the difference is negligible; on volatile symbols right at the TP boundary the two modes can disagree on the decision. The include_fees_in_exit toggle selects the NET vs GROSS threshold for both. Shared formula: common/src/pnl.rs::spread_exit_net_bps_*.
• Order types are per-exchange, set in [exchanges.<name>]: entry_order_type (usually ioc/market), exit_order_type (ioc taker or gtt maker), stop_loss_type (Lighter: market/limit; Extended: limit only).
• SQLite is the only store. It is never recreated — CREATE TABLE IF NOT EXISTS plus idempotent migrations.
• exchange/net.rs is the shared network layer. Every exchange imports its primitives from there, never from another exchange module.
• The WS freshness watchdog has two tiers: WS_STALE_RECONNECT_MS = 30s (the per-WS task reconnects) and WS_STALE_HALT_MS = 45s (the executor gate blocks new entries) — a safety net for when a reconnect doesn't restore the stream on its own.

Data flow

                    ┌──────────────────┐
                    │   WS connectors  │
                    │ Lighter OB + Acc │  (exchange/lighter/ws.rs)
                    │ Extended OB + Acc│  (exchange/extended/ws.rs)
                    │ Zero1   Account  │  (exchange/zero1/ws.rs)
                    └────────┬─────────┘
                             │ BestLevels (mpsc)
                             ▼
                    ┌──────────────────┐
                    │    ArbEngine     │  (execution/executor.rs)
                    │ SpreadDetector   │──── OpenSignal ───► tokio::spawn(execute_open)  (execution/open.rs)
                    │ PositionTracker  │──── CloseSignal ──► tokio::spawn(execute_close) (execution/close.rs)
                    │ should_close()   │
                    └────────┬─────────┘
                             │ PositionUpdate (mpsc)
                             ▼
                    ┌──────────────────┐
                    │   state machine  │
                    │ Opened/Closed/   │
                    │ CloseFailed/     │
                    │ OpeningClosed    │
                    └────────┬─────────┘
                             │
                    ┌────────┴─────────┐
                    │   RiskManager    │  ← try_reserve / on_position_closed
                    │   SQLite DB      │  ← insert_trade / close_trade
                    │   web dashboard  │  ← SSE stream
                    └──────────────────┘

  ┌─────────────────────────────────────────────────────────────────┐
  │  In-bot tick recording (when bot.log_ticks = true)               │
  │                                                                   │
  │  ArbEngine.on_market_update                                       │
  │    → tick_tx.try_send(arc.clone())   (~10ns refcount bump)        │
  │    → tick_writer_task (own tokio task)                            │
  │       └─ TickParquetWriter buffers in RAM                         │
  │          ├─ rotation timer (60s) → spawn_blocking(write+compact)  │
  │          ├─ rotation timer also writes data/last_prices.json      │
  │          └─ disk-check timer    → spawn_blocking(cleanup)         │
  │    → funding::rates_poll task (own tokio task, hourly)            │
  │       └─ funding::fetchers::fetch_lighter/extended (public API)   │
  │          → arb_common::funding::FundingWriter                     │
  │             → data/funding/{exchange}/{symbol}/...                │
  │                                                                   │
  │  Recorded ticks are byte-identical to what spread_detector        │
  │  observed — backtest replay reproduces the bot's view exactly.    │
  └─────────────────────────────────────────────────────────────────┘

  ┌─────────────────────────────────────────────────────────────────┐
  │  Signal audit log (signals SQLite table)                         │
  │                                                                   │
  │  ArbEngine.on_market_update                                       │
  │    ENTRY: detector returns Some(OpenSignal)                       │
  │      → executor checks (halt/grace/max_per_sym/risk)              │
  │      → tokio::spawn(execute_open(signal))                         │
  │        → insert_trade (creates trade row)                         │
  │        → spawn_log_entry_signal:                                  │
  │            spawn_blocking { build SignalRow + JSON + INSERT }     │
  │            (trade_id known up front — no race, no backfill)       │
  │                                                                   │
  │    EXIT: detector returns Some(CloseSignal)                       │
  │      → spawn_log_exit_signal (clones signal, then spawn_blocking) │
  │      → tokio::spawn(execute_close(signal))                        │
  │                                                                   │
  │  Programmatic closes (close-all, abort, SL-drift emergency) are   │
  │  NOT logged: they carry no market context (legs would be empty).  │
  │  They're tracked separately via trades.exit_reason.               │
  │                                                                   │
  │  Each row carries the full bot view at decision time:             │
  │    - top-N orderbook for both legs (current AND prior tick)       │
  │    - VWAP price/qty, best bid/ask                                 │
  │    - exchange-side timestamps + staleness                         │
  │    - spread vs threshold + fee deduction                          │
  │    - exit_reason / PnL math (for exits)                           │
  │                                                                   │
  │  The per-leg snapshot is a JSON array in the `legs` column, so    │
  │  N>2 exchanges need no schema change. Schema: storage/sqlite.rs   │
  │  (CREATE TABLE signals); helpers: execution/signal_log.rs.        │
  │                                                                   │
  │  Hot-path cost: ~150ns per signal (Arc clone + spawn). All        │
  │  serialization and INSERT run in spawn_blocking.                  │
  └─────────────────────────────────────────────────────────────────┘

Entry execution (execution/open.rs)

Signal → RiskManager.try_reserve() → round qty/price per exchange (per leg, not globally)
  ↓
[PARALLEL]  execute_open_parallel(signal, ctx)
  ├─ leg 1: ctx.client(exchange_1).place_order(...)  (dyn ExchangeClient)
  └─ leg 2: ctx.client(exchange_2).place_order(...)  (dyn ExchangeClient)
  Each client knows how to send: Lighter = WS sendtx, Extended = REST POST,
  Zero1 = REST (the response carries exec price + liquidation immediately).
  Order type comes from ExchangeExecConfig.entry_order_type (usually Market or IOC).
  ↓
[PARALLEL AWAIT]  (timeout: entry_confirmation_timeout_ms per exchange)
  The fill notification returns through PendingOrders + the fill_tx channel
  that each exchange receives when it is built, via BuildContext.
  ↓
[RESULT]
  Both filled   → Position(Active), hedged_qty = min(leg1, leg2), excess mismatch-closed
  One filled    → close the filled leg (market), insert OpeningClosed, book the mismatch loss to orders
  None filled   → release_intention, no cost
  ↓
tokio::spawn(place_stop_losses(...)) — per-exchange via ExchangeClient::place_stop_loss()

build_entry_order() and resolve_entry_slippage() build an OrderRequest with a widened slippage cap (typically 5–15% of the per-symbol limit_price_cap) so the IOC leg doesn't miss on depth. Real rejects are almost always margin- or risk-side from the exchange.

Close execution (execution/close.rs)

CloseSignal → round qty per exchange
  ↓
[PARALLEL]  execute_close_parallel(signal, ctx)
  ├─ leg 1: ctx.client(exchange_1).close_position(...)
  └─ leg 2: ctx.client(exchange_2).close_position(...)
  Exit order type is per-exchange: IOC taker or GTT maker (from exit_order_type).
  ↓
[AWAIT] (peer_failed early abort)
  Both filled   → Closed, PnL via compute_pnl_decimal()
  One filled    → close the residual on the other exchange through the generic path
                  (the same close_position on the surviving leg)
  Both failed   → CloseFailed, position → Active (retry on the next signal)
  ↓
After a REST-confirmed close, wait 500ms to settle before the clean-close verify,
so the exchange's /positions endpoint has time to update (an Extended edge case).

Asymmetric closes are forbidden: both legs close together. A None counterparty anywhere in the close paths is a bug, not an edge case.

PositionEventSink (external leg closes)

execution/position_events.rs is the single point through which every exchange reports position events — state updates, funding settlements, and external closes (a server-side SL firing, a liquidation, a manual close from the exchange's own UI).

Before this façade, each exchange detected external closes on its own and sent ExternalLegClose straight into the shared mpsc, duplicated across four sites (2× Lighter, 2× Extended — and Zero1 didn't detect them at all, which cost a real orphaned leg on LIT). Now:

1. The per-exchange WS loop determines the reason (ExternalCloseReason) in the exchange's native format.
2. It calls sink.report_external_close(...).
3. The sink checks in-flight suppression — a whitelist of !StopLoss only, because suppressing everything would let entry fills through and close a freshly opened position.
4. If the close is genuinely external, it dispatches ExternalLegClose, and the executor symmetrically closes the surviving leg.

A new exchange gets one contract: call sink.report_external_close(...) with the best reason it can determine.

Stop-loss

The stop-loss is liquidation protection, not an arbitrary percentage off entry.

liq_dist = per-symbol from the exchange_symbols table (MMF data), or fallback 1/leverage
trigger  = liq_price ± entry × stop_loss_buffer_pct/100
price    = midpoint(trigger, liq_price)

Lighter:  StopLoss Market (type=2, tif=0) or StopLossLimit (type=3, tif=1), per config
Extended: StopLossLimit GTT (Type=1002) — a pure stop-market isn't available on Extended
Zero1:    trigger-based stop via rest.rs (the exchange takes trigger_price + limit_price)

The optimizer and the bot check the SL through one function: common/pnl.rs::check_sl_trigger_with_dist().

SL movement (execution/sl_monitor.rs): when the liquidation price drifts by ≥ sl_movement_threshold_pct, the bot moves the SL order via ExchangeClient::modify_stop_loss() — an atomic modify on Lighter (tx_type=17), a cancel-and-place fallback elsewhere. InFlightGuard (a panic-safe RAII guard) holds the in_flight_sl entry so a concurrent check_sl_drift() can't start a second move on the same (exchange, order_id). check_sl_drift() runs on every position snapshot.

Timing architecture

Absolute UTC timestamps (ms since epoch) are stored in the orders table:

Field	Event
placed_at_ms	order sent to the exchange
exchange_ts_ms	event on the exchange's matching engine (Lighter: last_updated_at, µs→ms; Extended: ts, ms)
first_fill_at_ms	first fill confirmed
last_fill_at_ms	last fill (partial fills)
signal_price	market price at the moment of the signal

The trades table holds signal_at_ms (entry-spread detection), exit_signal_at_ms (the close decision), opened_at_ms, and closed_at_ms.

Why signal_at_ms / exit_signal_at_ms are trade attributes, not order attributes. A signal is a single detector event — the spread between two exchanges for entry, or a TP/PnL condition for exit — that spawns both legs at once. By construction the value is identical for entry_1 and entry_2 (and likewise for exit_1/exit_2); storing it per order would duplicate the same number twice. The dashboard shows Recv→Sig per leg, but the signal reference is resolved from the trades table with a SQL CASE on orders.purpose:

CASE WHEN o.purpose LIKE 'exit%' THEN t.exit_signal_at_ms ELSE t.signal_at_ms END

The runtime equivalents are OpenSignal.ts_ns and CloseSignal.ts_ns in bot/src/types.rs — also one u64 per signal, not per leg. If a future refactor introduces cases where the two legs get different signal times (asynchronous pairs, say), the column moves to orders. Until then it lives on trades.

Durations are computed on the fly: detect_ms = signal_at - exchange_tick_at, fill_ms = first_fill_at - placed_at, and so on. The Position struct keeps every timestamp in nanoseconds and converts to ms on write.

Storage

SQLite (db/arb_bot.db):

• trades — arbitrage trades. Trade-level fields only: id, symbol, direction, qty, state, opened_at_ms, closed_at_ms, exit_reason, signal_at_ms, exit_signal_at_ms, entry_spread_bps, funding_1, funding_2, plus a cfg_* config snapshot (cfg_entry_bps, cfg_include_fees_in_entry, cfg_exit_mode, cfg_exit_bps, cfg_exit_pnl_pct, cfg_include_fees_in_exit, cfg_leverage). There are no net_pnl, net_pnl_pct, or fees columns — they were removed as denormalized caches that drifted from reality. Every PnL/fee figure is derived on read (see below).
• orders — every order (entry, close, SL) with placed_at_ms, first_fill_at_ms, last_fill_at_ms, fee (actual USD paid — the single source of truth for fees), signal_price, close_reason.
• exchange_symbols — per-symbol, per-exchange parameters (precision, leverage, MMF, market_name, limit_price_cap).
• exchanges — id→name registry (seeded at startup, auto-registered via ensure_exchange).
• exchange_events — connect/disconnect/error log for post-mortems.
• bot_runs — session history.
• symbol_blocks — per-exchange runtime symbol blocking (leverage violations, qty_min, API errors); exchange_id FK to a specific exchange.
• trading_halts — halt/resume history.
• sessions — session start/end tracking.
• extended_positions / extended_funding — Extended position and funding-fee history.

Parquet ticks (written by the bot's tick_writer_task):

• Layout: db/ticks/{exchange}/{symbol}/{YYYY-MM-DD}.parquet (compacted daily).
• Schema: ts_ms (receive time), exchange_ts_ms (exchange event time), 10 bid + 10 ask levels (price, qty).
• Segments rotate every 60s and compact into daily files.
• exchange_ts_ms: Lighter = last_updated_at (matching-engine event time, µs precision); Extended = ts (event time, ms).

Market metadata lives in exchange_symbols, fetched from each exchange's API at startup via ExchangeFactory::fetch_market_meta().

The DB is never recreated. Migrations are idempotent ALTER TABLE ... ADD/DROP COLUMN (SQLite ≥3.35); the schema lives in storage/sqlite.rs, and the row types (BlockType, SymbolBlock, OrderRecord, TradeRecord, …) in storage/types.rs.

PnL derivation — never cache a derived value

The trades table stores only what cannot be derived from orders. Anything financial that can be computed from per-order data plus trade-level funding is derived on read, not stored. There are exactly two sources of truth:

• orders.fee — the actual USD fee per order, written on the fill event (update_order_fill) and optionally refined via spawn_fee_fetch → fetch_trade_fee for exchanges with a per-order endpoint (Extended). For Lighter and Zero1, fetch_trade_fee returns None, so the fill-time value (notional × taker_rate) stands.
• trades.funding_1/2 — cumulative funding on the open position (accrued by the exchange, not per-order). Refreshed periodically from exchange_positions state and finalized in close_trade.

Everything else is derived:

Quantity	Formula
gross_pnl	(exit_1 − entry_1) × qty + (entry_2 − exit_2) × qty for is_long_1=true (mirrored for false); prices JOINed from orders.avg_fill_price on entry/exit purpose
fees	SUM(orders.fee WHERE trade_id = t.id)
funding_total	funding_1 + funding_2
net_pnl	gross_pnl − fees + funding_total
net_pnl_pct	net_pnl / notional × 100, where notional = qty × (entry_1 + entry_2) / 2

The calculation lives in exactly one place — bot/src/storage/sqlite.rs::derive_pnl() (precise via rust_decimal, with a both_filled guard for partial-fill / failed-open trades). Every PnL path routes through it: the per-trade loader (closed_trade_row_from_sql → derive_pnl) for UI, Telegram, and history endpoints; and the session and all-time aggregates (load_all_sessions_summary, load_all_time_stats), which share load_trades_for_aggregation and then fold derive_pnl in Rust. No PnL arithmetic happens on the SQL side.

The rule earns its keep: an earlier duplicate SQL CTE in the aggregates had no both_filled guard, so a failed-open trade with one filled leg (entry_1=0.978, entry_2=0) produced a phantom ~$89 per session, while the per-trade drawer (through derive_pnl) showed the correct cents — and no test caught the divergence. One source for PnL arithmetic eliminates the whole class. Two invariant tests pin it: load_all_time_stats_excludes_failed_open_phantom_gross (an exact regression fixture) and load_all_sessions_summary_handles_failed_open_and_multisession (bit-exact sum(session.total_pnl) == all_time.total_pnl).

compute_pnl_decimal in common/src/pnl.rs is not for realized PnL. It estimates from fee rates and is used only in gating (spread_detector: should_open, should_close, estimate_net_pnl, pnl_for_log) and in the backtest simulator (common/src/sim.rs). The caller passes fee rates; the function returns an estimate. Its docstring says so and points at storage::derive_pnl as the source of truth for closed-trade PnL. (The materialized trades.fees + trades.net_pnl this replaced once disagreed with SUM(orders.fee) by 2.3× on a real trade — part a Zero1 fee bug, part general dual-source drift.)

WS session infrastructure (exchange/ws_reconnect.rs, exchange/ws_session.rs)

Every WS session — orderbook, public trades, account — follows one architecture.

Supervisor loop (WsReconnect::run()), a generic reconnect wrapper:

WsReconnect::run(config, health, || tx.is_closed(), || session_fn(...))

• Exponential backoff, reset after long sessions (>300s).
• classify_error — custom maintenance detection (Lighter: a CloudFront WAF response → fixed 30s/120s backoff).
• skip_sleep_on_clean — the Lighter trading WS doesn't sleep on a clean disconnect (a server Close frame).
• update_health_on_disconnect = false for per-market connections (one of N dropping ≠ exchange down).

Session function (per exchange), one reconnect cycle:

1. Connect (split read/write for Lighter/Extended; unified for Zero1).
2. Subscribe + auth handshake (exchange-specific).
3. health.set_channel(true) — mark connected.
4. Spawn the write task via spawn_write_task(WriteTaskMode):
   • RelayOnly — pong relay only (Extended: a client ping kills the subscription).
   • TextPingRelay — pong + JSON ping (Lighter all-markets: {"type":"ping"}).
   • WsPingRelay — pong + WS OpCode::Ping + dead-connection detection (Lighter per-market).
5. Spawn the read task in AbortOnDrop with make_send_fn() for pong relay over the obligated channel.
6. Spawn the freshness watchdog (spawn_stale_data_watchdog) — aborts the read task on data silence.
7. join_read_task() — await and map JoinError (cancelled = watchdog, panic = fatal).
8. Abort the write task, return the result.

Fill routing through PendingOrders<K, M> (ws_session.rs):

• register(key, meta) / insert_sender(key, meta, tx) — register an in-flight order.
• resolve(key) → (meta, tx) — remove and return the sender (to deliver a fill).
• reject(key, reason) — remove and send Err (cancel/expiry/error).
• get_sender(key) — partial fill without removal.
• cleanup_closed() — periodic cleanup (executor timed out).

Key types: Lighter PendingOrders<u64, Side>, Extended PendingOrders<String>, Zero1 PendingOrders<u64>.

Lighter: trading WS

Doesn't use WsReconnect::run() — a manual loop with begin() / on_disconnect(), because order_rx is returned from the session for reuse across reconnects, nonces are cleaned between sessions (signer.nonces.clear_pending()), and leverage/positions setup happens only on the first session. The write task is a select! biased over five branches: outgoing orders from order_rx (build_order_frame + nonce management), obligated frames (pong), a 10s keepalive ping, a 200ms nonce resync poll, and the shutdown signal.

Zero1

• Unified WS (not split) — fastwebsockets handles Ping/Pong itself, no write task.
• IOC fills arrive via the REST receipt; WS fills are secondary/diagnostic.
• The first account frame is a snapshot — dispatch is suppressed for it.
• exchange_ts_ns == ts_ns — there is no matching-engine timestamp.

Extended

• No client pings — the server pings every ~15s and the client only relays pong; a text ping kills the subscription.
• Account WS: API-Key header on connect, ORDER + POSITION messages.

Full per-exchange details: EXCHANGE_NOTES.md.

Signing (signing/)

Lighter (lighter.rs + ecgfp5.rs):

• Goldilocks field (p = 2⁶⁴ − 2³² + 1)
• Extension GFp5 = GF[x]/(x⁵ − 3)
• Curve ECgFp5 (y² = x(x² + 2x + 263w))
• Hash Poseidon2 (width 12, rate 8, sbox x⁷)
• Sign: Schnorr over ECgFp5
• Nonce: AtomicU64, auto-increment
• Auth token: deadline:account:api_key:hex_sig

lighter.rs is the public API (LighterSigner, sign_order, sign_leverage, auth_token); ecgfp5.rs holds the crypto primitives (GF, GFp5, Scalar, ECPoint, Poseidon2, Schnorr) — standalone, no external dependencies.

Extended (stark.rs): STARK curve (starknet-crypto), Pedersen hash. Body: {signature: {r, s}, starkKey, collateralPosition}.

Hyperliquid (hyperliquid.rs): EIP-712 over an msgpack-encoded action → keccak → an Agent struct signed with domain {name:"Exchange", chainId:1337}. Field and key order are load-bearing — one wrong byte recovers a different address; the byte-exact test that pins it is in EXCHANGE_ADD_GUIDE.md.

Bybit (bybit.rs): HMAC-SHA256 over the request payload with the API secret — the one centralized venue.

Network layer (exchange/net.rs)

Shared across exchanges:

• http_get() / http_post() — low-level HTTP over TLS (hyper + rustls).
• connect_wss() / connect_wss_split() — WebSocket connections with TCP keep-alive.
• build_tls_connector() — a TLS connector with native root certs.
• SpawnExecutor — a hyper-compatible executor for the WS upgrade.

Every exchange imports these primitives from net.rs, never from another exchange module.

Exchange abstraction

The bot, the backtester, and the optimizer are fully generic over N exchanges. An exchange integrates through two traits.

ExchangeFactory (exchange/factory.rs) — the bootstrap lifecycle:

• name() — short tag ("lighter", "extended", …).
• fetch_symbols(), fetch_taker_fee() — preflight (parallel, before any client is built).
• build(exchange_id, ctx) — constructs the ExchangeClient, spawns the WS/poll tasks, and returns BuiltExchange { client, fee_rate, ready, clean_close_done, funding_source }.

build_factories(&Config) is the only place in the code that names concrete exchanges (one match arm each). From there, main.rs iterates Vec<Box<dyn ExchangeFactory>> with no per-exchange branches.

The clean_close_done contract — a future the exchange returns from build() in clean mode, resolved before ready (in main.rs, the clean_futs close in parallel across exchanges before the bot waits for OB snapshots). Three invariants; breaking any of them leaves an asymmetric leg at startup:

1. Wait for the WS to be ready before the first close order. clean_close_done starts before ready, so the account WS may not be subscribed yet. An order sent in that window would route its fill notification to a channel that doesn't exist yet, lose the fill event, and burn a 10s timeout. Two ways to satisfy it: poll health.is_channel_connected(eid, WsAccount) at the top of the future (Extended, Zero1), or start the close inside the same WS session task, after set_channel(WsAccount, true) (Lighter).
2. REST fallback on a WS timeout. An IOC may actually execute on the exchange while the fill event is lost. On a WS timeout, reconcile with the exchange over REST — fetch_trade_details by external_id (Extended) or fetch_open_positions on the next retry iteration (Zero1, Lighter).
3. Hard-fail on a residual after retries. After N attempts (5 for Extended/Zero1, 3 for Lighter), if positions remain, return Err — not Ok(()) + error!(). A silent pass would let the bot open new arbitrage positions on top of a dangling leg. Lighter signals this through a oneshot::Sender<anyhow::Result<()>>; the others return it directly from the future.

ExchangeClient (exchange/traits.rs) — the runtime trading API: market_info(), taker_fee(), limit_price_cap(), place_order(), place_stop_loss(), close_position(), cancel_order(), cancel_all_orders(), modify_stop_loss() (atomic on Lighter, cancel-and-place fallback elsewhere), fetch_equity(), fetch_open_positions(), fetch_liquidation_price(), check_order_status(), forget_pending_order(), config_schema(), update_emergency_slippage().

FundingSource (exchange/factory.rs, optional) — the public funding API for the periodic poll task. Any exchange with a public funding endpoint implements it, and funding::rates_poll iterates Vec<Box<dyn FundingSource>> without knowing a concrete exchange.

ExchangeId (common/types.rs) — an i64 newtype, mapped id→name through a OnceLock registry populated from the DB at startup. An exchange gets its .name() with no hardcoded match.

Config.exchanges — an IndexMap<String, ExchangeConfig>. IndexMap preserves TOML insertion order, so the eid (a position in config.exchanges.keys()) is stable across process restarts. That matters because symbol_overrides in SQLite stores (symbol, exchange_id_1, exchange_id_2) — an unstable eid would break the lookup.

ExchangeExecConfig (execution/executor.rs) — per-exchange trading params (fee, slippage, timeouts), built generically via from_exchange_config(&ExchangeConfig, fee_rate). ExecContext holds exchanges: HashMap<ExchangeId, Arc<dyn ExchangeClient>>, so open.rs and close.rs work through ctx.client(eid) with no knowledge of the concrete exchange.

Adding a new exchange: see EXCHANGE_ADD_GUIDE.md.

Exchange health

ExchangeHealth {
    channels: DashMap<(i64, HealthChannel), ChannelState>,  // dynamic, per-exchange
}

• Channels register at startup: (exchange_id, WsOrderbook), (exchange_id, WsAccount), and so on.
• exchange_connected(eid) — all of an exchange's channels up → ~1ns check.
• Blocks entries only (closes keep working).
• Events are written to exchange_events through a dedicated writer task.

Two-tier WS freshness watchdog (constants in common/src/constants.rs):

Constant	Value	Reader	Effect
WS_STALE_RECONNECT_MS	30 000	each exchange's per-WS task	force-reconnect its own connection
WS_STALE_HALT_MS	45 000	executor freshness gate	data_age_ms > 45s → block entries for that exchange
PER_MARKET_WS_STALE_MS	(in constants.rs)	Lighter per-market mode	a more generous threshold for per-market mode

Tier 2 (HALT_MS) is the safety net: if a connection's own reconnect hasn't restored the stream 15s past the reconnect threshold, the executor itself stops opening new positions. Both tiers read the same last_data_ms atomic, so the executor-side gate never drifts out of sync with the WS-side watchdog.

Risk management

RiskManager.try_reserve(signal):
  1. kill_switch?                                  → reject
  2. open_count >= max_open_positions?             → reject
  3. count_for_symbol >= max_positions_per_symbol? → reject
  4. used_capital + margin > max_used_capital_usd? → reject
  5. per-exchange balance >= margin_per_leg?       → reject
  → reserve capital atomically

on_position_closed(qty_usd, pnl):
  → release capital, update daily_pnl
  → if daily_pnl < daily_loss_limit → halt(KillSwitch)

Daily reset at 00:00 UTC: zero daily_pnl, resume kill switch.

Fee model

Taker fees are fetched at startup from authenticated endpoints (Lighter: /api/v1/accountLimits, Extended: /api/v1/user/fees, Zero1: the per-market leverage/fee-tier endpoint). Real values depend on the account's tier.

Leg	Entry	Close (IOC)	Close (GTT)
Lighter Standard	0%	0%	N/A
Lighter Premium tier 0	0.028%	0.028%	N/A
Lighter Premium tier 500k	0.0196%	0.0196%	N/A
Extended	~0.025% taker	~0.025% taker	0% maker
Zero1	per-market from market_info()	same	GTT not used on Zero1

Round-trip (entry + exit, both legs taker) tracks the account tiers; the exact numbers live in the exchange_symbols.taker_fee column. The signing fee rate for Extended is always 0.05% (MAX_TAKER_FEE_RATE) to cover the worst case.

Constants (common/src/constants.rs)

The source of truth is common/src/constants.rs, read by both the bot and the optimizer. bot/src/constants.rs is just pub use arb_common::constants::*; — nothing is added there. Bot-only constants the optimizer never needs still go in common, to keep the "one place" rule simple. By group:

• Time: NANOS_PER_MS, SECS_PER_DAY, TICK_ROTATION_INTERVAL_SECS (60), TICK_DISK_CHECK_INTERVAL_SECS (3600)
• Finance: BPS_FACTOR (10 000), LEGS_PER_POSITION (2)
• Prices: SAFE_MARKET_PRICE_BUY_MULT (×1.5), SAFE_MARKET_PRICE_SELL_MULT (×0.5)
• Entry: MAX_ENTRY_PRICE_DEVIATION (50%)
• Stop-loss: FALLBACK_LIQ_DISTANCE (10%), FALLBACK_SL_BUFFER (1%)
• WS freshness: WS_STALE_RECONNECT_MS, WS_STALE_HALT_MS, PER_MARKET_WS_STALE_MS (see Exchange health)
• Exchange limits: EXTENDED_MAX_SLIPPAGE_PCT (15%), EXTENDED_MAX_ORDER_LIFETIME_DAYS (90), …

User-facing settings go to config/default.toml + config.rs; internal limits and multipliers go to common/src/constants.rs. No bare const anywhere else.

Config hierarchy

config/default.toml       ← defaults (git-tracked, never modified at runtime)
config/local.toml         ← local overrides (git-ignored, auto-created)
config/connections.toml   ← exchange URLs (git-tracked)
.env                      ← exchange secrets (git-ignored)

Merge order: default.toml → local.toml overlay, with env vars for secrets only. local.toml is auto-created on the first save from the dashboard (a copy of default.toml); every UI change writes there, keeping default.toml clean for git. bot_start.sh also writes [startup] mode into local.toml.

Telegram (telegram/)

A background worker with an mpsc channel for notifications:

• NotifyEvent: Opened, Closed, OpenFailed, CloseFailed, Halted, Resumed.
• Interactive commands: Status (positions, balances, health) and Trades (recent closes).
• Formatting: Markdown, PnL coloring, fee breakdown, timing info.
• Config: the [telegram] section (bot_token + chat_id); empty values disable it.

Web dashboard

Axum HTTP + SSE on 0.0.0.0:3030, with auth:

• Auth: Argon2 password hashing, session tokens, role-based permissions (admin/user).
• GET / — HTML dashboard (hot-reloadable).
• GET /api/events — SSE with live data (positions, trades, health, balances, spreads).
• GET /api/state — current snapshot (JSON).
• POST /api/halt/pause · POST /api/halt/resume — manual trading halt.
• POST /api/config — live parameter update (entry_bps, exit_bps, exit_mode, per-exchange settings).
• POST /api/close-trade/{id} — close a specific position.
• POST /api/close-all — close all active positions.
• GET /api/history/* — session history, closed trades.
• GET /api/extended/* — Extended funding, positions, rewards.

Backtest / optimizer data model

The simulator and the optimizer are both generic over N exchanges. A single run_simulation() in common/src/sim.rs serves the CLI optimizer (optimizer/src/main.rs), the web optimizer (bot/src/web/optimizer/handlers.rs), and the web backtest panel (bot/src/web/backtest.rs).

Data flow (TOML → DB rows)

default.toml [exchanges.X] sections          ← section order = eid order
        ↓ (IndexMap, deterministic)
Config.exchanges: IndexMap<String, ExchangeConfig>
        ↓ Config::configured_exchange_names()
Vec<String>  ["lighter", "extended", "newex"]
        ↓ build_sim_params(overrides, db_path, exchange_refs)
SimParams {
    exchanges: Vec<ExchangeSimSpec>,  ← one spec per exchange
    pairs: Vec<(usize, usize)>,       ← enumerate_pairs(N) = C(N, 2), lex order
    ...
}
        ↓ run_simulation(store, params, ...)
SimResult { trades: Vec<ClosedTrade>, ... }
ClosedTrade { eid_1: u8, eid_2: u8, ... }
        ↓ apply_optimizer_result_inner(db, run_id, rank)
N rows in symbol_overrides, one transaction, all linked by source_run_id

Key types (common/src/backtest.rs)

OrderType — a Market | Ioc | Gtt enum that replaced stringly-typed "ioc"/"gtt" fields. Used in ExchangeSimSpec and in the effective-fee dispatch (Market/Ioc → taker, Gtt → maker).

ExchangeSimSpec — a POD struct, one per exchange: name, taker_fee, maker_fee, entry_fee, close_fee, fill_delay_ms, entry_order_type, exit_order_type. It replaced seven parallel Vec<...> in SimParams. The hot loop reads params.exchanges[eid].field — one Vec index, no vtable, compiling identically to the old params.entry_fees[eid].

SimParams.pairs: Vec<(usize, usize)> — every trading pair the sim runs, filled by enumerate_pairs(n) (below).

Position { eid_1: u8, eid_2: u8, is_long_1, ... } — eid_1/eid_2 are runtime exchange indices into params.exchanges / store.bids[], not global DB ids and not "Lighter/Extended". Each open position knows which two exchanges hold it; is_long_1 means "long on eid_1".

enumerate_pairs(n) -> Vec<(usize, usize)> — lex order, [(0,1), (0,2), (1,2), ...]. Stable and deterministic. It carries a deliberate bias, documented in its doc-comment and pinned by a regression test — see DESIGN.md → Pair iteration order.

Sim hot loop (common/src/sim.rs::run_simulation)

for tick_idx in 0..store.len() {
    // Exit check: for each open position, read
    //   tick.bid(pos.eid_1), tick.ask(pos.eid_2), ...
    // → this position's exact pair of exchanges, not a hardcoded (0, 1)

    // Entry check: outer gates (kill_switch, capital, ...) +
    //              prev_ba snapshot capture-then-update
    for &(i, j) in &params.pairs {
        // Inner pair loop. Per-pair gates: cooldown, recv staleness,
        //   VWAP/spread, top-level depth, spike check.
        // Globally shared state (capital, max_open_positions) is re-checked
        //   every iteration → the second pair won't open if the first
        //   exhausted capital (`break`).
        if signaled { open_position(eid_1=i, eid_2=j, ...) }
    }
}

Details worth knowing:

• prev_for_tick — a hoisted Vec<Option<(f64, f64)>> allocated once before the loop and reused per tick. This removed the old compile-time cap of N ≤ 16.
• last_signal_time: HashMap<(SymbolId, u8, u8), f64> — keyed by (symbol, eid_1, eid_2), so each pair has an independent cooldown, exactly as in the live bot.
• prev_ba: HashMap<(SymbolId, u8), (f64, f64)> — keyed per (symbol, exchange); the spike check reads per leg of the pair being evaluated.
• The capital pool is global. max_used_capital_usd is shared across all pairs; if the first pair takes a slot, the rest break.

Apply best params (apply_optimizer_result_inner)

Single source of truth: bot/src/web/optimizer/handlers.rs::apply_optimizer_result_inner, a pure function tested against in-memory SQLite (bot/tests/apply_overrides.rs); the HTTP handler api_optimizer_apply is a thin wrapper. In global mode it:

1. Reads optimizer_runs.config_json["exchanges"] — the run's list of exchange names.
2. Resolves names to DB ids via Storage::resolve_exchange_ids (loud error on an unknown name).
3. enumerate_pairs(n) → C(n, 2) pairs.
4. Within with_transaction, writes an identical row to symbol_overrides for each pair (i, j) — eid_1 = exchange_ids[i], eid_2 = exchange_ids[j], shared source_run_id. upsert_symbol_override canonicalizes (id1, id2) to smaller-first.

N=2 → 1 row (as before), N=3 → 3 rows, N=4 → 6, all linked by source_run_id. Saved runs with no exchanges field fall back to ["lighter","extended"] — a legitimate compat shim for older persisted rows.

Snapshot regression net

The contract: after any change to common/src/sim.rs or common/src/backtest.rs, the snapshot_2leg + snapshot_3leg tests must pass byte-exact, with no baseline regeneration. If they fail, it's a refactor bug, not intentional drift. Regenerate only via the INSERT_SNAPSHOT=1 env var, and only after inspecting the diff by hand (it should be understandable, not "an f64 moved a couple of ULP").

Alongside the snapshots, multi_pair.rs (5 cases) documents the behavioral invariants: capital-pool sharing, the per-symbol cap across pairs, per-eid funding-cursor indexing, simultaneous opens on multiple pairs, and the pair-iteration bias (lex order wins capital ties).

Optimizer modes

Four modes — grid (exhaustive, rayon-parallel), tpe (Bayesian), cmaes (evolutionary), funnel (multi-phase) — all driven by the one run_simulation(). Shared with the bot through arb_common: compute_taker_spreads(), select_entry_direction(), compute_vwap(), compute_order_qty(), compute_pnl_f64(), check_sl_trigger_with_dist(). Where they differ: OPTIMIZER_VS_BOT.md.

Tests

Unit tests (cargo test): config parsing, orderbook deltas, GF/GFp5/Scalar arithmetic, ECgFp5 Schnorr vectors, STARK signing, OrderRequest fields.

Integration tests live in bot/tests/ (most are #[ignore] — they need a live exchange):

• exchanges/lighter/ — REST, WS, orders, fees, SL, close_all (~21 files)
• exchanges/extended/ — REST, WS, orders, liquidation_check, TP/SL, sl_price_type (~18)
• exchanges/zero1/ — e2e_single_trade, ws_integration, liq_price_live, sl_move_sim, funding_liq (~18)
• integration/ — full_flow, close_all, close_single, margin_reject, apply_overrides
• leverage/ — prober, batch, speed
• tools/ — spread analyzer, latency measure, market-params fetcher, depth check, ws_many_vs_single, ws_sniffer, timestamp semantics
• common/ — shared helpers (TLS, auth, API clients)

Run exchange tests with cargo test --test exchanges -- exchanges::lighter::orders --ignored --nocapture. Run tools with cargo test --test tools -- <exact_path> --ignored --nocapture. Careful: --ignored matches on substring, so for destructive tests (close_all_*, cancel_all_*) always give the full path, or you can wipe a live account by accident. Test-writing conventions: PATTERNS.md → Tests.