Skip to content

Latest commit

 

History

History
935 lines (700 loc) · 55.5 KB

EXCHANGE_ADD_GUIDE.md

File metadata and controls

935 lines (700 loc) · 55.5 KB

Adding a New Exchange — An Executable Playbook

How to use this playbook. Work through the phases in order — each gate is a check to pass before moving on, not a rough guide to skim. Turn every phase, sub-step, gate, and test into a checklist and tick them off; a skipped step is a bug found in review instead of one prevented up front. For any WebSocket keepalive/ping/timeout question, read the exchange's docs first, then write code. Every design decision here — tick feed, delta mode, ping strategy, liq-price source — is made from the Phase 1 empirical data plus the exchange's documentation, never guessed.

Architecture (one screen)

The bot, backtester, and optimizer are all generic over N exchanges. An exchange implements:

  1. ExchangeFactory (exchange/factory.rs) — bootstrap: fetch_symbols, fetch_taker_fee, build() → BuiltExchange.
  2. ExchangeClient (exchange/traits.rs) — runtime: place_order, cancel_order, place_stop_loss, close_position, fetch_liquidation_price, …
  3. FundingSource (exchange/factory.rs) — optional: the public funding-rate API.

The backtester and optimizer need no new code — they run on ExchangeSimSpec, which is data-only.


Phases

Phase What you do Artifact Code in exchange/newex/?
1 Empirically explore the API An EXCHANGE_NOTES.md entry + probe tests No
2 Config + module skeleton + symbols + signing Compiles; signing tests pass Skeleton
3 REST client REST smoke tests pass rest.rs
4 WebSocket (OB + Account + Trades) Ticks flow; fills route ws.rs
5 Factory + wiring + clean_close_done cargo test --workspace --lib green Complete
6 Live validation 3 high-risk tests pass Final

Phase 1 — Empirical exploration (no exchange code)

Goal: gather every empirical fact that Phases 2–5 depend on. Write no code under bot/src/exchange/newex/. Probes are disposable — curl, scratch scripts, #[ignore] tests in bot/tests/exchanges/newex/.

Read first

Before the first curl:

  • DESIGN.md §2 (trade lifecycle), §5 (order execution + leg risk), §12 (leverage/margin/SL/liquidation).
  • ARCHITECTURE.md — "Exchange abstraction" + the clean_close_done contract (three invariants).
  • EXCHANGE_NOTES.md — the entry format and the empirical data for the existing venues.
  • The existing implementations — open bot/src/exchange/{lighter,extended,zero1}/ side by side as reference.

1.0 Read the docs before probing

Before writing any WS/REST probe, find and read the official docs:

WebSearch "exchange_name websocket ping heartbeat timeout"
WebSearch "exchange_name API rate limits connection limits"
WebSearch "exchange_name order types IOC trigger stop loss"
WebFetch "https://docs.exchange.com/api/websocket"

Record in EXCHANGE_NOTES.md before writing code: keepalive (who pings — server or client — the ping format, interval, timeout: Camp A/B/C), connection limits (max connections, max subscriptions per connection), rate limits (requests/min, weight per endpoint), order types (IOC/FOK/GTC, trigger orders, reduce-only), the signing scheme (HMAC / Ed25519 / EIP-712 / …), and the wire format (JSON / protobuf / msgpack, with examples from the docs).

A lesson from Hyperliquid: guessing Camp A (server pings) cost 30 minutes of bot restarts. The reality — Camp B (the client pings every 60s) — was stated plainly in the docs.

1.1 WS write-capability probe

Before designing the REST client, check whether the exchange supports placing orders and other write operations over WebSocket:

WebSearch "exchange_name websocket place order cancel order"
WebSearch "exchange_name websocket API write operations"

Look for order placement, cancel, trigger/SL, modify, and leverage/margin set over WS. If the exchange offers the same write operation over both WS and REST, write an #[ignore] test that benchmarks both:

// bot/tests/exchanges/newex/ws_vs_rest_latency.rs
// 1. Prepare a signed IOC order at a deliberately unfillable price (no fill).
// 2. Send it N times over REST, measure round-trip (send → ack/reject).
// 3. Send it N times over WS,   measure round-trip (send → ack/reject).
// 4. Report p50/p90/p99 for both, and the difference in ms and %.

Record in EXCHANGE_NOTES.md: which write operations are available over WS, the WS action format (JSON-RPC, custom JSON, msgpack, …), whether the WS returns a response/ack with fill info, and the WS-vs-REST latency benchmark.

Why it's critical: REST means a new TCP/TLS connection or HTTP request overhead on every order; WS reuses a persistent connection at ~0ms overhead — an order of magnitude faster. For arbitrage, entry/exit latency is money. (Lesson: Hyperliquid supports place/cancel/modify/SL over WS, but the integration was written against REST — forcing a full rewrite of the write path.)

1.2 REST probes

For each endpoint (markets, fees, positions, orders, trades, funding, balance):

# example — adapt to the exchange
curl -s -H "Authorization: ..." https://api.newex.com/v1/markets | jq '.' | head -50
# ALWAYS redact secrets (awk filter) before logging

Record: the HTTP auth scheme (header, query param, signed body), the response shape and field types (number vs string), latency from a regional probe (Tokyo if that's the deploy target), and the /positions freshness lag (open a position → close it → measure how long REST takes to update).

1.3 WS probes — raw, no FragmentCollector

Write a disposable probe with ws.set_auto_pong(false) (raw WebSocket) — FragmentCollector hides Ping frames.

Property How to check Why
Who pings? Sit idle 3+ min, log every opcode Determines the write-task strategy
Ping interval Time between Ping frames For the keepalive timeout
Timeout with no pong Don't answer a Ping, time the disconnect For reconnect logic
Client ping allowed? Send {"type":"ping"}, watch On Extended it kills the subscription!
Close-frame payload Log the Close code + reason For classify_error

The three keepalive camps:

Camp Server pings? Client must ping? Example WriteTaskMode
A Yes No, and mustn't Extended, Zero1 RelayOnly
B No Yes, required Lighter TextPingRelay or WsPingRelay
C Yes Allowed, not required (hypothetical) RelayOnly

1.4 Tick feed — probe every channel

The most important decision. Don't commit to a channel without data.

1.4.a Inventory the channels. Check every source: the exchange's WS reference page, the official SDK source (@exchange/ts, exchange-python) for channels the docs omit, GitHub examples, the Discord/Telegram #api channel, and the exchange team directly. Look for a full L2 order book (orderbook, depth, book, deltas), top-of-book / BBO / L1 (bookTicker, ticker, bbo, quote, depth=1) — which can be 5–10× faster — public trades, undocumented depth parameters (?depth=N, ?level=1, ?compress=true), and combined streams.

1.4.b Live latency tests. For each candidate channel, an #[ignore] test in bot/tests/exchanges/newex/: connect, subscribe, hold 120+ seconds; per frame record Instant::now() + the engine timestamp if any; report frame count, rate/s, and inter-arrival gap p50/p90/p99/max; end with a === VERDICT === block naming the winning channel and by how many ms. Two key comparisons: full L2 vs BBO/ticker, and orderbook vs public-trades alignment (shared sequence number?).

1.4.c Solo vs shared topology. Two sockets side by side — SOLO (one subscription on the most active symbol) and SHARED (that symbol + N−1 others) — measured over 120+ seconds. If SOLO is tighter by >5ms p50, you need a per_market dedication option.

1.5 Connection limits

  • Stagger: spawn N WS connections at varying intervals (100/250/500/1000ms); find the minimum that avoids a 429.
  • Subscription cap per connection: Zero1 = 10 (→ batch supervisor), Extended = unlimited.
  • REST rate limits: a burst probe; record the real limit vs the documented one.

1.6 Min-order rules — probe both floors

An exchange can reject a submit on two independent rules: a quantity floor (size < min_order_size) and/or a notional floor (size × price < min_notional_usd). Before writing MarketInfo (§2.8), find which your exchange has:

  1. Read the markets / trading-info endpoint. Look for fields rooted in min / minimum — qty (minOrderSize, min_base_amount, minBaseSize, minQty, lot_size) and notional (min_quote_amount, minOrderValue, minNotional, minOrderValueQuote).
  2. If a floor isn't published, probe it empirically. An #[ignore] test that sends a deliberately-too-small IOC and logs the exact server error, repeated around the expected threshold. A real Hyperliquid rejection — "Order must have minimum value of $10. asset=217" — is how we learned it enforces a $10 notional with no explicit API field.
  3. Both are independent. Even after finding one, check the other. Lighter publishes both min_base_amount and min_quote_amount per symbol; parsing only the notional one once let qty-based rejects reach production as orphans.
  4. Record it in EXCHANGE_NOTES.md — the exact field name, a typical value, the endpoint, and an example rejection message. Phase 2.8 carries this into the MarketInfo impl.

1.7 Delta mode + thin markets

  • Order-book delta mode: incremental (size = delta, needs a REST snapshot) or absolute-replace (size = final, the first WS frame bootstraps the book) — this decides DeltaMode::Incremental vs DeltaMode::AbsoluteReplace.
  • Thin-market idle gap: watch the thinnest market for 2+ min and record the largest gap between deltas — a lower bound for ws_read_timeout_secs.

1.8 Liquidation-price source

Open a tiny leveraged position and read the liquidation price over both REST and WS:

Option When Implementation
Engine WS push liquidation_price in the position-update frame Read from the WS cache, REST fallback
Engine REST only No WS push, but REST returns it REST poll in fetch_liquidation_price
Client-side compute The engine returns nothing liq_price.rs + REST for a fresh mark price

1.9 Register the test target

Create bot/tests/exchanges/newex/main.rs and a [[test]] entry in bot/Cargo.toml:

[[test]]
name = "newex"
path = "tests/exchanges/newex/main.rs"

Commit the probes from 1.3–1.7 here as #[ignore] tests. Run them with cargo test --test newex -- <name> --ignored --nocapture.

Gate: EXCHANGE_NOTES.md has an entry with all of the data above — WS write capabilities, REST auth scheme + /positions lag, WS ping semantics (camp A/B/C), the full channel inventory + cadence results + topology verdict + chosen tick feed, stagger limit, subscription cap, delta mode, thin-market idle gap, and liquidation-price source. All cadence tests are committed.


Phase 2 — Config + module skeleton + symbols + signing

Goal: the skeleton compiles and signing tests pass. Trait implementations are empty stubs.

2.1 URLs in constants.rs

common/src/constants.rs, beside LIGHTER_REST_URL / EXTENDED_REST_URL / ZERO1_REST_URL:

pub const NEWEX_REST_URL: &str = "https://api.newex.com";
pub const NEWEX_WS_URL:   &str = "wss://ws.newex.com/stream";
// + channel-split variants if orderbook/account/trades use separate sockets

2.2 Config struct + builder

bot/src/config.rs — the struct beside LighterConfig / ExtendedConfig / Zero1Config:

pub struct NewexConfig {
    pub api_key: String,
    pub private_key: String,
    pub rest_url: String,
    pub ws_url: String,
    pub ws_read_timeout_secs: u64,
    // + exchange-specific fields
}

The builder inside impl Config, beside fn lighter() / fn extended() / fn zero1():

impl Config {
    pub fn newex(&self) -> NewexConfig {
        let ec = self.exchange_config("newex");
        let s = self.secrets.get("newex");
        NewexConfig {
            api_key: s.and_then(|s| s.require("API_KEY").ok()).unwrap_or("").to_string(),
            private_key: s.and_then(|s| s.require("PRIVATE_KEY").ok()).unwrap_or("").to_string(),
            rest_url: arb_common::constants::NEWEX_REST_URL.to_string(),
            ws_url: arb_common::constants::NEWEX_WS_URL.to_string(),
            ws_read_timeout_secs: ec.ws_read_timeout_secs,
        }
    }
}

URLs come from arb_common::constants, never from a local const inside the exchange module.

2.3 TOML config

config/default.toml — commented out so auto-deploy doesn't activate it:

# [exchanges.newex]
# margin_mode                   = "isolated"
# entry_order_type              = "ioc"
# entry_slippage_pct            = "max"
# exit_order_type               = "ioc"
# exit_slippage_pct             = "max"
# emergency_slippage_pct        = "max"
# stop_loss_type                = "ioc"
# ws_read_timeout_secs          = 30
# sim_fill_delay_ms             = 200
# # confirmation_timeout: default 1500ms (config.rs). Override only if the exchange is faster/slower:
# # entry_confirmation_timeout_ms = 1000
# # exit_confirmation_timeout_ms  = 1000

Section order in default.toml is the eid assignment order (IndexMap).

2.4 Secrets

.env:

NEWEX_API_KEY=...
NEWEX_PRIVATE_KEY=...

ExchangeSecrets::from_env("newex") loads everything prefixed NEWEX_.

2.5 Module directory

Create bot/src/exchange/newex/:

bot/src/exchange/newex/
├── mod.rs              # pub mod declarations
├── client.rs           # NewexClient (stub impl ExchangeClient)
├── factory.rs          # NewexFactory (stub impl ExchangeFactory)
├── rest.rs             # REST client with typed response structs
├── ws.rs               # WS handlers (placeholder)
├── symbols.rs          # to_canonical / from_canonical
├── market_info.rs      # NewexMarketInfo impl MarketInfo
└── (optional, as needed)
    ├── wire.rs         # wire-format helpers
    ├── nonce.rs        # replay protection
    ├── session.rs      # ephemeral sessions
    ├── liq_price.rs    # client-side liq formula
    └── funding_source.rs  # impl FundingSource

Register in bot/src/exchange/mod.rs: pub mod newex;.

2.6 Symbol mapping

bot/src/exchange/newex/symbols.rs:

pub fn to_canonical(exchange_native: &str) -> &str;     // "BTCUSD" → "BTC"
pub fn from_canonical(canonical: &str) -> String;       // "BTC" → "BTCUSD"

Existing patterns: Lighter is identity (BTC → BTC); Extended strips/adds -USD; Zero1 strips/adds USD.

2.7 Signing (if needed)

bot/src/signing/newex.rs, registered in bot/src/signing/mod.rs. Examples: lighter.rs (ECgFp5 Schnorr), stark.rs (Starknet ECDSA, Extended), zero1.rs (Ed25519). Capture one real signed transaction from Phase 1 and hard-code it as a test vector.

Signing pitfalls (from the Hyperliquid integration):

  1. Serialization key order defines the signing hash. If the exchange hashes a msgpack/JSON action before signing, key order is critical. serde_json::Value (a BTreeMap) sorts alphabetically — which won't match a Python SDK's insertion order. Use typed structs with #[derive(Serialize)], where struct field order is key order.
  2. Pin byte-exact tests. For every action type (limit order, trigger order, cancel, leverage), a unit test comparing the Rust serialization hex against the Python SDK's. One mismatched byte → a different hash → a different recovered address → "User does not exist."
  3. Nested-struct field order matters too. TriggerSpec { triggerPx, isMarket } ≠ { isMarket, triggerPx } — different msgpack even when the JSON looks the same.
  4. Wire string format: trailing zeros, decimal precision, integer vs float ("0.00030" ≠ "0.0003", "74218.0" ≠ "74218") — every character affects the hash. Always go through float_to_wire() + price_to_wire(price, decimals).
  5. Nonce uniqueness: current_unix_ms() isn't unique under parallel sends. Use an AtomicU64 counter initialized from ms at startup.

2.8 MarketInfo

bot/src/exchange/newex/market_info.rs:

use crate::exchange::traits::{MarketInfo, decimal_qty_step, round_qty_down_decimals};

pub struct NewexMarketInfo {
    pub size_decimals: u32,
    pub price_decimals: u32,
    pub max_leverage: u16,
    /// Qty floor in BASE UNITS — the exchange's true published rule.
    /// Must NOT be computed from startup mark × USD floor (see §2.8.1).
    pub min_order_size: Decimal,
    /// USD notional floor if the exchange enforces one at submit time
    /// in addition to `min_order_size`. `None` when the venue's only
    /// size rule is qty-based. See §2.8.1 / §1.6 for how to detect.
    pub min_notional_usd: Option<Decimal>,
    // exchange-specific fields ...
}

impl MarketInfo for NewexMarketInfo {
    fn size_decimals(&self) -> u32 { self.size_decimals }
    fn price_decimals(&self) -> u32 { self.price_decimals }
    fn qty_step(&self) -> Decimal { decimal_qty_step(self.size_decimals) }
    fn round_qty_down(&self, qty: Decimal) -> Decimal { round_qty_down_decimals(qty, self.size_decimals) }
    fn max_leverage(&self) -> u16 { self.max_leverage }
    fn min_order_size(&self) -> Decimal { self.min_order_size }
    fn min_notional_usd(&self) -> Option<Decimal> { self.min_notional_usd }
}

Shared helpers from exchange/traits.rs: decimal_qty_step(size_decimals) (10^(-n), covers Lighter/Zero1/Hyperliquid); round_qty_down_decimals(qty, size_decimals) (truncate to n places); effective_min_qty_for_leg(info, leg_price) — read the sizing hot path through this, not through min_order_size directly, because it combines the qty floor and the notional floor at the live price. Override qty_step only when it comes from the API as a non-power-of-10 (like Extended).

2.8.1 Min-order rules — get this right

Exchanges have up to two independent floors, checked at submit: a qty floor (min_order_size in base units, e.g. BTC = 0.0001, MEGA = 1) and a notional floor (a USD floor on qty × price; HL = $10, Lighter = min_quote_amount ≈ $10 per perp). An exchange may have only one, both, or neither — check both, empirically via rejection messages (§1.6). The current matrix:

Exchange min_order_size source min_notional_usd() Note
Hyperliquid ceil($10 / startup_mark) — not recommended without a backup Some($10) $10 is hardcoded, not in the API; empirically verified
Lighter min_base_amount from /api/v1/orderBooks Some(min_quote_amount) from the same endpoint Both fields per symbol; MKR has 0 → None
Extended minOrderSize from trading-info None Qty floor only; maxOrderValue/maxPositionValue exist but are upper bounds
Zero1 qty_step() (no explicit floor in the API) None No notional rule in the proto

The anti-pattern (which once cost a one-legged fill): don't cache min_order_size = ceil(min_notional / mark) from the startup mark as your only guard. The mark drifts down, the stale cache passes a qty that now violates the exchange's real notional rule → a one-legged fill. The right pattern: if the exchange publishes literal qty/notional fields (Lighter-style), parse them straight into MarketInfo; if it publishes only a notional rule (HL-style), return it from min_notional_usd() as a const and keep min_order_size() at ZERO or a conservative floor (never derived-from-mark). Either way, effective_min_qty_for_leg(info, leg_price) — read by sizing in open.rs and executor.rs::check_unblock_symbols — recomputes both at the live price.

2.8.2 Runtime unblock

When sizing blocks a symbol via BlockType::QtyMin, it's released by a reactive hook in on_market_update: on every BBO tick for (exchange, symbol), the bot checks for such a block and recomputes effective_min_qty_for_leg at the new price; if potential_qty ≥ min_qty, the block lifts. A second path — hot-reloading max_order_size_usd — covers the case where the operator raised the target. The MarketInfo impl does nothing extra — a new exchange gets this for free once it fills both trait methods correctly.

Gate: cargo check passes. Signing unit tests green. The module is registered.


Phase 3 — REST client

Goal: every REST method works and smoke tests pass.

WS-first: the REST client implements only operations that aren't available over WS (see §4.3.1). If Phase 1 (§1.1) showed that place/cancel/modify orders are available over WS, do not implement them in REST. REST is for startup queries (symbols, fees, market meta) and fallback/verify operations; write operations go over WS in Phase 4.

3.1 REST client struct

bot/src/exchange/newex/rest.rs — typed response structs + HTTP methods, all URLs from parameters (NewexConfig), never hardcoded. Two paths depending on call frequency:

// 1. Pooled client — frequent calls (orders, balance, fees):
use crate::utils::http::{build_https_connector, ResilientHttpClient};

pub struct NewexRest {
    base_url: String,
    http: ResilientHttpClient,  // retry-on-transient, connection pool
}

impl NewexRest {
    pub fn new(base_url: impl Into<String>) -> Result<Self> {
        let https = build_https_connector()?;
        let inner = hyper_util::client::legacy::Client::builder(
            hyper_util::rt::TokioExecutor::new(),
        ).build(https);
        let headers = ...;
        Ok(Self { base_url: base_url.into(), http: ResilientHttpClient::new(inner, headers) })
    }
}
// 2. Raw TLS — rare one-shot calls (startup-only):
use crate::exchange::net::{http_get, http_post, build_tls_connector};

pub struct NewexRest {
    rest_url: String,
    tls: TlsConnector,
}

3.2 Factory preflight methods

All three use &self.rest (the shared REST client created in from_config()). Don't create a new client inside each method — that's a burst of TLS handshakes.

  • fetch_symbols() — self.rest.info(), parse markets, build Vec<(String, NewexMarketInfo)>.
  • fetch_taker_fee() — self.rest.user_fees(), prefer the account-specific/VIP rate, cache into self.cached_taker_fee, and log the number.
  • fetch_market_meta() — for qty_step/min_qty strings use crate::exchange::factory::qty_step_from_decimals(size_decimals), don't duplicate the formula.

3.3 Client REST methods

Method What it does Critical notes
fetch_equity() Account balance —
fetch_open_positions() Open positions Account for the /positions lag from Phase 1
check_order_status() Order status by ID Critical: must return real filled_qty + avg_fill_price if the order executed, Some(filled_qty=0) if cancelled, None if not yet settled — never a stub. It's the REST fallback in fill_fut_1/fill_fut_2 and in clean_close_done after a WS confirmation_timeout. Without it, a late WS/account fill reads as "expired," the bot unwinds the peer leg, and you get a naked orphan. Live-test it by artificially delaying WS handling and checking REST recovery.
fetch_liquidation_price() Liquidation price Engine value or client-side compute
fetch_trade_fee() Per-order fee post-fill Critical: if there's no per-order endpoint, return None — never Some((rate, true)) (that's a rate-vs-USD bug; see below)

The fetch_trade_fee trap: with a per-order fee endpoint (Extended /trades), return Some((fee_usd, is_taker)); without one (Lighter, Zero1), return None — the fill-event path already wrote notional × taker_rate, and None leaves that value in place. Returning Some((taker_rate, true)) instead writes the rate as a USD amount and undercounts fees by roughly the notional — a real incident.

3.4 Fixture tests

Capture real responses as fixture files for the Phase 4 parsers: poke (REST reachable, auth works, fees correct), session_probe (session lifecycle, if any), trade_probe (a signed PlaceOrder accepted — a no-fill IOC at a far price).

Gate: all REST smoke tests pass. Fixture files committed.


Phase 4 — WebSocket

Goal: OB ticks reach the executor, fills route through PendingOrders, external closes are detected.

4.1 WS supervisors — the shared pattern

Every WS connection goes through WsReconnect::run() — don't hand-roll a reconnect loop.

use crate::exchange::ws_reconnect::{WsReconnect, WsReconnectConfig};

pub async fn run_newex_ob_ws(..., health: Arc<ExchangeHealth>) {
    let session_health = health.clone();
    WsReconnect::run(
        WsReconnectConfig::new("newex OB", exchange_id, HealthChannel::WsOrderbook)
        // Defaults: 500ms backoff, 30s max, default_classify, health=true, skip_clean=false.
        // To customize, uncomment and add a leading dot:
        //  .initial_backoff(Duration::from_secs(2))          // slow reconnect
        //  .classify_error(classify_newex_maintenance)        // CDN/WAF detection
        //  .no_health_on_disconnect()                         // for per-market supervisors
        //  .skip_sleep_on_clean()                             // immediate reconnect on Close
        ,
        health,
        || market_tx.is_closed(),
        || {
            let h = session_health.clone();
            async move { run_ob_session(..., h).await }
        },
    ).await;
}

Exception: an account WS holding borrowed refs (&HashMap, &DashMap) uses a manual begin() / on_disconnect() loop. Carry state across reconnects in an Arc<AtomicBool>, not a plain bool (Copy → copied into the async move).

4.2 Config-driven decisions (from Phase 1 data)

Phase 1 finding → Config/code decision
Camp A (server pings, no client ping) WriteTaskMode::RelayOnly
Camp B (client must ping) WriteTaskMode::TextPingRelay or WsPingRelay
Thin market idle >30s ws_read_timeout_secs > idle gap
Subscription cap per WS MAX_MARKETS_PER_WS for the batch supervisor
Absolute-replace deltas DeltaMode::AbsoluteReplace, no REST snapshot
Incremental deltas DeltaMode::Incremental, REST snapshot first
Solo tighter than shared Support ws_ob_mode = per_market

4.3 Orderbook WS

The session function inside the supervisor:

  1. Connect via connect_wss_timeout with read_timeout_secs from config.
  2. Subscribe + initial snapshot (REST if incremental, the first WS frame if absolute-replace).
  3. health.set_channel(eid, WsOrderbook, true).
  4. Spawn the write task: spawn_write_task(ws_write, rx, mode, label).
  5. Spawn the read task in AbortOnDrop with make_send_fn() for pong relay.
  6. Spawn the freshness watchdog: spawn_stale_data_watchdog (threshold WS_STALE_RECONNECT_MS = 30s).
  7. join_read_task() — await.
  8. Abort the write task, return.

BestLevels needs both timestamps:

Field Value Meaning
ts_ns now_ns() at parse time Local recv time → Recv→Sig
exchange_ts_ns Venue event time if any, else now_ns() (= ts_ns) Venue time → Tick→Recv

Never set exchange_ts_ns = 0 — it produces nonsense in the UI. If the venue gives no event time, set exchange_ts_ns = ts_ns and the UI shows —.

Call health.mark_data() after every processed delta, or a data_stale halt fires after 30s:

health.mark_data(exchange_id.db_id(), HealthChannel::WsOrderbook);

Gap detection: track last_update_id / begin_seq; on a SequenceGap, return Err so the supervisor reconnects.

4.3.1 WS-first — prefer WS over REST for everything

If the exchange supports an operation over WebSocket, use WS — for reads and writes alike. Use REST only for what WS can't do.

A. Writes (orders, cancels, SL) over WS. REST writes carry HTTP overhead (headers, TLS framing, possibly a new connection) → +5–50ms per order; a WS write is a frame into an already-open socket → <1ms. An arbitrage entry is two parallel orders, so REST doubles the HTTP overhead, and on exit latency is slippage is money. WS write operations also often have a separate, more generous rate limit.

// The account WS connection is used for:
//   1. receiving fills (read) — already there
//   2. sending orders (write) — through the write-task tx channel
//   3. cancelling orders (write) — the same channel
//
// In ExchangeClient::place_order():
//   1. serialize the order into the WS action format
//   2. send it through ws_write_tx (the channel to the write task)
//   3. await the fill through PendingOrders (as usual)
//
// REST fallback ONLY if the WS is disconnected at send time.

B. Reads — WS push over REST poll. Position state, funding payments, balance updates, and order-status changes are often pushed over WS at the moment of the event — no 200–500ms poll latency and no rate-limit consumption. For SL movement (which depends on fetch_liquidation_price), the difference between a 60s REST poll and an instant WS push is the difference between "the SL moved in time" and a liquidation.

C. What to use when:

Operation WS available? Use
Place / cancel order, place/modify SL Yes WS (REST = fallback on disconnect)
Set leverage Yes WS (or REST at startup — one-time)
Position / funding / balance Yes (push) WS push (REST = fallback/verify)
Market info / symbols, fee rate, historical trades No (startup / history) REST

4.4 Account WS

Two jobs beyond fill routing.

A. Fill routing through PendingOrders:

let pending_orders = Arc::new(PendingOrders::<String>::new()); // or <u64> or <u64, Side>

// place_order: register
let (_tx, rx) = self.pending_orders.register(order_id, ());

// WS fill handler: resolve
if let Some((meta, tx)) = pending_orders.resolve(&order_id) {
    tx.send(Ok(fill));
}

// cancel/reject
pending_orders.reject(&order_id, "cancelled".into());

// periodic cleanup
pending_orders.cleanup_closed();

B. Position updates + external-close detection. The PositionEventSink contract is the only path for position updates and external leg closes. Writing to the DashMap cache for the fetch_liquidation_price fast path is fine, but update_position() and add_funding() must be called through the sink on every position-update frame — otherwise the executor never sees the update, funding is 0 at close (the data sat in the DB, not the cache), and the SL monitor never gets liq-price drift. (Lesson: on Hyperliquid the DashMap cache updated but the sink calls were skipped, so the executor saw stale state and funding was lost at close.)

// on EVERY position-update frame:
position_events.update_position(exchange_id, symbol_id, liq_price, unrealized_pnl, avg_entry, margin, now_ms);
// on a funding settlement:
position_events.add_funding(exchange_id, symbol_id, change);
// on an EXTERNAL close (SL fire / liquidation / manual close from the exchange UI):
position_events.report_external_close(exchange_id, symbol_id, reason, filled_qty, fill_price, source_order_id);

ExternalCloseReason: StopLoss (our SL fired on the exchange), Liquidation (the exchange force-closed), ExternalManual (everything else — manual close, ADL). If the exchange doesn't distinguish reasons (like Zero1), always use ExternalManual — it's more honest than silence.

Forbidden: a direct leg_closed_tx.send(...) (a grep for leg_closed_tx\.send under bot/src/exchange/newex/ must return 0), and a direct write to the exchange_positions DashMap (only via position_events.update_position()).

4.5 Public-trades WS

PublicTrade::side is the taker/aggressor (who crossed the spread). If the exchange reports the maker side, invert it (maker bid → taker Sell, maker ask → taker Buy). Timestamps: with no engine timestamp, ts_ns = exchange_ts_ns = now_ns().

4.6 WS fixture tests

ws_sniff_ob (OB WS connects, deltas arrive), ws_sniff_account (account WS connects, auth works), e2e_single_trade (real fill + position update + liq price).

Gate: ticks flow into market_tx, fills route, external closes are detected. Fixture files committed.


Phase 5 — Factory + wiring + clean_close_done

Goal: every file outside exchange/newex/ is edited and cargo test --workspace --lib is green.

5.1 Factory implementation

bot/src/exchange/newex/factory.rs:

pub struct NewexFactory {
    config: NewexConfig,
    /// Shared REST client — created once in from_config(), reused by all
    /// preflight methods. Avoids burst TLS handshakes (12→4 simultaneous).
    rest: NewexRest,
    cached_market_data: OnceLock<Vec<MarketData>>,
    cached_taker_fee: OnceLock<Decimal>,
}

impl ExchangeFactory for NewexFactory {
    fn name(&self) -> &str { "newex" }
    fn fetch_symbols(&self) -> Pin<Box<...>>;    // uses &self.rest
    fn fetch_taker_fee(&self) -> Pin<Box<...>>;  // uses &self.rest, caches into self.cached_taker_fee
    fn build(self: Box<Self>, exchange_id, ctx) -> Pin<Box<...>> {
        Box::pin(async move {
            // 1. Load keys, verify account
            // 2. Build REST client, fetch market info
            // 3. Create session (if ephemeral sessions)
            // 4. Spawn WS supervisors (OB + Account + Trades)
            // 5. Build NewexClient
            // 6. Wire clean_close_done (§5.2)
            // 7. Wire the ready future (§5.1.1)
            // 8. Wire FundingSource (optional)
            Ok(BuiltExchange { client, fee_rate, ready, clean_close_done, funding_source })
        })
    }
}

5.1.1 The ready future — don't block startup

BuiltExchange::ready is the future main.rs awaits before "all systems online." Don't put a blocking poll loop that waits for data from every symbol. ready should be instant (Box::pin(async { Ok(()) })) when the OB/BBO channel is event-driven, there's no guaranteed snapshot on subscribe, and thin markets can go quiet for minutes — the spread detector's max_recv_staleness gate already blocks entries on a symbol with no fresh data, so a blocking wait here just duplicates that protection and stalls every exchange's startup for one thin market. A blocking wait is only appropriate when the exchange guarantees a snapshot on subscribe (Lighter: the first WS frame is a full snapshot) and ready is awaiting a specific handshake, not arbitrary data frames.

5.1.2 Startup REST calls — don't block sequentially

Any per-symbol REST loop at startup (leverage, margin mode, …) must be parallel via JoinSet, not a sequential for ... await. Sequential is N × RTT (98 symbols × 100ms = 10s from Tokyo, ×500ms = 50s trans-Pacific); parallel is ~max(RTT) regardless of count.

// ✗ sequential — 98 × 100ms = 10s from Tokyo
for mi in markets {
    client.update_leverage(mi.asset_id, leverage, false).await;
}

// ✓ parallel — ~100ms total
let mut set = tokio::task::JoinSet::new();
for mi in markets {
    let c = client.clone();
    set.spawn(async move { c.update_leverage(mi.asset_id, leverage, false).await });
}
while set.join_next().await.is_some() {}

Before parallelizing, check the exchange's rate limits (Hyperliquid: 1200 weight/min, 98 requests = 98 weight — safe), the operation's idempotency (a leverage set is idempotent — safe to repeat), and whether it can be skipped (if a read endpoint exists, skip symbols already at the right value). Extended's set_leverage() is the pattern: parallel via JoinSet, blocking symbols whose max_leverage < config_leverage. (Lesson: 98 sequential updateLeverage calls were 50s of startup from a Mac, 10s from Tokyo; parallelized, 1–2s.)

5.2 clean_close_done — residual-position cleanup

Three rules. Breaking any one leaves directional exposure.

Rule 1 — wait for the account WS before the first close order. clean_close_done starts before ready, so the account WS may not be connected yet.

// Pattern A (Extended/Zero1): poll health
let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
    if health.is_channel_connected(eid, HealthChannel::WsAccount) { break; }
    if tokio::time::Instant::now() >= deadline {
        tracing::warn!("account WS not ready after 10s — REST fallback will catch fills");
        break;
    }
    tokio::time::sleep(Duration::from_millis(50)).await;
}
// Pattern B (Lighter): close inline inside the WS session task AFTER set_channel(WsAccount, true)

Rule 2 — REST fallback on a WS timeout. Don't trust rx.recv() alone; a WS fill event can be lost.

match tokio::time::timeout(Duration::from_secs(10), rx.recv()).await {
    Ok(Some(Ok(f))) if !f.is_empty() => { closed += 1; }
    _ => {
        // WS missed the fill — check REST
        tokio::time::sleep(Duration::from_secs(2)).await;
        let positions = client.fetch_open_positions().await?;
        if positions.iter().all(|p| p.market != market) {
            tracing::info!("position closed (recovered from REST)");
            closed += 1;
            continue;
        }
        tracing::warn!("close not filled");
    }
}

Rule 3 — hard-fail on a residual after retries. Never Ok(()) + error!("close manually") — an Ok(()) lets the bot open new positions on top of the residual.

for attempt in 1..=5u8 {
    let positions = client.fetch_open_positions().await?;
    if positions.is_empty() { break; }
    for pos in &positions {
        if let Err(e) = client.close_position(&pos.market, pos.size, &pos.side).await {
            tracing::warn!(%e, attempt, "close_position errored, will re-verify");
        }
    }
    if attempt < 5 { tokio::time::sleep(Duration::from_secs(2)).await; }
}
let final_positions = client.fetch_open_positions().await?;
if !final_positions.is_empty() {
    bail!("cannot start with {} residual positions on newex", final_positions.len());
}

5.3 ExchangeClient — required-methods checklist

The compiler forces you to implement every required method — but don't leave stubs:

Method Critical notes
place_order() Soft errors ("no fills" as an error code) → translate to Fill { filled_qty: 0 }. Transient errors → retry after 2s
place_stop_loss() Not a stub! Called after every entry — a stub means "position UNPROTECTED"
close_position() Not Ok(())! A clean restart calls it for residuals. Pattern: IOC reduce-only at mark ± limit_price_cap
fetch_liquidation_price() Called every 60s — without it the SL never moves
limit_price_cap() Critical. None only if the exchange truly has no limit; otherwise Some(0.10) for 10%. With no cap, SlippagePct::Max = 35% and every order is rejected — invisible in tests priced near mark!

Default methods — review each against Lighter/Extended/Zero1: modify_stop_loss (default cancel+place, 3× retry; override if there's an atomic-modify API), config_schema (default empty — always override, a blank panel is confusing), refresh_funding_now (default no-op; override if the WS doesn't push cumulative funding), forget_pending_order (default no-op; override if the key type matches the executor's REST-recovery path).

5.4 Wiring — every file outside exchange/newex/

Do these in order — skipping a REQUIRED row means the exchange won't load or its credentials won't validate.

# Required? File What to add
1 REQUIRED common/src/constants.rs NEWEX_REST_URL, NEWEX_WS_URL (done in Phase 2)
2 REQUIRED bot/src/config.rs NewexConfig + Config::newex() (done in Phase 2)
3 REQUIRED bot/src/exchange/mod.rs pub mod newex; (done in Phase 2)
4 REQUIRED bot/src/exchange/factory.rs The match arm in build_factories(): "newex" => Box::new(NewexFactory::from_config(config)?)
5 REQUIRED bot/src/web/credentials.rs NewexCredentials struct + impl + Box::new(NewexCredentials) in registry() + a validate_newex() fn + update the assert_eq!(s.len(), N) test
6 RECOMMENDED bot/src/storage/sqlite.rs Seed a stable id in Storage::open: INSERT OR IGNORE INTO exchanges (id, name, display_name, exchange_type, ...) VALUES (N, 'newex', 'NewEx', 'dex_offchain', ?1)
7 optional config/default.toml [exchanges.newex] (commented out)
8 optional .env NEWEX_* secrets
9 optional bot/src/main.rs Add "newex" to TICK_LATENCY_COUNT_ONLY only if the exchange doesn't publish matching-engine timestamps
10 optional bot/src/web/server.rs clamp_exchange_slippage — only if there's a hard protocol-level max slippage
11 optional bot/tests/common/mod.rs TestUrls + newex_helpers.rs

Credentials implementation (#5):

// bot/src/web/credentials.rs
struct NewexCredentials;

impl ExchangeCredentials for NewexCredentials {
    fn schema(&self) -> ExchangeCredentialSchema {
        ExchangeCredentialSchema {
            exchange_id: "newex",
            display_name: "NewEx",
            fields: vec![
                f("API_KEY", "NEWEX_API_KEY", "API Key", FieldKind::Secret, true),
                // ... per .env field
            ],
        }
    }
    fn validate(&self, env_path, overrides) -> Pin<Box<...>> {
        Box::pin(async move { validate_newex(&env_path, overrides).await })
    }
}

fn registry() -> Vec<Box<dyn ExchangeCredentials>> {
    vec![
        Box::new(LighterCredentials),
        Box::new(ExtendedCredentials),
        Box::new(Zero1Credentials),
        Box::new(NewexCredentials),   // ← new
    ]
}

validate_newex(): read secrets from .env via read_env_secrets_with_overrides, shape-check the required fields, build a throw-away Config with the candidate secrets, run fetch_taker_fee() (which exercises the full signing + auth), and return ValidationOk { summary: "taker fee ...", resolved }.

5.5 Market metadata (fetch_market_meta)

Implement fetch_market_meta() in NewexFactory (a required trait method). The data is auto-upserted into exchange_symbols at startup and used by leverage blocking and the backtester's SL distance.

What to find in the API:

Field Why How to find it
max_leverage Block symbols when config leverage > max maxLeverage, max_leverage, or from IMF: max_leverage = 1/IMF
MMF (maintenance margin fraction) Backtester SL distance: liq_dist = max(1/lev, 1/max_lev) − mmf mmf, maintenance_margin, maintenanceMargin, riskFactor — the formula is exchange-specific (below)
size_decimals, price_decimals Qty/price rounding Usually in the market-info endpoint
qty_step, min_qty Min-order validation Usually in the market-info endpoint
limit_price_cap Max price deviation from mark Not on every exchange

Verify with curl before writing code — confirm the field exists in the response (don't guess from someone else's code), the type is right (string vs number vs basis points), and the units are what you think (fraction 0.02 vs percent 2% vs basis points 200):

curl -s 'https://api.newex.com/markets' | python3 -c "
import sys, json; data = json.load(sys.stdin)
m = data['markets'][0]
for k,v in sorted(m.items()): print(f'  {k}: {type(v).__name__} = {v}')
"

MMF patterns (don't copy a formula from another exchange — each defines MMF its own way; verify in the docs):

Pattern Example How to get MMF
Explicit mmf field Zero1: mmf: 0.01 mmf directly
Explicit field in bps Lighter: maintenance_margin_fraction: 120 value / 10000 = 0.012
riskFactor = MMR Extended/StarkEx: riskFactor: "0.02" riskFactor directly (do not halve it)
IMF/2 heuristic Hyperliquid: maxLeverage: 40 0.5 / maxLeverage (per the docs)
Not in the API — None → operator override via exchange_symbol_overrides

Max-leverage patterns: explicit field (maxLeverage: "50.00"); from IMF (floor(1 / imf)); from IMF in bps (10000 / value); or absent (0 = unknown, leverage blocking skips it). Per-exchange details: EXCHANGE_NOTES.md → "Margin / Leverage / MMF."

fn fetch_market_meta(&self) -> Pin<Box<dyn Future<Output = Result<Vec<MarketMeta>>> + Send + '_>> {
    Box::pin(async move {
        let data = rest.markets().await?;
        let mut out = Vec::new();
        for m in &data {
            out.push(MarketMeta {
                symbol: canonical_name(&m.symbol),
                market_name: m.symbol.clone(),
                size_decimals: m.size_decimals,
                price_decimals: m.price_decimals,
                qty_step: format_step(m.size_decimals),
                min_qty: m.min_order_size.clone(),
                max_leverage: compute_max_leverage(&m),  // from API docs
                mmf: compute_mmf(&m),                     // from API docs
                limit_price_cap: m.limit_price_cap,
            });
        }
        Ok(out)
    })
}

Shared REST client + caching. The factory creates the REST client once in from_config() and keeps it as a field; all three preflight methods use &self.rest. Otherwise each method opens its own client → 3 simultaneous TLS handshakes per exchange × 4 exchanges = 12 burst handshakes, and a server with a small accept queue (Zero1) rejects with "tls handshake eof." fetch_market_meta() and fetch_taker_fee() cache into OnceLock fields that build() drains (with a fresh-fetch fallback), eliminating duplicate REST calls between preflight and build. All four current exchanges do both.

5.6 Post-implementation tests

liq_price_live (fetch_liquidation_price vs engine or a second source, across leverages) and sl_move_sim (place SL → modify → verify via REST after each step).

5.7 Files not to touch

File Why
main.rs (except TICK_LATENCY_COUNT_ONLY) Iterates factories generically
spread_detector.rs C(N, 2) pairs automatically
execution/{open,close,sl_monitor}.rs Dispatch via ctx.client(eid)
position_events.rs One façade; a new exchange plugs in via BuildContext
executor.rs::on_external_leg_close Symmetric close works for any exchange
common/src/sim.rs, backtest.rs, optimizer/ Sim hot loop generic; Vec<ExchangeSimSpec>; build_sim_params takes &[&str]
Web UI (backtest/optimizer multi-select) Automatic

Gate: cargo test --workspace --lib + cargo test --test apply_overrides green.


Phase 6 — Live validation (on a real account)

Unit tests don't replace live validation. Signing, price precision, min notional, keepalive timing, fill routing — these break only on real money. Requirements: a real account with a small deposit ($15–30) and credentials in .env. Run the live tests as soon as Phase 5 compiles — don't defer them.

The minimum set (all required), each of which must leave the account clean (0 positions, 0 orders); if a test panics, use close_all_positions to recover:

# Test What it checks Cost
1 credentials_probe Signing key → address, fee fetch, clearinghouse $0
2 trade_probe_no_fill_ioc Signing + submission + rejection handling $0
3 e2e_fill_round_trip Open + fill + position + liq price + close ~$0.01 fees
4 liq_price_live fetch_liquidation_price vs engine, ratio sanity ~$0.01
5 sl_move_sim Place SL → modify → verify via REST → cancel → close ~$0.01
6 ws_fill_routing WS fill arrives, cloid round-trip, PendingOrders routing ~$0.01
7 close_all_positions Emergency-cleanup utility $0

Three bugs are invisible in unit tests — only live validation catches them.

6.1 clean_close_done restart test

  1. Open a real position (a one-off e2e_open_only test, or the bot + SIGTERM).
  2. Confirm it's open in the exchange UI.
  3. Restart in clean mode (tools/bot_start.sh with startup_mode = "clean").
  4. Required in the logs: Verified: 0 open positions exchange=newex; Clean close done; no WARN close not filled without a following recovered from REST.
  5. Negative test: force a residual (stub limit_price_cap to something impossible) → restart → the bot exits with Err, it doesn't keep trading.

6.2 External-close detection test

  1. Point the bot at one symbol shared with an existing exchange.
  2. Wait for POSITION OPENED on both legs.
  3. Manually close only the newex leg from the exchange UI.
  4. Required in the logs: ExternalLegClose exchange=newex reason=external_manual; the executor closes the paired leg on the other exchange; both legs go to zero and fetch_open_positions is empty for both.
  5. Static check: grep "leg_closed_tx\.send" bot/src/exchange/newex/ = 0 matches.

6.3 limit_price_cap + thin-market test

  1. Find the exchange's thinnest symbol.
  2. Set entry_slippage_pct = "max".
  3. A small entry order is filled, not rejected with a price-band error.
  4. Negative: stub limit_price_cap() → None → the order is rejected → revert.

Gate: all three tests pass. Update EXCHANGE_CANDIDATES.md → "code-complete."


Reference

Common pitfalls

Symptom Cause Fix
"unknown exchange 'newex'" Missing arm in build_factories() Add the arm
schemas_cover_all_configured_factories fails Missing NewexCredentials in registry() Add the struct + impl
"position UNPROTECTED" place_stop_loss is a stub Implement it
Clean restart leaves positions close_position is Ok(()) IOC reduce-only
SL never moves fetch_liquidation_price returns None Implement it
Every real order rejected limit_price_cap() = None → 35% slippage Override with a real cap
WS hangs forever connect_wss without a timeout connect_wss_timeout + read_timeout_secs
Funding = 0 at close The WS doesn't push funding Wire refresh_funding_now
Leg closed externally, paired leg hangs No report_external_close in the account WS Wire detection (§4.4)
Direction leak Direct write to the exchange_positions DashMap Use position_events.update_position()
fetch_trade_fee returns rate instead of USD Some((rate, true)) instead of None Return None if there's no per-order endpoint
URLs break when the exchange changes domain Hardcoded const in the exchange module URLs from arb_common::constants only
apply_n3 test fails Stable-id seed shifted autoincrement Update the test assertions

Pitfalls discovered during the Hyperliquid integration

Real bugs from a production integration. Each cost 30+ minutes of debugging — and each is now a rule above.

Pitfall How it showed Root cause Guard
msgpack key order "User does not exist" on every order serde_json::Value sorts keys alphabetically → different msgpack → different keccak hash → the server recovers the wrong address Typed structs with #[derive(Serialize)] (field order = key order); pinned byte-exact tests vs the Python SDK
msgpack field order in nested structs SL trigger orders "User does not exist" despite a correct top-level msgpack TriggerSpec { triggerPx, isMarket, tpsl } field order ≠ the SDK's { isMarket, triggerPx, tpsl } A pinned test for every order type (limit + trigger); compare hex byte for byte
Trailing zeros in wire strings "0.00030" ≠ "0.0003" → different msgpack format!("{:.5}") keeps a trailing zero; the SDK strips it Always float_to_wire() — strip trailing zeros
Guessing WS keepalive Account WS dropped every ~50s Assumed Camp A (server pings); really Camp B (client pings every 60s) Read the docs before coding, not probes
WS connection limits Wrote a per-market WS (140+ connections) Didn't check the limit (10 max) Phase 1: check the connection limit before the architecture decision
Price-precision rejection "Order has invalid price" on every order BTC price with 2 decimals; the exchange expects 1 or an integer price_to_wire(price, decimals) — round before signing
Min notional undocumented "Order must have minimum value of $10" Not documented, only empirical Phase 1 or Phase 6: a test order at the minimum
position_events bypass Executor never saw position updates Wrote to the DashMap cache but not through PositionEventSink.update_position() Audit: grep position_events.update_position in the new module
Funding bypass funding_1/2 = 0 at close Didn't call position_events.add_funding() Audit: grep add_funding
f64→Decimal liq-price drift Phantom SL-movement triggers Decimal::try_from(74218.3_f64) = 74218.2999... Keep liq price as a String; parse with Decimal::from_str
Nonce collision Parallel orders with the same timestamp_ms current_unix_ms() isn't unique under concurrent sends An AtomicU64 counter, initialized from ms, incremented per action
Agent vs account address "User does not exist" on REST queries The agent-wallet address ≠ the main-account address; info queries need the account address Clear naming: API_PRIVATE_KEY, API_WALLET_ADDRESS, ACCOUNT_ADDRESS
REST instead of WS for orders +5–50ms per order, a full write-path rewrite Didn't check in Phase 1 which write operations were on WS Phase 1 §1.1: check WS write capabilities first — orders over WS reuse the connection at <1ms overhead
WS response latency > confirmation_timeout Parallel entry: Lighter filled, the HL IOC marked status=expired ms=0 → Lighter unwound. The HL fill actually arrived over the WS account feed 200ms after the timeout → a naked HL short, and the Lighter unwind went off at 225 bps of slip fill_fut_2 had no REST fallback (unlike fill_fut_1); when HL's WS response ran >1.5s and the account fill >1.7s, the bot timed out and called the order expired though it had actually executed — an asymmetry between leg 1 and leg 2 recovery (1) check_order_status is mandatory, not a stub (§5.3). (2) fill_fut_2's REST fallback is now symmetric with fill_fut_1 — after a WS timeout with qty=0 && error=None, go to check_order_status before unwinding the peer leg. (3) When adding an exchange, run a live test with the WS response artificially delayed and confirm the REST fallback fires and no orphan is created

What works automatically

After the N-exchange generalization, these need no changes: main.rs (iterates factories generically), spread_detector.rs (C(N, 2) pairs), execution/{open,close,sl_monitor}.rs (dispatch via ctx.client(eid)), position_events.rs (façade), executor.rs::on_external_leg_close (symmetric close), common/src/sim.rs + backtest.rs + optimizer/, the web UI (backtest/optimizer multi-select, apply best params), the tick writer + Parquet partitioning, and the symbol_overrides schema.