How to use this playbook. Work through the phases in order — each gate is a check to pass before moving on, not a rough guide to skim. Turn every phase, sub-step, gate, and test into a checklist and tick them off; a skipped step is a bug found in review instead of one prevented up front. For any WebSocket keepalive/ping/timeout question, read the exchange's docs first, then write code. Every design decision here — tick feed, delta mode, ping strategy, liq-price source — is made from the Phase 1 empirical data plus the exchange's documentation, never guessed.
The bot, backtester, and optimizer are all generic over N exchanges. An exchange implements:
ExchangeFactory(exchange/factory.rs) — bootstrap:fetch_symbols,fetch_taker_fee,build()→BuiltExchange.ExchangeClient(exchange/traits.rs) — runtime:place_order,cancel_order,place_stop_loss,close_position,fetch_liquidation_price, …FundingSource(exchange/factory.rs) — optional: the public funding-rate API.
The backtester and optimizer need no new code — they run on ExchangeSimSpec, which is data-only.
| Phase | What you do | Artifact | Code in exchange/newex/? |
|---|---|---|---|
| 1 | Empirically explore the API | An EXCHANGE_NOTES.md entry + probe tests |
No |
| 2 | Config + module skeleton + symbols + signing | Compiles; signing tests pass | Skeleton |
| 3 | REST client | REST smoke tests pass | rest.rs |
| 4 | WebSocket (OB + Account + Trades) | Ticks flow; fills route | ws.rs |
| 5 | Factory + wiring + clean_close_done | cargo test --workspace --lib green |
Complete |
| 6 | Live validation | 3 high-risk tests pass | Final |
Goal: gather every empirical fact that Phases 2–5 depend on. Write no code under bot/src/exchange/newex/. Probes are disposable — curl, scratch scripts, #[ignore] tests in bot/tests/exchanges/newex/.
Before the first curl:
DESIGN.md§2 (trade lifecycle), §5 (order execution + leg risk), §12 (leverage/margin/SL/liquidation).ARCHITECTURE.md— "Exchange abstraction" + theclean_close_donecontract (three invariants).EXCHANGE_NOTES.md— the entry format and the empirical data for the existing venues.- The existing implementations — open
bot/src/exchange/{lighter,extended,zero1}/side by side as reference.
Before writing any WS/REST probe, find and read the official docs:
WebSearch "exchange_name websocket ping heartbeat timeout"
WebSearch "exchange_name API rate limits connection limits"
WebSearch "exchange_name order types IOC trigger stop loss"
WebFetch "https://docs.exchange.com/api/websocket"
Record in EXCHANGE_NOTES.md before writing code: keepalive (who pings — server or client — the ping format, interval, timeout: Camp A/B/C), connection limits (max connections, max subscriptions per connection), rate limits (requests/min, weight per endpoint), order types (IOC/FOK/GTC, trigger orders, reduce-only), the signing scheme (HMAC / Ed25519 / EIP-712 / …), and the wire format (JSON / protobuf / msgpack, with examples from the docs).
A lesson from Hyperliquid: guessing Camp A (server pings) cost 30 minutes of bot restarts. The reality — Camp B (the client pings every 60s) — was stated plainly in the docs.
Before designing the REST client, check whether the exchange supports placing orders and other write operations over WebSocket:
WebSearch "exchange_name websocket place order cancel order"
WebSearch "exchange_name websocket API write operations"
Look for order placement, cancel, trigger/SL, modify, and leverage/margin set over WS. If the exchange offers the same write operation over both WS and REST, write an #[ignore] test that benchmarks both:
// bot/tests/exchanges/newex/ws_vs_rest_latency.rs
// 1. Prepare a signed IOC order at a deliberately unfillable price (no fill).
// 2. Send it N times over REST, measure round-trip (send → ack/reject).
// 3. Send it N times over WS, measure round-trip (send → ack/reject).
// 4. Report p50/p90/p99 for both, and the difference in ms and %.Record in EXCHANGE_NOTES.md: which write operations are available over WS, the WS action format (JSON-RPC, custom JSON, msgpack, …), whether the WS returns a response/ack with fill info, and the WS-vs-REST latency benchmark.
Why it's critical: REST means a new TCP/TLS connection or HTTP request overhead on every order; WS reuses a persistent connection at ~0ms overhead — an order of magnitude faster. For arbitrage, entry/exit latency is money. (Lesson: Hyperliquid supports place/cancel/modify/SL over WS, but the integration was written against REST — forcing a full rewrite of the write path.)
For each endpoint (markets, fees, positions, orders, trades, funding, balance):
# example — adapt to the exchange
curl -s -H "Authorization: ..." https://api.newex.com/v1/markets | jq '.' | head -50
# ALWAYS redact secrets (awk filter) before loggingRecord: the HTTP auth scheme (header, query param, signed body), the response shape and field types (number vs string), latency from a regional probe (Tokyo if that's the deploy target), and the /positions freshness lag (open a position → close it → measure how long REST takes to update).
Write a disposable probe with ws.set_auto_pong(false) (raw WebSocket) — FragmentCollector hides Ping frames.
| Property | How to check | Why |
|---|---|---|
| Who pings? | Sit idle 3+ min, log every opcode | Determines the write-task strategy |
| Ping interval | Time between Ping frames | For the keepalive timeout |
| Timeout with no pong | Don't answer a Ping, time the disconnect | For reconnect logic |
| Client ping allowed? | Send {"type":"ping"}, watch |
On Extended it kills the subscription! |
| Close-frame payload | Log the Close code + reason | For classify_error |
The three keepalive camps:
| Camp | Server pings? | Client must ping? | Example | WriteTaskMode |
|---|---|---|---|---|
| A | Yes | No, and mustn't | Extended, Zero1 | RelayOnly |
| B | No | Yes, required | Lighter | TextPingRelay or WsPingRelay |
| C | Yes | Allowed, not required | (hypothetical) | RelayOnly |
The most important decision. Don't commit to a channel without data.
1.4.a Inventory the channels. Check every source: the exchange's WS reference page, the official SDK source (@exchange/ts, exchange-python) for channels the docs omit, GitHub examples, the Discord/Telegram #api channel, and the exchange team directly. Look for a full L2 order book (orderbook, depth, book, deltas), top-of-book / BBO / L1 (bookTicker, ticker, bbo, quote, depth=1) — which can be 5–10× faster — public trades, undocumented depth parameters (?depth=N, ?level=1, ?compress=true), and combined streams.
1.4.b Live latency tests. For each candidate channel, an #[ignore] test in bot/tests/exchanges/newex/: connect, subscribe, hold 120+ seconds; per frame record Instant::now() + the engine timestamp if any; report frame count, rate/s, and inter-arrival gap p50/p90/p99/max; end with a === VERDICT === block naming the winning channel and by how many ms. Two key comparisons: full L2 vs BBO/ticker, and orderbook vs public-trades alignment (shared sequence number?).
1.4.c Solo vs shared topology. Two sockets side by side — SOLO (one subscription on the most active symbol) and SHARED (that symbol + N−1 others) — measured over 120+ seconds. If SOLO is tighter by >5ms p50, you need a per_market dedication option.
- Stagger: spawn N WS connections at varying intervals (100/250/500/1000ms); find the minimum that avoids a
429. - Subscription cap per connection: Zero1 = 10 (→ batch supervisor), Extended = unlimited.
- REST rate limits: a burst probe; record the real limit vs the documented one.
An exchange can reject a submit on two independent rules: a quantity floor (size < min_order_size) and/or a notional floor (size × price < min_notional_usd). Before writing MarketInfo (§2.8), find which your exchange has:
- Read the markets / trading-info endpoint. Look for fields rooted in
min/minimum— qty (minOrderSize,min_base_amount,minBaseSize,minQty,lot_size) and notional (min_quote_amount,minOrderValue,minNotional,minOrderValueQuote). - If a floor isn't published, probe it empirically. An
#[ignore]test that sends a deliberately-too-small IOC and logs the exact server error, repeated around the expected threshold. A real Hyperliquid rejection —"Order must have minimum value of $10. asset=217"— is how we learned it enforces a $10 notional with no explicit API field. - Both are independent. Even after finding one, check the other. Lighter publishes both
min_base_amountandmin_quote_amountper symbol; parsing only the notional one once let qty-based rejects reach production as orphans. - Record it in
EXCHANGE_NOTES.md— the exact field name, a typical value, the endpoint, and an example rejection message. Phase 2.8 carries this into the MarketInfo impl.
- Order-book delta mode: incremental (size = delta, needs a REST snapshot) or absolute-replace (size = final, the first WS frame bootstraps the book) — this decides
DeltaMode::IncrementalvsDeltaMode::AbsoluteReplace. - Thin-market idle gap: watch the thinnest market for 2+ min and record the largest gap between deltas — a lower bound for
ws_read_timeout_secs.
Open a tiny leveraged position and read the liquidation price over both REST and WS:
| Option | When | Implementation |
|---|---|---|
| Engine WS push | liquidation_price in the position-update frame |
Read from the WS cache, REST fallback |
| Engine REST only | No WS push, but REST returns it | REST poll in fetch_liquidation_price |
| Client-side compute | The engine returns nothing | liq_price.rs + REST for a fresh mark price |
Create bot/tests/exchanges/newex/main.rs and a [[test]] entry in bot/Cargo.toml:
[[test]]
name = "newex"
path = "tests/exchanges/newex/main.rs"Commit the probes from 1.3–1.7 here as #[ignore] tests. Run them with cargo test --test newex -- <name> --ignored --nocapture.
Gate: EXCHANGE_NOTES.md has an entry with all of the data above — WS write capabilities, REST auth scheme + /positions lag, WS ping semantics (camp A/B/C), the full channel inventory + cadence results + topology verdict + chosen tick feed, stagger limit, subscription cap, delta mode, thin-market idle gap, and liquidation-price source. All cadence tests are committed.
Goal: the skeleton compiles and signing tests pass. Trait implementations are empty stubs.
common/src/constants.rs, beside LIGHTER_REST_URL / EXTENDED_REST_URL / ZERO1_REST_URL:
pub const NEWEX_REST_URL: &str = "https://api.newex.com";
pub const NEWEX_WS_URL: &str = "wss://ws.newex.com/stream";
// + channel-split variants if orderbook/account/trades use separate socketsbot/src/config.rs — the struct beside LighterConfig / ExtendedConfig / Zero1Config:
pub struct NewexConfig {
pub api_key: String,
pub private_key: String,
pub rest_url: String,
pub ws_url: String,
pub ws_read_timeout_secs: u64,
// + exchange-specific fields
}The builder inside impl Config, beside fn lighter() / fn extended() / fn zero1():
impl Config {
pub fn newex(&self) -> NewexConfig {
let ec = self.exchange_config("newex");
let s = self.secrets.get("newex");
NewexConfig {
api_key: s.and_then(|s| s.require("API_KEY").ok()).unwrap_or("").to_string(),
private_key: s.and_then(|s| s.require("PRIVATE_KEY").ok()).unwrap_or("").to_string(),
rest_url: arb_common::constants::NEWEX_REST_URL.to_string(),
ws_url: arb_common::constants::NEWEX_WS_URL.to_string(),
ws_read_timeout_secs: ec.ws_read_timeout_secs,
}
}
}URLs come from arb_common::constants, never from a local const inside the exchange module.
config/default.toml — commented out so auto-deploy doesn't activate it:
# [exchanges.newex]
# margin_mode = "isolated"
# entry_order_type = "ioc"
# entry_slippage_pct = "max"
# exit_order_type = "ioc"
# exit_slippage_pct = "max"
# emergency_slippage_pct = "max"
# stop_loss_type = "ioc"
# ws_read_timeout_secs = 30
# sim_fill_delay_ms = 200
# # confirmation_timeout: default 1500ms (config.rs). Override only if the exchange is faster/slower:
# # entry_confirmation_timeout_ms = 1000
# # exit_confirmation_timeout_ms = 1000Section order in default.toml is the eid assignment order (IndexMap).
.env:
NEWEX_API_KEY=...
NEWEX_PRIVATE_KEY=...ExchangeSecrets::from_env("newex") loads everything prefixed NEWEX_.
Create bot/src/exchange/newex/:
bot/src/exchange/newex/
├── mod.rs # pub mod declarations
├── client.rs # NewexClient (stub impl ExchangeClient)
├── factory.rs # NewexFactory (stub impl ExchangeFactory)
├── rest.rs # REST client with typed response structs
├── ws.rs # WS handlers (placeholder)
├── symbols.rs # to_canonical / from_canonical
├── market_info.rs # NewexMarketInfo impl MarketInfo
└── (optional, as needed)
├── wire.rs # wire-format helpers
├── nonce.rs # replay protection
├── session.rs # ephemeral sessions
├── liq_price.rs # client-side liq formula
└── funding_source.rs # impl FundingSource
Register in bot/src/exchange/mod.rs: pub mod newex;.
bot/src/exchange/newex/symbols.rs:
pub fn to_canonical(exchange_native: &str) -> &str; // "BTCUSD" → "BTC"
pub fn from_canonical(canonical: &str) -> String; // "BTC" → "BTCUSD"Existing patterns: Lighter is identity (BTC → BTC); Extended strips/adds -USD; Zero1 strips/adds USD.
bot/src/signing/newex.rs, registered in bot/src/signing/mod.rs. Examples: lighter.rs (ECgFp5 Schnorr), stark.rs (Starknet ECDSA, Extended), zero1.rs (Ed25519). Capture one real signed transaction from Phase 1 and hard-code it as a test vector.
Signing pitfalls (from the Hyperliquid integration):
- Serialization key order defines the signing hash. If the exchange hashes a msgpack/JSON action before signing, key order is critical.
serde_json::Value(a BTreeMap) sorts alphabetically — which won't match a Python SDK's insertion order. Use typed structs with#[derive(Serialize)], where struct field order is key order. - Pin byte-exact tests. For every action type (limit order, trigger order, cancel, leverage), a unit test comparing the Rust serialization hex against the Python SDK's. One mismatched byte → a different hash → a different recovered address → "User does not exist."
- Nested-struct field order matters too.
TriggerSpec { triggerPx, isMarket }≠{ isMarket, triggerPx }— different msgpack even when the JSON looks the same. - Wire string format: trailing zeros, decimal precision, integer vs float (
"0.00030"≠"0.0003","74218.0"≠"74218") — every character affects the hash. Always go throughfloat_to_wire()+price_to_wire(price, decimals). - Nonce uniqueness:
current_unix_ms()isn't unique under parallel sends. Use anAtomicU64counter initialized from ms at startup.
bot/src/exchange/newex/market_info.rs:
use crate::exchange::traits::{MarketInfo, decimal_qty_step, round_qty_down_decimals};
pub struct NewexMarketInfo {
pub size_decimals: u32,
pub price_decimals: u32,
pub max_leverage: u16,
/// Qty floor in BASE UNITS — the exchange's true published rule.
/// Must NOT be computed from startup mark × USD floor (see §2.8.1).
pub min_order_size: Decimal,
/// USD notional floor if the exchange enforces one at submit time
/// in addition to `min_order_size`. `None` when the venue's only
/// size rule is qty-based. See §2.8.1 / §1.6 for how to detect.
pub min_notional_usd: Option<Decimal>,
// exchange-specific fields ...
}
impl MarketInfo for NewexMarketInfo {
fn size_decimals(&self) -> u32 { self.size_decimals }
fn price_decimals(&self) -> u32 { self.price_decimals }
fn qty_step(&self) -> Decimal { decimal_qty_step(self.size_decimals) }
fn round_qty_down(&self, qty: Decimal) -> Decimal { round_qty_down_decimals(qty, self.size_decimals) }
fn max_leverage(&self) -> u16 { self.max_leverage }
fn min_order_size(&self) -> Decimal { self.min_order_size }
fn min_notional_usd(&self) -> Option<Decimal> { self.min_notional_usd }
}Shared helpers from exchange/traits.rs: decimal_qty_step(size_decimals) (10^(-n), covers Lighter/Zero1/Hyperliquid); round_qty_down_decimals(qty, size_decimals) (truncate to n places); effective_min_qty_for_leg(info, leg_price) — read the sizing hot path through this, not through min_order_size directly, because it combines the qty floor and the notional floor at the live price. Override qty_step only when it comes from the API as a non-power-of-10 (like Extended).
Exchanges have up to two independent floors, checked at submit: a qty floor (min_order_size in base units, e.g. BTC = 0.0001, MEGA = 1) and a notional floor (a USD floor on qty × price; HL = $10, Lighter = min_quote_amount ≈ $10 per perp). An exchange may have only one, both, or neither — check both, empirically via rejection messages (§1.6). The current matrix:
| Exchange | min_order_size source |
min_notional_usd() |
Note |
|---|---|---|---|
| Hyperliquid | ceil($10 / startup_mark) — not recommended without a backup |
Some($10) |
$10 is hardcoded, not in the API; empirically verified |
| Lighter | min_base_amount from /api/v1/orderBooks |
Some(min_quote_amount) from the same endpoint |
Both fields per symbol; MKR has 0 → None |
| Extended | minOrderSize from trading-info |
None |
Qty floor only; maxOrderValue/maxPositionValue exist but are upper bounds |
| Zero1 | qty_step() (no explicit floor in the API) |
None |
No notional rule in the proto |
The anti-pattern (which once cost a one-legged fill): don't cache min_order_size = ceil(min_notional / mark) from the startup mark as your only guard. The mark drifts down, the stale cache passes a qty that now violates the exchange's real notional rule → a one-legged fill. The right pattern: if the exchange publishes literal qty/notional fields (Lighter-style), parse them straight into MarketInfo; if it publishes only a notional rule (HL-style), return it from min_notional_usd() as a const and keep min_order_size() at ZERO or a conservative floor (never derived-from-mark). Either way, effective_min_qty_for_leg(info, leg_price) — read by sizing in open.rs and executor.rs::check_unblock_symbols — recomputes both at the live price.
When sizing blocks a symbol via BlockType::QtyMin, it's released by a reactive hook in on_market_update: on every BBO tick for (exchange, symbol), the bot checks for such a block and recomputes effective_min_qty_for_leg at the new price; if potential_qty ≥ min_qty, the block lifts. A second path — hot-reloading max_order_size_usd — covers the case where the operator raised the target. The MarketInfo impl does nothing extra — a new exchange gets this for free once it fills both trait methods correctly.
Gate: cargo check passes. Signing unit tests green. The module is registered.
Goal: every REST method works and smoke tests pass.
WS-first: the REST client implements only operations that aren't available over WS (see §4.3.1). If Phase 1 (§1.1) showed that place/cancel/modify orders are available over WS, do not implement them in REST. REST is for startup queries (symbols, fees, market meta) and fallback/verify operations; write operations go over WS in Phase 4.
bot/src/exchange/newex/rest.rs — typed response structs + HTTP methods, all URLs from parameters (NewexConfig), never hardcoded. Two paths depending on call frequency:
// 1. Pooled client — frequent calls (orders, balance, fees):
use crate::utils::http::{build_https_connector, ResilientHttpClient};
pub struct NewexRest {
base_url: String,
http: ResilientHttpClient, // retry-on-transient, connection pool
}
impl NewexRest {
pub fn new(base_url: impl Into<String>) -> Result<Self> {
let https = build_https_connector()?;
let inner = hyper_util::client::legacy::Client::builder(
hyper_util::rt::TokioExecutor::new(),
).build(https);
let headers = ...;
Ok(Self { base_url: base_url.into(), http: ResilientHttpClient::new(inner, headers) })
}
}// 2. Raw TLS — rare one-shot calls (startup-only):
use crate::exchange::net::{http_get, http_post, build_tls_connector};
pub struct NewexRest {
rest_url: String,
tls: TlsConnector,
}All three use &self.rest (the shared REST client created in from_config()). Don't create a new client inside each method — that's a burst of TLS handshakes.
fetch_symbols()—self.rest.info(), parse markets, buildVec<(String, NewexMarketInfo)>.fetch_taker_fee()—self.rest.user_fees(), prefer the account-specific/VIP rate, cache intoself.cached_taker_fee, and log the number.fetch_market_meta()— forqty_step/min_qtystrings usecrate::exchange::factory::qty_step_from_decimals(size_decimals), don't duplicate the formula.
| Method | What it does | Critical notes |
|---|---|---|
fetch_equity() |
Account balance | — |
fetch_open_positions() |
Open positions | Account for the /positions lag from Phase 1 |
check_order_status() |
Order status by ID | Critical: must return real filled_qty + avg_fill_price if the order executed, Some(filled_qty=0) if cancelled, None if not yet settled — never a stub. It's the REST fallback in fill_fut_1/fill_fut_2 and in clean_close_done after a WS confirmation_timeout. Without it, a late WS/account fill reads as "expired," the bot unwinds the peer leg, and you get a naked orphan. Live-test it by artificially delaying WS handling and checking REST recovery. |
fetch_liquidation_price() |
Liquidation price | Engine value or client-side compute |
fetch_trade_fee() |
Per-order fee post-fill | Critical: if there's no per-order endpoint, return None — never Some((rate, true)) (that's a rate-vs-USD bug; see below) |
The fetch_trade_fee trap: with a per-order fee endpoint (Extended /trades), return Some((fee_usd, is_taker)); without one (Lighter, Zero1), return None — the fill-event path already wrote notional × taker_rate, and None leaves that value in place. Returning Some((taker_rate, true)) instead writes the rate as a USD amount and undercounts fees by roughly the notional — a real incident.
Capture real responses as fixture files for the Phase 4 parsers: poke (REST reachable, auth works, fees correct), session_probe (session lifecycle, if any), trade_probe (a signed PlaceOrder accepted — a no-fill IOC at a far price).
Gate: all REST smoke tests pass. Fixture files committed.
Goal: OB ticks reach the executor, fills route through PendingOrders, external closes are detected.
Every WS connection goes through WsReconnect::run() — don't hand-roll a reconnect loop.
use crate::exchange::ws_reconnect::{WsReconnect, WsReconnectConfig};
pub async fn run_newex_ob_ws(..., health: Arc<ExchangeHealth>) {
let session_health = health.clone();
WsReconnect::run(
WsReconnectConfig::new("newex OB", exchange_id, HealthChannel::WsOrderbook)
// Defaults: 500ms backoff, 30s max, default_classify, health=true, skip_clean=false.
// To customize, uncomment and add a leading dot:
// .initial_backoff(Duration::from_secs(2)) // slow reconnect
// .classify_error(classify_newex_maintenance) // CDN/WAF detection
// .no_health_on_disconnect() // for per-market supervisors
// .skip_sleep_on_clean() // immediate reconnect on Close
,
health,
|| market_tx.is_closed(),
|| {
let h = session_health.clone();
async move { run_ob_session(..., h).await }
},
).await;
}Exception: an account WS holding borrowed refs (&HashMap, &DashMap) uses a manual begin() / on_disconnect() loop. Carry state across reconnects in an Arc<AtomicBool>, not a plain bool (Copy → copied into the async move).
| Phase 1 finding | → Config/code decision |
|---|---|
| Camp A (server pings, no client ping) | WriteTaskMode::RelayOnly |
| Camp B (client must ping) | WriteTaskMode::TextPingRelay or WsPingRelay |
| Thin market idle >30s | ws_read_timeout_secs > idle gap |
| Subscription cap per WS | MAX_MARKETS_PER_WS for the batch supervisor |
| Absolute-replace deltas | DeltaMode::AbsoluteReplace, no REST snapshot |
| Incremental deltas | DeltaMode::Incremental, REST snapshot first |
| Solo tighter than shared | Support ws_ob_mode = per_market |
The session function inside the supervisor:
- Connect via
connect_wss_timeoutwithread_timeout_secsfrom config. - Subscribe + initial snapshot (REST if incremental, the first WS frame if absolute-replace).
health.set_channel(eid, WsOrderbook, true).- Spawn the write task:
spawn_write_task(ws_write, rx, mode, label). - Spawn the read task in
AbortOnDropwithmake_send_fn()for pong relay. - Spawn the freshness watchdog:
spawn_stale_data_watchdog(thresholdWS_STALE_RECONNECT_MS= 30s). join_read_task()— await.- Abort the write task, return.
BestLevels needs both timestamps:
| Field | Value | Meaning |
|---|---|---|
ts_ns |
now_ns() at parse time |
Local recv time → Recv→Sig |
exchange_ts_ns |
Venue event time if any, else now_ns() (= ts_ns) |
Venue time → Tick→Recv |
Never set exchange_ts_ns = 0 — it produces nonsense in the UI. If the venue gives no event time, set exchange_ts_ns = ts_ns and the UI shows —.
Call health.mark_data() after every processed delta, or a data_stale halt fires after 30s:
health.mark_data(exchange_id.db_id(), HealthChannel::WsOrderbook);Gap detection: track last_update_id / begin_seq; on a SequenceGap, return Err so the supervisor reconnects.
If the exchange supports an operation over WebSocket, use WS — for reads and writes alike. Use REST only for what WS can't do.
A. Writes (orders, cancels, SL) over WS. REST writes carry HTTP overhead (headers, TLS framing, possibly a new connection) → +5–50ms per order; a WS write is a frame into an already-open socket → <1ms. An arbitrage entry is two parallel orders, so REST doubles the HTTP overhead, and on exit latency is slippage is money. WS write operations also often have a separate, more generous rate limit.
// The account WS connection is used for:
// 1. receiving fills (read) — already there
// 2. sending orders (write) — through the write-task tx channel
// 3. cancelling orders (write) — the same channel
//
// In ExchangeClient::place_order():
// 1. serialize the order into the WS action format
// 2. send it through ws_write_tx (the channel to the write task)
// 3. await the fill through PendingOrders (as usual)
//
// REST fallback ONLY if the WS is disconnected at send time.B. Reads — WS push over REST poll. Position state, funding payments, balance updates, and order-status changes are often pushed over WS at the moment of the event — no 200–500ms poll latency and no rate-limit consumption. For SL movement (which depends on fetch_liquidation_price), the difference between a 60s REST poll and an instant WS push is the difference between "the SL moved in time" and a liquidation.
C. What to use when:
| Operation | WS available? | Use |
|---|---|---|
| Place / cancel order, place/modify SL | Yes | WS (REST = fallback on disconnect) |
| Set leverage | Yes | WS (or REST at startup — one-time) |
| Position / funding / balance | Yes (push) | WS push (REST = fallback/verify) |
| Market info / symbols, fee rate, historical trades | No (startup / history) | REST |
Two jobs beyond fill routing.
A. Fill routing through PendingOrders:
let pending_orders = Arc::new(PendingOrders::<String>::new()); // or <u64> or <u64, Side>
// place_order: register
let (_tx, rx) = self.pending_orders.register(order_id, ());
// WS fill handler: resolve
if let Some((meta, tx)) = pending_orders.resolve(&order_id) {
tx.send(Ok(fill));
}
// cancel/reject
pending_orders.reject(&order_id, "cancelled".into());
// periodic cleanup
pending_orders.cleanup_closed();B. Position updates + external-close detection. The PositionEventSink contract is the only path for position updates and external leg closes. Writing to the DashMap cache for the fetch_liquidation_price fast path is fine, but update_position() and add_funding() must be called through the sink on every position-update frame — otherwise the executor never sees the update, funding is 0 at close (the data sat in the DB, not the cache), and the SL monitor never gets liq-price drift. (Lesson: on Hyperliquid the DashMap cache updated but the sink calls were skipped, so the executor saw stale state and funding was lost at close.)
// on EVERY position-update frame:
position_events.update_position(exchange_id, symbol_id, liq_price, unrealized_pnl, avg_entry, margin, now_ms);
// on a funding settlement:
position_events.add_funding(exchange_id, symbol_id, change);
// on an EXTERNAL close (SL fire / liquidation / manual close from the exchange UI):
position_events.report_external_close(exchange_id, symbol_id, reason, filled_qty, fill_price, source_order_id);ExternalCloseReason: StopLoss (our SL fired on the exchange), Liquidation (the exchange force-closed), ExternalManual (everything else — manual close, ADL). If the exchange doesn't distinguish reasons (like Zero1), always use ExternalManual — it's more honest than silence.
Forbidden: a direct leg_closed_tx.send(...) (a grep for leg_closed_tx\.send under bot/src/exchange/newex/ must return 0), and a direct write to the exchange_positions DashMap (only via position_events.update_position()).
PublicTrade::side is the taker/aggressor (who crossed the spread). If the exchange reports the maker side, invert it (maker bid → taker Sell, maker ask → taker Buy). Timestamps: with no engine timestamp, ts_ns = exchange_ts_ns = now_ns().
ws_sniff_ob (OB WS connects, deltas arrive), ws_sniff_account (account WS connects, auth works), e2e_single_trade (real fill + position update + liq price).
Gate: ticks flow into market_tx, fills route, external closes are detected. Fixture files committed.
Goal: every file outside exchange/newex/ is edited and cargo test --workspace --lib is green.
bot/src/exchange/newex/factory.rs:
pub struct NewexFactory {
config: NewexConfig,
/// Shared REST client — created once in from_config(), reused by all
/// preflight methods. Avoids burst TLS handshakes (12→4 simultaneous).
rest: NewexRest,
cached_market_data: OnceLock<Vec<MarketData>>,
cached_taker_fee: OnceLock<Decimal>,
}
impl ExchangeFactory for NewexFactory {
fn name(&self) -> &str { "newex" }
fn fetch_symbols(&self) -> Pin<Box<...>>; // uses &self.rest
fn fetch_taker_fee(&self) -> Pin<Box<...>>; // uses &self.rest, caches into self.cached_taker_fee
fn build(self: Box<Self>, exchange_id, ctx) -> Pin<Box<...>> {
Box::pin(async move {
// 1. Load keys, verify account
// 2. Build REST client, fetch market info
// 3. Create session (if ephemeral sessions)
// 4. Spawn WS supervisors (OB + Account + Trades)
// 5. Build NewexClient
// 6. Wire clean_close_done (§5.2)
// 7. Wire the ready future (§5.1.1)
// 8. Wire FundingSource (optional)
Ok(BuiltExchange { client, fee_rate, ready, clean_close_done, funding_source })
})
}
}BuiltExchange::ready is the future main.rs awaits before "all systems online." Don't put a blocking poll loop that waits for data from every symbol. ready should be instant (Box::pin(async { Ok(()) })) when the OB/BBO channel is event-driven, there's no guaranteed snapshot on subscribe, and thin markets can go quiet for minutes — the spread detector's max_recv_staleness gate already blocks entries on a symbol with no fresh data, so a blocking wait here just duplicates that protection and stalls every exchange's startup for one thin market. A blocking wait is only appropriate when the exchange guarantees a snapshot on subscribe (Lighter: the first WS frame is a full snapshot) and ready is awaiting a specific handshake, not arbitrary data frames.
Any per-symbol REST loop at startup (leverage, margin mode, …) must be parallel via JoinSet, not a sequential for ... await. Sequential is N × RTT (98 symbols × 100ms = 10s from Tokyo, ×500ms = 50s trans-Pacific); parallel is ~max(RTT) regardless of count.
// ✗ sequential — 98 × 100ms = 10s from Tokyo
for mi in markets {
client.update_leverage(mi.asset_id, leverage, false).await;
}
// ✓ parallel — ~100ms total
let mut set = tokio::task::JoinSet::new();
for mi in markets {
let c = client.clone();
set.spawn(async move { c.update_leverage(mi.asset_id, leverage, false).await });
}
while set.join_next().await.is_some() {}Before parallelizing, check the exchange's rate limits (Hyperliquid: 1200 weight/min, 98 requests = 98 weight — safe), the operation's idempotency (a leverage set is idempotent — safe to repeat), and whether it can be skipped (if a read endpoint exists, skip symbols already at the right value). Extended's set_leverage() is the pattern: parallel via JoinSet, blocking symbols whose max_leverage < config_leverage. (Lesson: 98 sequential updateLeverage calls were 50s of startup from a Mac, 10s from Tokyo; parallelized, 1–2s.)
Three rules. Breaking any one leaves directional exposure.
Rule 1 — wait for the account WS before the first close order. clean_close_done starts before ready, so the account WS may not be connected yet.
// Pattern A (Extended/Zero1): poll health
let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
loop {
if health.is_channel_connected(eid, HealthChannel::WsAccount) { break; }
if tokio::time::Instant::now() >= deadline {
tracing::warn!("account WS not ready after 10s — REST fallback will catch fills");
break;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
// Pattern B (Lighter): close inline inside the WS session task AFTER set_channel(WsAccount, true)Rule 2 — REST fallback on a WS timeout. Don't trust rx.recv() alone; a WS fill event can be lost.
match tokio::time::timeout(Duration::from_secs(10), rx.recv()).await {
Ok(Some(Ok(f))) if !f.is_empty() => { closed += 1; }
_ => {
// WS missed the fill — check REST
tokio::time::sleep(Duration::from_secs(2)).await;
let positions = client.fetch_open_positions().await?;
if positions.iter().all(|p| p.market != market) {
tracing::info!("position closed (recovered from REST)");
closed += 1;
continue;
}
tracing::warn!("close not filled");
}
}Rule 3 — hard-fail on a residual after retries. Never Ok(()) + error!("close manually") — an Ok(()) lets the bot open new positions on top of the residual.
for attempt in 1..=5u8 {
let positions = client.fetch_open_positions().await?;
if positions.is_empty() { break; }
for pos in &positions {
if let Err(e) = client.close_position(&pos.market, pos.size, &pos.side).await {
tracing::warn!(%e, attempt, "close_position errored, will re-verify");
}
}
if attempt < 5 { tokio::time::sleep(Duration::from_secs(2)).await; }
}
let final_positions = client.fetch_open_positions().await?;
if !final_positions.is_empty() {
bail!("cannot start with {} residual positions on newex", final_positions.len());
}The compiler forces you to implement every required method — but don't leave stubs:
| Method | Critical notes |
|---|---|
place_order() |
Soft errors ("no fills" as an error code) → translate to Fill { filled_qty: 0 }. Transient errors → retry after 2s |
place_stop_loss() |
Not a stub! Called after every entry — a stub means "position UNPROTECTED" |
close_position() |
Not Ok(())! A clean restart calls it for residuals. Pattern: IOC reduce-only at mark ± limit_price_cap |
fetch_liquidation_price() |
Called every 60s — without it the SL never moves |
limit_price_cap() |
Critical. None only if the exchange truly has no limit; otherwise Some(0.10) for 10%. With no cap, SlippagePct::Max = 35% and every order is rejected — invisible in tests priced near mark! |
Default methods — review each against Lighter/Extended/Zero1: modify_stop_loss (default cancel+place, 3× retry; override if there's an atomic-modify API), config_schema (default empty — always override, a blank panel is confusing), refresh_funding_now (default no-op; override if the WS doesn't push cumulative funding), forget_pending_order (default no-op; override if the key type matches the executor's REST-recovery path).
Do these in order — skipping a REQUIRED row means the exchange won't load or its credentials won't validate.
| # | Required? | File | What to add |
|---|---|---|---|
| 1 | REQUIRED | common/src/constants.rs |
NEWEX_REST_URL, NEWEX_WS_URL (done in Phase 2) |
| 2 | REQUIRED | bot/src/config.rs |
NewexConfig + Config::newex() (done in Phase 2) |
| 3 | REQUIRED | bot/src/exchange/mod.rs |
pub mod newex; (done in Phase 2) |
| 4 | REQUIRED | bot/src/exchange/factory.rs |
The match arm in build_factories(): "newex" => Box::new(NewexFactory::from_config(config)?) |
| 5 | REQUIRED | bot/src/web/credentials.rs |
NewexCredentials struct + impl + Box::new(NewexCredentials) in registry() + a validate_newex() fn + update the assert_eq!(s.len(), N) test |
| 6 | RECOMMENDED | bot/src/storage/sqlite.rs |
Seed a stable id in Storage::open: INSERT OR IGNORE INTO exchanges (id, name, display_name, exchange_type, ...) VALUES (N, 'newex', 'NewEx', 'dex_offchain', ?1) |
| 7 | optional | config/default.toml |
[exchanges.newex] (commented out) |
| 8 | optional | .env |
NEWEX_* secrets |
| 9 | optional | bot/src/main.rs |
Add "newex" to TICK_LATENCY_COUNT_ONLY only if the exchange doesn't publish matching-engine timestamps |
| 10 | optional | bot/src/web/server.rs |
clamp_exchange_slippage — only if there's a hard protocol-level max slippage |
| 11 | optional | bot/tests/common/mod.rs |
TestUrls + newex_helpers.rs |
Credentials implementation (#5):
// bot/src/web/credentials.rs
struct NewexCredentials;
impl ExchangeCredentials for NewexCredentials {
fn schema(&self) -> ExchangeCredentialSchema {
ExchangeCredentialSchema {
exchange_id: "newex",
display_name: "NewEx",
fields: vec![
f("API_KEY", "NEWEX_API_KEY", "API Key", FieldKind::Secret, true),
// ... per .env field
],
}
}
fn validate(&self, env_path, overrides) -> Pin<Box<...>> {
Box::pin(async move { validate_newex(&env_path, overrides).await })
}
}
fn registry() -> Vec<Box<dyn ExchangeCredentials>> {
vec![
Box::new(LighterCredentials),
Box::new(ExtendedCredentials),
Box::new(Zero1Credentials),
Box::new(NewexCredentials), // ← new
]
}validate_newex(): read secrets from .env via read_env_secrets_with_overrides, shape-check the required fields, build a throw-away Config with the candidate secrets, run fetch_taker_fee() (which exercises the full signing + auth), and return ValidationOk { summary: "taker fee ...", resolved }.
Implement fetch_market_meta() in NewexFactory (a required trait method). The data is auto-upserted into exchange_symbols at startup and used by leverage blocking and the backtester's SL distance.
What to find in the API:
| Field | Why | How to find it |
|---|---|---|
| max_leverage | Block symbols when config leverage > max | maxLeverage, max_leverage, or from IMF: max_leverage = 1/IMF |
| MMF (maintenance margin fraction) | Backtester SL distance: liq_dist = max(1/lev, 1/max_lev) − mmf |
mmf, maintenance_margin, maintenanceMargin, riskFactor — the formula is exchange-specific (below) |
| size_decimals, price_decimals | Qty/price rounding | Usually in the market-info endpoint |
| qty_step, min_qty | Min-order validation | Usually in the market-info endpoint |
| limit_price_cap | Max price deviation from mark | Not on every exchange |
Verify with curl before writing code — confirm the field exists in the response (don't guess from someone else's code), the type is right (string vs number vs basis points), and the units are what you think (fraction 0.02 vs percent 2% vs basis points 200):
curl -s 'https://api.newex.com/markets' | python3 -c "
import sys, json; data = json.load(sys.stdin)
m = data['markets'][0]
for k,v in sorted(m.items()): print(f' {k}: {type(v).__name__} = {v}')
"MMF patterns (don't copy a formula from another exchange — each defines MMF its own way; verify in the docs):
| Pattern | Example | How to get MMF |
|---|---|---|
Explicit mmf field |
Zero1: mmf: 0.01 |
mmf directly |
| Explicit field in bps | Lighter: maintenance_margin_fraction: 120 |
value / 10000 = 0.012 |
riskFactor = MMR |
Extended/StarkEx: riskFactor: "0.02" |
riskFactor directly (do not halve it) |
| IMF/2 heuristic | Hyperliquid: maxLeverage: 40 |
0.5 / maxLeverage (per the docs) |
| Not in the API | — | None → operator override via exchange_symbol_overrides |
Max-leverage patterns: explicit field (maxLeverage: "50.00"); from IMF (floor(1 / imf)); from IMF in bps (10000 / value); or absent (0 = unknown, leverage blocking skips it). Per-exchange details: EXCHANGE_NOTES.md → "Margin / Leverage / MMF."
fn fetch_market_meta(&self) -> Pin<Box<dyn Future<Output = Result<Vec<MarketMeta>>> + Send + '_>> {
Box::pin(async move {
let data = rest.markets().await?;
let mut out = Vec::new();
for m in &data {
out.push(MarketMeta {
symbol: canonical_name(&m.symbol),
market_name: m.symbol.clone(),
size_decimals: m.size_decimals,
price_decimals: m.price_decimals,
qty_step: format_step(m.size_decimals),
min_qty: m.min_order_size.clone(),
max_leverage: compute_max_leverage(&m), // from API docs
mmf: compute_mmf(&m), // from API docs
limit_price_cap: m.limit_price_cap,
});
}
Ok(out)
})
}Shared REST client + caching. The factory creates the REST client once in from_config() and keeps it as a field; all three preflight methods use &self.rest. Otherwise each method opens its own client → 3 simultaneous TLS handshakes per exchange × 4 exchanges = 12 burst handshakes, and a server with a small accept queue (Zero1) rejects with "tls handshake eof." fetch_market_meta() and fetch_taker_fee() cache into OnceLock fields that build() drains (with a fresh-fetch fallback), eliminating duplicate REST calls between preflight and build. All four current exchanges do both.
liq_price_live (fetch_liquidation_price vs engine or a second source, across leverages) and sl_move_sim (place SL → modify → verify via REST after each step).
| File | Why |
|---|---|
main.rs (except TICK_LATENCY_COUNT_ONLY) |
Iterates factories generically |
spread_detector.rs |
C(N, 2) pairs automatically |
execution/{open,close,sl_monitor}.rs |
Dispatch via ctx.client(eid) |
position_events.rs |
One façade; a new exchange plugs in via BuildContext |
executor.rs::on_external_leg_close |
Symmetric close works for any exchange |
common/src/sim.rs, backtest.rs, optimizer/ |
Sim hot loop generic; Vec<ExchangeSimSpec>; build_sim_params takes &[&str] |
| Web UI (backtest/optimizer multi-select) | Automatic |
Gate: cargo test --workspace --lib + cargo test --test apply_overrides green.
Unit tests don't replace live validation. Signing, price precision, min notional, keepalive timing, fill routing — these break only on real money. Requirements: a real account with a small deposit ($15–30) and credentials in .env. Run the live tests as soon as Phase 5 compiles — don't defer them.
The minimum set (all required), each of which must leave the account clean (0 positions, 0 orders); if a test panics, use close_all_positions to recover:
| # | Test | What it checks | Cost |
|---|---|---|---|
| 1 | credentials_probe |
Signing key → address, fee fetch, clearinghouse | $0 |
| 2 | trade_probe_no_fill_ioc |
Signing + submission + rejection handling | $0 |
| 3 | e2e_fill_round_trip |
Open + fill + position + liq price + close | ~$0.01 fees |
| 4 | liq_price_live |
fetch_liquidation_price vs engine, ratio sanity |
~$0.01 |
| 5 | sl_move_sim |
Place SL → modify → verify via REST → cancel → close | ~$0.01 |
| 6 | ws_fill_routing |
WS fill arrives, cloid round-trip, PendingOrders routing | ~$0.01 |
| 7 | close_all_positions |
Emergency-cleanup utility | $0 |
Three bugs are invisible in unit tests — only live validation catches them.
- Open a real position (a one-off
e2e_open_onlytest, or the bot + SIGTERM). - Confirm it's open in the exchange UI.
- Restart in
cleanmode (tools/bot_start.shwithstartup_mode = "clean"). - Required in the logs:
Verified: 0 open positions exchange=newex;Clean close done; noWARN close not filledwithout a followingrecovered from REST. - Negative test: force a residual (stub
limit_price_capto something impossible) → restart → the bot exits withErr, it doesn't keep trading.
- Point the bot at one symbol shared with an existing exchange.
- Wait for
POSITION OPENEDon both legs. - Manually close only the newex leg from the exchange UI.
- Required in the logs:
ExternalLegClose exchange=newex reason=external_manual; the executor closes the paired leg on the other exchange; both legs go to zero andfetch_open_positionsis empty for both. - Static check:
grep "leg_closed_tx\.send" bot/src/exchange/newex/= 0 matches.
- Find the exchange's thinnest symbol.
- Set
entry_slippage_pct = "max". - A small entry order is filled, not rejected with a price-band error.
- Negative: stub
limit_price_cap()→None→ the order is rejected → revert.
Gate: all three tests pass. Update EXCHANGE_CANDIDATES.md → "code-complete."
| Symptom | Cause | Fix |
|---|---|---|
| "unknown exchange 'newex'" | Missing arm in build_factories() |
Add the arm |
schemas_cover_all_configured_factories fails |
Missing NewexCredentials in registry() |
Add the struct + impl |
| "position UNPROTECTED" | place_stop_loss is a stub |
Implement it |
| Clean restart leaves positions | close_position is Ok(()) |
IOC reduce-only |
| SL never moves | fetch_liquidation_price returns None |
Implement it |
| Every real order rejected | limit_price_cap() = None → 35% slippage |
Override with a real cap |
| WS hangs forever | connect_wss without a timeout |
connect_wss_timeout + read_timeout_secs |
| Funding = 0 at close | The WS doesn't push funding | Wire refresh_funding_now |
| Leg closed externally, paired leg hangs | No report_external_close in the account WS |
Wire detection (§4.4) |
| Direction leak | Direct write to the exchange_positions DashMap |
Use position_events.update_position() |
fetch_trade_fee returns rate instead of USD |
Some((rate, true)) instead of None |
Return None if there's no per-order endpoint |
| URLs break when the exchange changes domain | Hardcoded const in the exchange module |
URLs from arb_common::constants only |
apply_n3 test fails |
Stable-id seed shifted autoincrement | Update the test assertions |
Real bugs from a production integration. Each cost 30+ minutes of debugging — and each is now a rule above.
| Pitfall | How it showed | Root cause | Guard |
|---|---|---|---|
| msgpack key order | "User does not exist" on every order | serde_json::Value sorts keys alphabetically → different msgpack → different keccak hash → the server recovers the wrong address |
Typed structs with #[derive(Serialize)] (field order = key order); pinned byte-exact tests vs the Python SDK |
| msgpack field order in nested structs | SL trigger orders "User does not exist" despite a correct top-level msgpack | TriggerSpec { triggerPx, isMarket, tpsl } field order ≠ the SDK's { isMarket, triggerPx, tpsl } |
A pinned test for every order type (limit + trigger); compare hex byte for byte |
| Trailing zeros in wire strings | "0.00030" ≠ "0.0003" → different msgpack |
format!("{:.5}") keeps a trailing zero; the SDK strips it |
Always float_to_wire() — strip trailing zeros |
| Guessing WS keepalive | Account WS dropped every ~50s | Assumed Camp A (server pings); really Camp B (client pings every 60s) | Read the docs before coding, not probes |
| WS connection limits | Wrote a per-market WS (140+ connections) | Didn't check the limit (10 max) | Phase 1: check the connection limit before the architecture decision |
| Price-precision rejection | "Order has invalid price" on every order | BTC price with 2 decimals; the exchange expects 1 or an integer | price_to_wire(price, decimals) — round before signing |
| Min notional undocumented | "Order must have minimum value of $10" | Not documented, only empirical | Phase 1 or Phase 6: a test order at the minimum |
| position_events bypass | Executor never saw position updates | Wrote to the DashMap cache but not through PositionEventSink.update_position() |
Audit: grep position_events.update_position in the new module |
| Funding bypass | funding_1/2 = 0 at close |
Didn't call position_events.add_funding() |
Audit: grep add_funding |
| f64→Decimal liq-price drift | Phantom SL-movement triggers | Decimal::try_from(74218.3_f64) = 74218.2999... |
Keep liq price as a String; parse with Decimal::from_str |
| Nonce collision | Parallel orders with the same timestamp_ms | current_unix_ms() isn't unique under concurrent sends |
An AtomicU64 counter, initialized from ms, incremented per action |
| Agent vs account address | "User does not exist" on REST queries | The agent-wallet address ≠ the main-account address; info queries need the account address | Clear naming: API_PRIVATE_KEY, API_WALLET_ADDRESS, ACCOUNT_ADDRESS |
| REST instead of WS for orders | +5–50ms per order, a full write-path rewrite | Didn't check in Phase 1 which write operations were on WS | Phase 1 §1.1: check WS write capabilities first — orders over WS reuse the connection at <1ms overhead |
| WS response latency > confirmation_timeout | Parallel entry: Lighter filled, the HL IOC marked status=expired ms=0 → Lighter unwound. The HL fill actually arrived over the WS account feed 200ms after the timeout → a naked HL short, and the Lighter unwind went off at 225 bps of slip |
fill_fut_2 had no REST fallback (unlike fill_fut_1); when HL's WS response ran >1.5s and the account fill >1.7s, the bot timed out and called the order expired though it had actually executed — an asymmetry between leg 1 and leg 2 recovery |
(1) check_order_status is mandatory, not a stub (§5.3). (2) fill_fut_2's REST fallback is now symmetric with fill_fut_1 — after a WS timeout with qty=0 && error=None, go to check_order_status before unwinding the peer leg. (3) When adding an exchange, run a live test with the WS response artificially delayed and confirm the REST fallback fires and no orphan is created |
After the N-exchange generalization, these need no changes: main.rs (iterates factories generically), spread_detector.rs (C(N, 2) pairs), execution/{open,close,sl_monitor}.rs (dispatch via ctx.client(eid)), position_events.rs (façade), executor.rs::on_external_leg_close (symmetric close), common/src/sim.rs + backtest.rs + optimizer/, the web UI (backtest/optimizer multi-select, apply best params), the tick writer + Parquet partitioning, and the symbol_overrides schema.