feat(gateway): adopt the unified route surface, and record what the executors actually did

[fengtality]

Summary

hummingbot-api's half of the Gateway route unification, plus the recording gaps found by running the LP and swap executors against Solana mainnet and then looking for the evidence. Supersedes #217, which was opened from a branch that has since been replaced.

Requires hummingbot/gateway#683 and hummingbot/hummingbot#8431.

Note on history: this branch was rewritten to remove bots/archived/ from every commit — six archived bot instances had been committed, carrying conf/connectors/*.yml (encrypted API keys) and conf/.password_verification. conf/ is now ignored at any depth. The branch's tip content is unchanged; only the removed paths differ. The rewrite is scoped to this branch's own 46 commits, so history shared with main is untouched.

The database said everything succeeded

Every row in both position event tables read CONFIRMED with no error_message — not because nothing had failed, but because a failure could not be written. The recording code runs only when Gateway returns; a transaction that lands and reverts makes Gateway raise, and control skips every create_event to reach an except that persists nothing. A close that reverted at slot 440494812, costing 0.000011772 SOL, left no row at all.

Only failures carrying a transaction id are recorded: a pre-flight simulation failure never got a signature and cost nothing, while a landed revert has one and paid gas.

The recommended workflow recorded nothing

gateway_clmm_positions.position_rent is written by the OPEN route and position_rent_refunded by CLOSE. An executor holds its position through the wheel and talks to Gateway directly, so neither route runs — the poller discovers the position and files it with both columns NULL. That is ~0.0100572 SOL per Orca position, more than the liquidity in a small one, unrecorded on the path that is actually recommended.

The same architecture hid every executor swap: gateway_swaps is written by the /gateway/swap/* routes, so two CONFIRMED MARKET swaps reconciling exactly against the wallet reached no row. The table was not wrong, it was silently partial — /gateway/swaps/search described only hand-driven swaps, with no marker saying so.

Both are now recorded from the executor at completion. Zero is never stored: the executor defaults its rent figures to 0.0, and a stored 0.0 claims an observation where NULL is the truth.

Pool discovery buried the only usable pool

list_pools defaulted to sort_key="volume". On a token whose DLMM pools are all idle every row ties at volume_24h = 0.00, so the order is arbitrary and liquidity is never consulted. Of 73 UMBRA-USDC pools, the one holding 15.34K ranked 68th and one holding $1.07 ranked 47th — so an agent reading top-down took the $1.07 pool and reported the deep one as "not found".

Default is now tvl. The documented keys were also wrong; probing the live upstreams:

meteora   tvl, volume_24h, fee_tvl_ratio_24h   OK
          fees_24h, apr, liquidity, volume     400
orca      tvl, volume, fees, rewards, yieldovertvl   OK

So feetvlratio was real under another name, and this router's own _24h suffixing manufactured fees_24h, which Meteora rejects. Both surfaced as an opaque 500. Keys are translated per connector; anything else is refused here, naming the ones that work.

lp_rebalancer was listed and unusable

Its __init__.py used an absolute from controllers.generic... import — the layout inside a bot container, not the bots.controllers.* one hummingbot-api mounts — so /config/template 404'd while /controllers/ still advertised it. It also called parse_provider(..., default_trading_type="clmm") after the wheel deliberately removed that argument, and strategy_v2_base catches the resulting TypeError and carries on: the bot came up healthy with no controller at all, reporting "stopped" and "N/A".

Both are fixed, with a test that constructs every package-style controller — importing was never enough, since the module imported fine and the config class resolved fine.

Companion PRs

• gateway feat(trading): unify the route surface, and fix what validating it surfaced gateway#683 — unified /trading routes, signing-route gate, decimal strings on the wire
• hummingbot feat(gateway): adopt the unified route surface, and give executors a slippage ramp hummingbot#8431 — executor slippage ramp, chainNetwork on tokens/pools, volume from fees
• hummingbot-api — this PR
• hummingbot-api-client feat(gateway): execute a held quote, and surface what a quote did not say hummingbot-api-client#26 — execute_quote, and the spec refreshed for both
• condor feat(clmm): manage_clmm tool to recover orphaned LP positions; bin_count on get_pool_info condor#204 — agent visibility, orphaned/resolve_orphan actions, bin_count on get_pool_info

Validation

279 tests pass (2 pre-existing failures unrelated to this branch: test_controller_config_class_loading parametrises over an ema_trend_v1 controller the repo does not ship). flake8 and isort clean.

Verified against a deployed container: the config template returns 200, the controller constructs, pool listings rank by depth, and the volume aggregate excludes LP deposits — 41.09 of phantom volume removed across the live executors table.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GuR4iDtH4ZfT51rJfs3DhK

[greptile-apps]

Greptile Summary

This PR adopts Gateway's unified trading routes and expands persistence for executor swaps, LP rent, failed transactions, and generated volume. It also regenerates Gateway models, improves pool sorting and connector validation, and repairs the LP rebalancer's package loading and terminal-position handling.

• Routes swap, AMM, and CLMM operations through the unified Gateway surface.
• Records executor-produced swaps, LP account rent/refunds, failures with transaction IDs, and actual traded volume.
• Adds database models, repositories, and migrations supporting the new persisted data.
• Defaults pool discovery to TVL and validates connector-specific sorting and extra parameters.
• Fixes LP rebalancer imports, provider parsing, and orphaned-position behavior.

Confidence Score: 4/5

The PR should not merge until unified write requests preserve exact decimal amounts instead of rounding them through binary floats.

The new shared body serializer can alter transaction amounts before every affected swap or liquidity request reaches Gateway, causing execution and persisted records to differ from the caller's exact value.

Files Needing Attention: services/gateway_client.py

Important Files Changed

Filename	Overview
services/gateway_client.py	Introduces unified route and payload handling, but Decimal write values are narrowed to binary floats before transmission.
services/executor_service.py	Adds lifecycle recording for executor swaps, LP rent/refunds, generated volume, and terminal failures with explicit retry and cleanup bookkeeping.
database/connection.py	Adds existing-database migrations for rent fields and executor traded volume, including historical non-LP backfill.
routers/gateway_swap.py	Unifies quote, execute, and execute-quote flows and centralizes transaction recording and fill reporting.
routers/gateway_clmm.py	Migrates CLMM operations to unified Gateway contracts and adds connector-specific pool-sort translation and validation.
bots/controllers/generic/lp_rebalancer/lp_rebalancer.py	Updates provider parsing and prevents new LP exposure when a terminated executor still reports an on-chain position.
models/gateway_generated.py	Adds generated Pydantic models corresponding to the vendored unified Gateway OpenAPI schema.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant API as Hummingbot API
    participant Gateway
    participant Chain
    participant DB
    Caller->>API: Quote / execute / LP request
    API->>Gateway: Unified /trading request
    Gateway->>Chain: Submit transaction
    Chain-->>Gateway: Signature, status, amounts, gas/rent
    Gateway-->>API: Unified response
    API->>DB: Record swap, position event, rent, or failure
    API-->>Caller: Execution result

Loading

_{Reviews (1): Last reviewed commit: "chore: untrack every bot conf directory,..." | Re-trigger Greptile}

[greptile-apps]

Decimal amounts lose precision

When a transaction amount exceeds IEEE-754 precision, _body converts its Decimal to a binary float before transmission, causing Gateway to execute and record a rounded value instead of the caller's exact swap or liquidity amount.

[fengtality]

Checked this one carefully — the observation is accurate but the conclusion doesn't hold, and I don't think it should block the merge.

float(Decimal) does narrow to IEEE-754. But the receiving field is a JSON number: gateway declares all 41 request-side decimal fields as type: number (verified against the vendored openapi.json on this branch). A JSON number is parsed by Fastify/AJV into a JS double, so the precision is bounded by the receiver, not by this conversion.

Sending the full-precision literal instead changes nothing:

sent by _body        {"amount": 0.12345678901234568}
sent as raw literal  {"amount": 0.12345678901234567890123}

parsed from _body        0.12345678901234568
parsed from raw literal  0.12345678901234568
identical after parse    True

float() yields the nearest double; JSON.parse yields the nearest double. Same value either way — the conversion is lossless relative to what the receiver can represent. And the alternative the docstring rules out — sending a string — would now be rejected outright, because this PR also turns on additionalProperties/type strictness on gateway's side (a string in a type: number field is a 400, not a silently-coerced number).

The real fix is to change the request schema to string + format: decimal and parse at each of those 41 boundaries in gateway, which is deliberately not in this PR. The response half is done here — 98 response money fields now cross the wire as decimal strings, which is where the precision actually mattered, because those are figures the chain produced rather than figures a caller chose. The request half is tracked as a follow-up.

Worth noting the practical exposure too: a request amount is a value the caller picked, usually with few significant digits, and it is echoed back through the now-exact response before anything settles against it.

[greptile-apps]

That's a thorough and correct analysis. The precision boundary is the receiver's double, not the sender's representation — float() and a raw literal serialize to the same IEEE-754 value, and JSON.parse collapses them identically. The docstring's rationale for avoiding mode="json" (which would stringify Decimal) is also sound given gateway's strict type: number enforcement added in this PR.

The asymmetry between request and response sides is the right framing: response values originate on-chain and must survive the round-trip exactly, which is why the 98 response fields are converted to decimal strings. Request values are caller-chosen, low-precision inputs that the caller holds and can verify against the echoed response. The schema change to string + format: decimal on the request side is the correct long-term fix and fits naturally as a follow-up once the response half is stable.

Comment withdrawn — no issue here.