Skip to content

feat: capture TXE server logs in CI and hexdump IPC response frames on decode errors - #25160

Open
charlielye wants to merge 1 commit into
nextfrom
cl/txe-log-capture-and-ipc-frame-dump
Open

feat: capture TXE server logs in CI and hexdump IPC response frames on decode errors#25160
charlielye wants to merge 1 commit into
nextfrom
cl/txe-log-capture-and-ipc-frame-dump

Conversation

@charlielye

Copy link
Copy Markdown
Contributor

Why

A flaky noir-contracts TXE failure (Expected size in TreeStateReference deserialization) proved undiagnosable after the fact, for two independent reasons:

  1. The TXE's server-side logs were discarded. start_txes ran the TXE under dump_fail, which only emits captured output when the process itself exits non-zero — and the TXE survives to a clean shutdown even when tests against it fail. Every wsdb spawn/exit line, IPC warning, and stack trace from the flaky run was silently dropped.
  2. The client had no record of what arrived on the wire. Generated IPC clients correlate responses positionally (no request-id envelope), so when a response fails a decode or shape check, the raw frame is the only evidence of what actually arrived — and it was thrown away.

What

TXE logs are now captured like the test engine's (bootstrap.sh):

  • start_txesstart_txe: starts the single TXE via setsid color_prefix "txe" "denoise …", so its full output lives in its own CI log (live-published, persisted on shutdown — denoise finalizes the log on SIGTERM). The oracle test resolver gets the same treatment.
  • stop_txesstop_txe: kills each setsid'd process group so node dies with its wrappers.
  • NUM_TXES is removed — only one TXE is ever run. The aztec-nr test_cmds round-robin port assignment collapses to the fixed port.

Generated TS clients hexdump the offending frame on decode errors (ipc-codegen/src/typescript_codegen.ts, both Async and Sync APIs):

  • msgpackCall now returns { decoded, raw }; msgpack decode failures, variant-name mismatches, and body-shape (to*) failures all rethrow with a bounded hexdump appended (first 4096 bytes + total length). The original error object is mutated rather than wrapped, preserving its type and stack.
  • The createError (server-reported error) branch deliberately bypasses the dump — those are well-formed frames whose message already says everything, and wrapping would break custom IpcErrorFactory error types.

Example of what a failure now looks like:

Expected variant name 'EchoFieldsResponse' but got 'SomeOtherResponse'; response frame (25 bytes): 92b1536f6d654f74686572526573706f6e7365de0001a17801
Data read, but end of buffer not reached {"name":"MessagePack 0xC1"}; response frame (3 bytes): c1ff00

One glance distinguishes a mispaired frame (wrong variant) from corruption (arity/bytes) from truncation — the classification that took days of archaeology for the original flake.

Validation

  • bash -n on both bootstrap scripts; repo-wide grep shows no remaining NUM_TXES references.
  • ipc-codegen schema tests pass; echo_example ts_package regenerated from the modified templates, rebuilt, and its UDS package + reliability tests pass.
  • Negative tests against the regenerated package verify the dump fires for wrong-variant and undecodable frames on both AsyncApi and SyncApi, with the formats shown above.

Note: the wsdb/avm TS packages consumed from npm pick up the client-side change on their next regeneration/publish from the labs pipeline; this PR changes the generator and the in-repo TXE/CI plumbing.

@charlielye
charlielye requested a review from nventuro as a code owner August 10, 2026 16:25
@charlielye
charlielye force-pushed the cl/txe-log-capture-and-ipc-frame-dump branch from b512066 to a58e958 Compare August 11, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant