evm-rpc: retry transient 'invalid block height' on finalized-head probe (fixes hyperliquid-testnet dump crash-loop)

[elina-chertova]

Symptom

hyperliquid-testnet_Writer_Short_Stall (evm-archive, kind=ingest). The parquet writer's last block has not advanced for 48h; the write container logs "no blocks were found. waiting 300 sec" in a steady loop.

Root cause (proven)

The writer is starved because the upstream dump-hyperliquid-testnet-0 pod crash-loops (6 restarts/1h, 119/24h, 317/48h — spanning the entire 48h stall). The dump fetches blocks fine for ~10 min, then dies. The previous (crashed) container's last line:

RpcError: invalid block height: 57603085
    at EvmRpcClient.receiveResult (/squid/util/rpc-client/lib/client.js:376:27)
code: -32603, rpcMethod: "eth_getBlockByNumber", rpcParams: ["finalized", false]
rpcResponse: { error: { code: -32603, message: "invalid block height: 57603085" } }

The dump probes the finalized head via Rpc.getLatestBlockhash('finalized') → eth_getBlockByNumber(["finalized", false]). The load-balanced provider (uniblock) momentarily returns -32603 "invalid block height" when a backend node lags behind its finalized pointer (57603085 ≈ chain head 57603305). This getLatestBlockhash call passes only validateResult and no validateError, so receiveResult throws a plain RpcError — which is not a connection error and so is not retried — and the process exits. On restart the dump resumes from the last persisted raw chunk, re-fetches the same range, hits the same transient error before it can flush a new chunk, and crash-loops indefinitely → the writer never receives new raw data → Writer_Short_Stall.

This is a missed path of the existing Hyperliquid fix: getBlocks() already maps this exact error to a RetryError (commit 7946278, "add retries for hl specific error"), but getLatestBlockhash() — the finalized-head probe — was not given the same handler.

Fix (tested)

Add the same validateError to the getLatestBlockhash call so "invalid block height" throws RetryError (retried by the client's built-in retry machinery — isConnectionError(RetryError) === true) instead of crashing. Other RPC errors still propagate as RpcError. One block re-fetch hits a healthy backend, so the probe converges. Minimal change, mirroring the established pattern.

Verification

• npx vitest --run finalized-head-retry in evm/evm-rpc: red → green.
  • Pre-fix: getLatestBlockhash('finalized') throws RpcError: invalid block height (reproduces the production crash at src/rpc.ts:113).
  • Post-fix: it retries and returns the head; a second test asserts non-transient errors (e.g. execution reverted) still surface.
• Full evm/evm-rpc suite: 154 passed, 27 skipped, 0 failed. rush build --to @subsquid/evm-rpc compiles clean.

Falsification

If the dump still crash-loops after deploy with eth_getBlockByNumber(["finalized", …]) raising an error whose message does not contain "invalid block height", or if the provider returns this error persistently (every attempt, so retries never converge and the dump hangs without progress), then the trigger is a different/persistent provider failure, not the transient one this handles.

Operator note (not part of this PR)

The trigger is uniblock (chainId=998) intermittently failing the finalized tag — https://status.uniblock.dev. A provider swap would remove today's trigger but is a temporary mitigation, not a durable fix (prior provider-swap PRs for this exact symptom — infra #578, #581 — were closed); this code change is the durable resolution so a single provider hiccup can no longer crash-loop the dump.

Related to #502 (recover from transient ForkException on the finalized dump) — same theme (don't crash-loop on transient finalized-stream errors), different cause: that PR handles a ForkException in util-internal-dump-cli and explicitly re-throws non-fork errors; this handles a plain RpcError in @subsquid/evm-rpc's finalized-head probe, which #502 does not catch.