feat(evm): RPC-backed data source + fallback data source#510
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds two new additive EVM packages to enable preference-ordered data sources so processors can continue indexing through Portal outages: an RPC-backed source that matches Portal output, and a meta “fallback” source with health/freshness-based switching and exported metrics.
Changes:
- Register
@subsquid/evm-rpc-streamand@subsquid/evm-fallback-streaminrush.json, and move@subsquid/evm-rpc+@subsquid/evm-normalizationto the publishable version policy. - Implement
@subsquid/evm-rpc-streamby normalizing RPC blocks, decoding them via the Portal schema, and applying a ported filter engine; add per-network presets + parity tests. - Implement
@subsquid/evm-fallback-streamwith trinary health, lag/staleness freshness triggers, Prometheus gauges, and an EVM-specific convenience builder; add unit + live e2e tests.
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| rush.json | Registers new packages and adjusts version policies so required deps can be published. |
| evm/evm-rpc-stream/tsconfig.json | TypeScript build config for the new RPC-backed stream package. |
| evm/evm-rpc-stream/package.json | Declares package metadata, build/test scripts, and dependencies for RPC stream. |
| evm/evm-rpc-stream/src/index.ts | Public exports for rpc-stream (decode/filter/source/presets/builder). |
| evm/evm-rpc-stream/src/builder.ts | Convenience builder evmRpcStream(...) resolving network settings and wiring Rpc. |
| evm/evm-rpc-stream/src/source/data-source.ts | Core RPC-backed DataSource that normalizes, decodes via Portal schema, filters, and (when needed) projects output. |
| evm/evm-rpc-stream/src/source/augment.ts | Field-selection augmentation for evaluating where clauses on unselected fields. |
| evm/evm-rpc-stream/src/source/parity.e2e.test.ts | Live parity checks comparing RPC source output to Portal output (gated). |
| evm/evm-rpc-stream/src/networks/types.ts | Types for per-network presets (validation + method selection). |
| evm/evm-rpc-stream/src/networks/_families.ts | Reusable preset fragments (standard validation, OP-stack method toggles). |
| evm/evm-rpc-stream/src/networks/index.ts | Preset registry and resolver (preset ⊕ overrides). |
| evm/evm-rpc-stream/src/networks/ethereum-mainnet.ts | Ethereum mainnet preset. |
| evm/evm-rpc-stream/src/networks/optimism-mainnet.ts | Optimism preset (OP-stack). |
| evm/evm-rpc-stream/src/networks/base-mainnet.ts | Base preset (OP-stack). |
| evm/evm-rpc-stream/src/networks/arbitrum-one.ts | Arbitrum preset. |
| evm/evm-rpc-stream/src/networks/gnosis-mainnet.ts | Gnosis preset (trace API, adjusted validation). |
| evm/evm-rpc-stream/src/networks/polygon-mainnet.ts | Polygon preset (Bor-specific validation/finality). |
| evm/evm-rpc-stream/src/networks/cronos-mainnet.ts | Cronos preset (Ethermint-specific validation choices). |
| evm/evm-rpc-stream/src/networks/networks.test.ts | Golden tests asserting presets match documented infra config expectations. |
| evm/evm-rpc-stream/src/filter/request.ts | Request flattening + coarse “required data” derivation for RPC fetch toggles/capability needs. |
| evm/evm-rpc-stream/src/filter/request.test.ts | Unit tests for request flattening and required-data derivation. |
| evm/evm-rpc-stream/src/filter/filter.ts | Ported client-side filter engine and relation expansion on the flat block model. |
| evm/evm-rpc-stream/src/filter/filter.test.ts | Unit tests validating filter semantics and relation expansion. |
| evm/evm-rpc-stream/src/decode/decode.ts | Portal-schema-based decode pipeline for normalized RPC blocks. |
| evm/evm-rpc-stream/src/decode/portal-map.ts | Copied Portal mapping helpers to keep output shape identical. |
| evm/evm-rpc-stream/src/decode/shim.ts | Small wire-shim to satisfy Portal schema tagged-union expectations for traces. |
| evm/evm-rpc-stream/src/decode/shim.test.ts | Unit tests for the wire-shim transformations. |
| evm/evm-rpc-stream/src/decode/decode.lfs.test.ts | LFS-gated offline decode test on a real fixture. |
| evm/evm-fallback-stream/tsconfig.json | TypeScript build config for the fallback package. |
| evm/evm-fallback-stream/package.json | Declares fallback package metadata, scripts, deps, and peers. |
| evm/evm-fallback-stream/src/index.ts | Public exports for fallback, health, policy, metrics, and EVM convenience source. |
| evm/evm-fallback-stream/src/policy.ts | Fallback policy configuration, defaults, and AllSourcesDownError. |
| evm/evm-fallback-stream/src/health.ts | Trinary health state machine used by the fallback selector/driver. |
| evm/evm-fallback-stream/src/selector.ts | Selection logic for failover vs switch-up using trinary health. |
| evm/evm-fallback-stream/src/fallback.ts | Core fallback DataSource implementing switching and freshness checks. |
| evm/evm-fallback-stream/src/metrics.ts | Prometheus gauges exposing fallback state. |
| evm/evm-fallback-stream/src/evm/source.ts | EVM-specific helper building a FallbackDataSource<Block<F>> from Portal/RPC sources. |
| evm/evm-fallback-stream/src/fallback.test.ts | Unit tests covering selection, switching, forks, all-down, freshness, getHead, metrics snapshot. |
| evm/evm-fallback-stream/src/health.test.ts | Unit tests for health transitions and cooldown behavior. |
| evm/evm-fallback-stream/src/metrics.test.ts | Unit test asserting Prometheus exposition output contains expected gauges/labels. |
| evm/evm-fallback-stream/src/evm/source.e2e.test.ts | Live E2E test for Portal+RPC fallback and forced failover (gated). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
abernatskiy
added a commit
that referenced
this pull request
Jun 25, 2026
Wire up two half-finished pieces and correct two stale comments flagged in
the Copilot review:
- evm-fallback-stream entrypoint no longer crashes for Portal-only consumers.
The EVM glue's peer imports are now type-only and @subsquid/evm-rpc-stream
is loaded with a lazy require reached only on the `rpc` branch, so
require('@subsquid/evm-fallback-stream') touches neither optional peer.
Configuring an `rpc` source without them now fails with an actionable
message. @subsquid/evm-rpc joins the optional peers (the public config type
references its Rpc).
- Proactive probing is now actually wired. The head poll feeds onLivenessPass
on success (it is the liveness probe), and probeCapability is invoked once a
source is reachable, feeding onCapability. Eager switch-up no longer depends
on the freshness head-polls: each boundary proactively probes the
higher-preference standbys, so a recovered source reaches `healthy` and is
reclaimed even with maxLagBlocks/maxStalenessMs disabled. Two integration
tests cover liveness-driven recovery and capability-gated switch-up.
- Corrected the EvmRpcStreamDataSource / augmentFields comments that still
described the project-back step as unimplemented.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 25, 2026
- policy doc: AllSourcesDown → AllSourcesDownError (actual exported name). - metrics: rename the switch-count gauge from `_switches_total` to `_switches`; the `_total` suffix is reserved for Counters by Prometheus convention and this is a set-on-scrape Gauge. Test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
- evm-rpc-stream: add the `test:lfs` script (VITEST_LFS=1) so `rush test:lfs`
actually runs decode.lfs.test.ts, matching the repo convention.
- fallback: clear `activeIndex` while all sources are down, so metrics
(`${prefix}_active`) no longer report a source as active when none is
eligible. Switch-counting now keys off a retained `#lastActive`, so a
resume on a different source after the gap still registers as a switch and a
resume on the same one does not. Covers both runStream and delegateHead.
- Add Rush change files for the two new publishable packages so the
change-management/release flow recognises them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
Wire up two half-finished pieces and correct two stale comments flagged in
the Copilot review:
- evm-fallback-stream entrypoint no longer crashes for Portal-only consumers.
The EVM glue's peer imports are now type-only and @subsquid/evm-rpc-stream
is loaded with a lazy require reached only on the `rpc` branch, so
require('@subsquid/evm-fallback-stream') touches neither optional peer.
Configuring an `rpc` source without them now fails with an actionable
message. @subsquid/evm-rpc joins the optional peers (the public config type
references its Rpc).
- Proactive probing is now actually wired. The head poll feeds onLivenessPass
on success (it is the liveness probe), and probeCapability is invoked once a
source is reachable, feeding onCapability. Eager switch-up no longer depends
on the freshness head-polls: each boundary proactively probes the
higher-preference standbys, so a recovered source reaches `healthy` and is
reclaimed even with maxLagBlocks/maxStalenessMs disabled. Two integration
tests cover liveness-driven recovery and capability-gated switch-up.
- Corrected the EvmRpcStreamDataSource / augmentFields comments that still
described the project-back step as unimplemented.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
- policy doc: AllSourcesDown → AllSourcesDownError (actual exported name). - metrics: rename the switch-count gauge from `_switches_total` to `_switches`; the `_total` suffix is reserved for Counters by Prometheus convention and this is a set-on-scrape Gauge. Test updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
- evm-rpc-stream: add the `test:lfs` script (VITEST_LFS=1) so `rush test:lfs`
actually runs decode.lfs.test.ts, matching the repo convention.
- fallback: clear `activeIndex` while all sources are down, so metrics
(`${prefix}_active`) no longer report a source as active when none is
eligible. Switch-counting now keys off a retained `#lastActive`, so a
resume on a different source after the gap still registers as a switch and a
resume on the same one does not. Covers both runStream and delegateHead.
- Add Rush change files for the two new publishable packages so the
change-management/release flow recognises them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
force-pushed
the
feat/rpc-fallback
branch
from
55f3ef5 to
6837083
Compare
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
- loadRpcStream: only translate a MODULE_NOT_FOUND (peers not installed) into the "install the optional peers" hint; rethrow any real error thrown from inside the package so it isn't masked. - laggingTooFar: reset `lag` to 0 when there's no independent head reference, so metrics don't report a stale lag when it's currently not computable. - nextWithStaleness: clear `chainStalled` once a batch resolves or the source errors, so the global-stall flag can't stick after progress resumes or a failover. Asserted in the global-stall test. - evm-rpc-stream: drop unused deps @subsquid/logger and util-internal-hex. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
abernatskiy
added a commit
that referenced
this pull request
Jun 26, 2026
- loadRpcStream: narrow the "peers not installed" translation to a MODULE_NOT_FOUND that names @subsquid/evm-rpc-stream itself (matched by its quoted name in the message). A missing transitive dep inside the package names a different module and now surfaces unchanged instead of being masked by a misleading install hint. - Drop the unused @subsquid/util-internal dependency from evm-fallback-stream (only util-internal-data-source / util-internal-range are imported). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…parity) The RPC source streamed every block in the caller's window and only filtered those covered by a request range — so a block in a *gap* between non-contiguous ranges was emitted unfiltered (all logs/txs/traces/stateDiffs), which the Portal would never return, and which disagreed with getBlocksCountInRange (counts only request ranges). That broke the drop-in guarantee. Now getStream/getFinalizedStream iterate the request ranges intersected with the caller's [from, to] (like the Portal's one-query-per-range), via a standalone streamBoundedRanges() generator. Gaps are never streamed. parentHash is threaded through contiguous ranges (so the inner source's continuity/fork detection still works across a seam) and dropped across a gap; the caller's parentHash is kept for the first block, preserving fork-on-resume detection for the fallback. Unit-tested with a mock inner: gap-skipping, parentHash threading across contiguous ranges + reset across gaps, and window clipping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tream - setActive: on a real switch, clear lag/staleness/chainStalled/chainHead. They describe the active source, and the previous source's values (e.g. a non-zero staleness left when nextWithStaleness returns STALE) would otherwise linger in the gauges until the new source's next batch/head poll. - runStream: reset #lagArmed at the start of each stream. It latches on the instance once the tip is reached; a reused instance starting a later far-behind-head backfill must not inherit it and spuriously fail over. Tests: freshness reset on switch (empty standby so no boundary recomputes), and per-stream re-arming (same instance: tip-reaching stream then a backfill). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… frontier
Adds a generic `makeCapabilityProbe(source)` (any DataSource<B>): it fetches a
one-block slice of exactly the data the source is configured to serve — the
query is baked into the source, so a bounded getStream re-exercises the whole
pipeline — and reports whether the source could serve it. Catches the
reachable-but-incapable failures liveness alone misses (trace/debug_ API
disabled, pruned state at depth, missing receipts). Capable iff the slice yields
a batch or ends cleanly; a ForkException counts as capable; any other error or a
timeout is not-capable.
The supervisor anchors the probed block to the indexing frontier, not the chain
tip: `target = clamp(lastCommitted + capabilityLookahead, 0, head - capabilityTipMargin)`.
During a backfill this verifies the source can serve at the depth it is about to
read in bulk; when caught up it lands on a settled block off the reorg-prone tip.
The callback receives the resolved block (`probeCapability(atBlock)`), so it
needs no head of its own — the supervisor reuses the head it already polled.
createEvmFallbackSource attaches the probe to every source by default
(`capabilityProbe?: false | {timeoutMs}` to opt out / tune). lookahead and
tipMargin default to 16, configurable on the policy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A source with a capability probe could flap back to `healthy` on liveness alone after going unhealthy: `#capabilityOk` was sticky-true once confirmed, so `#toUnhealthy()` left it set and recovery needed only `M` liveness passes. That is exactly the churn loop a probe is meant to prevent — a node that stays reachable but keeps failing the real query (e.g. a Portal returning HTTP 400 on a query that passed type-level validation, or an RPC node with traces disabled) would recover, get re-promoted, fail again. Store `hasCapabilityProbe` and drop `#capabilityOk` on `#toUnhealthy()` for probed sources, so health can never return to `healthy` until a fresh probe succeeds. Probe-less sources are unaffected (capability stays always-confirmed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… + metrics
Since the fallback relies on sources *throwing* to detect incapability, the
supervisor never knows the error type up front. New `diagnostics.ts` normalizes
any thrown value into a `SourceErrorInfo` by duck-typing the conventional fields
the SDK already stamps (`addErrorContext`'s `rpcUrl`/`rpcMethod`/`rpcParams`;
`HttpError.response.{status,url}`; `RpcError.code`; transport error names):
- `check` (stream|liveness|capability) + `reason` (http|rpc|timeout|connection|
stale|lag|fork|unknown) + `code` — bounded, safe as metric labels.
- `detail` — full text incl. the request, for logs only; URLs are run through a
credential redactor (drops userinfo/query, masks `sqd_` tokens) so API keys
never reach logs.
Wiring:
- The capability probe now returns `{ok, cause?}` instead of a bare boolean.
- `SourceHealth` stores the cause; `FallbackDataSource.failSource()` feeds it to
health and logs (via an injectable `logger`, default `console.warn`) only on a
real transition to unhealthy.
- The `source_health` Prometheus gauge gains `check`/`reason`/`code` labels on
the unhealthy row (empty otherwise); the gauge is reset per scrape so stale
cause series don't linger. The request body is never a label.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ected one
Sibling data packages don't take an injectable logger — `*-stream` packages don't
log at all, and the ones that do (`@subsquid/evm-rpc`, `evm-processor`) create it
statically with `createLogger('sqd:…')`. So drop the `logger?` option (and the
ad-hoc `FallbackLogger`/`consoleLogger` shim) and create the cause logger
statically as `createLogger('sqd:evm-fallback-stream')`, matching the namespace
convention (`sqd:evm-rpc`, `sqd:evm-data-service`). The cause is still also exposed
through `metrics()` for callers preferring a metrics surface.
Tests raise the `sqd:evm-fallback*` level to ERROR via `setLogLevelCallback` so the
expected unhealthy-transition WARNs stay out of the test output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… stream `delegateHead()` (backing `getHead`/`getFinalizedHead`) classified a failed head fetch as `check: 'stream'`, marking the source unhealthy on the first error. But a head query is the liveness signal everywhere else (`getCachedHead`), and head + stream share one `SourceHealth` — so a single transient head blip would condemn the source (cooldown) and needlessly switch an otherwise-healthy stream. Classify it as `'liveness'` so it goes through the K-consecutive-fail threshold and a blip just retries the same source instead. (Per PR #510 review.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…robing Remove the `churnOnGlobalStall` policy knob. On a global stall every source sits at the same head, so there is nowhere fresher to switch to — churning through equally-stuck sources is pointless. The supervisor now always holds the active source in that case (the former `false` default). The hold is not passive: each time staleness re-trips, `chainHeadOthers` re-polls the other sources, which doubles as their liveness/capability probe — so a held supervisor keeps watching for recovery and leaves the hold as soon as any of: the active source delivers the next batch, the active source errors (failover), or another source advances past us (`others > lastNumber` ⇒ STALE). Added test (d2) asserting the held source keeps probing (liveness + capability) and recovers by switching to a source that becomes fresher, without the active ever resolving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per PR #510 review: the EVM glue's types reference the optional peers, so the package `.d.ts` does too. Spell out that this is type-check-only (runtime and all transpile-only runners are unaffected), neutralized by `skipLibCheck: true` (our repo/template default), and only surfaces under `skipLibCheck: false` with the peers omitted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`clearActive()` zeroed `activeIndex` when no source is eligible but left `lag`/`staleness`/`chainStalled`/`chainHead` reporting the last active source's freshness, contradicting the "these gauges describe the active source" contract. Clear them too, and cover it with an all-down test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…510 review) `#lagArmed` is meant to arm once the active source reaches the tip, so a deep backfill never trips lag failover. But the arming condition `lag <= maxLagBlocks` also armed on a *negative* lag — when the independent reference (max head of the other sources) is itself behind the active source's last committed block. A stale/lagging standby would then arm the latch mid-backfill, and once that standby recovered to the real tip its head jump would trip a spurious failover. Gate arming on `lag >= 0` so it only arms when the active is genuinely at/near the tip (within threshold and not ahead of the reference). Added a regression test: a standby first behind us, then far ahead, must not cause a failover. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The doc comments referenced an internal planning doc (`CHAIN_SPECIFIC.md` §2 / §4.1) that was never committed, leaving consumers with broken pointers. Describe the "C1" per-chain config concept inline instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the trace-relation helper (and its call site) to the standard spelling so the misspelling doesn't propagate to future uses/searches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ction Four issues found reviewing my own PR: 1. [availability] A sick standby whose `getHead` hangs (TCP up, no response) could stall the healthy active source, because head polls are awaited on the batch-critical path (lag check every batch, staleness hold, switch-up). Add `headPollTimeoutMs` (default 500ms; null defers to the client's own timeout) and time-box each poll — a timeout counts as a liveness failure for that standby and never blocks the primary. 2. [observability] The active source's capability probe never runs (it only fires for standbys), so a cold-start primary served forever yet stayed `unknown`. A delivered batch proves capability (an incapable source throws rather than yields), so confirm capability on each batch — the primary now reaches `healthy`. 3. [correctness] Project-back matched items by a synthesized structural key, which collides for items that share one (block-reward traces carry no transactionIndex), projecting the wrong one. Replace with position/identity matching against the pre-filter decode (`keptByPosition`) — the two decodes align 1:1 by index. 4. [cleanup] Drop the dead `transactions` toggle from `RequiredData`/`toRequiredData` (and now-unused `TX_FIELDS`): the RPC source always fetches full transactions, so it could never be false. Comment left explaining why. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold @subsquid/evm-rpc-stream and @subsquid/evm-fallback-stream into a new @subsquid/squid-sdk package as the /evm-rpc-stream and /evm-fallback-stream subpaths, removing the standalone packages. Re-export the core *-stream and util packages (evm-stream, solana-stream, fuel-stream, starknet-stream, the util-internal family, logger, portal-client, http-client, rpc-client, util-timeout) as additional subpaths, kept standalone. Subpath resolution ships both an `exports` map and a `typesVersions` map so imports type-check under classic `moduleResolution: node` (the standard squid template's tsconfig) as well as node16/nodenext/bundler; Node honors `exports` at runtime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
…nal EVM peers - Hierarchical subpaths replacing flat old-name slugs: evm, evm/rpc, evm/fallback, solana, fuel, starknet, fallback, util, util/<x>, client/<x>, logger. - Extract the VM-agnostic fallback supervisor into its own `fallback` subpath; the EVM binding (createEvmFallbackSource) lives at `evm/fallback`. - Make @subsquid/evm-rpc and @subsquid/evm-normalization optional peer dependencies. The fallback core has no EVM in its runtime graph, and evm/fallback requires evm/rpc lazily, so a Portal-only/non-EVM squid needs neither peer. exports + typesVersions updated for the new tree; verified type-checking under classic (template) and nodenext resolution and runtime require() for all 19 subpaths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
The @subsquid/squid-sdk root doc comment still showed the pre-restructure subpaths (/evm-fallback-stream, /util-internal). Point them at the actual exports (/evm/fallback, /util). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
Rename the project folder sdk/squid-sdk -> meta/squid-sdk and update its Rush projectFolder registration. No package contents change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
Making @subsquid/evm-rpc and @subsquid/evm-normalization optional peers means
the lazy require('../rpc') in the EVM fallback binding can now fail with a raw
MODULE_NOT_FOUND when a squid configures an 'rpc' source without installing the
peers. Reinstate the actionable guard (dropped during the fold-in, when the RPC
stack was still a hard dep) — now keyed on the two optional peers — and fix the
stale "always resolves" doc comment. Extracted into a testable helper with unit
coverage for translate-vs-passthrough.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
A capability probe that reports not-capable is not a freshness failure, but
both the supervisor's no-cause path and the probe helper's timeout path built
their SourceErrorInfo via freshnessFailure(..., 'stale', ...), surfacing a
misleading `stale` reason in logs/metrics. Add a capabilityFailure() builder
(reason defaults to `unknown`, or an accurate category) and use it: `unknown`
for a bare {ok:false}, `timeout` for a probe that outran its timeout. Tests
updated/added accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
The supervisor is VM-agnostic (@subsquid/squid-sdk/fallback) but still logged under the fold-in-era sqd:evm-fallback-stream namespace, which misleads log filtering/alerting for non-EVM users. Use sqd:fallback (matching the repo's sqd:<component> convention) and update the test's log-level callback to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
@subsquid/starknet-stream@1.2.0 is uninstallable from npm: it requires @subsquid/starknet-rpc@^0.0.2, but starknet-rpc is a docker-policy package and only 0.0.1 is published — so pulling starknet-stream into the umbrella broke `npm install @subsquid/squid-sdk` with ETARGET. Remove the `starknet` subpath (dependency, export, typesVersions, barrel, README) and bump to 0.0.2. The solana and fuel streams install cleanly and are kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
.npmrc was accidentally added in the previous commit. Its content is only the
`${NPM_TOKEN}` placeholder (no secret), but it is a local publishing artifact
and must not be tracked. Untrack it and add it to .gitignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
Test-first refactors from the PR #510 code-quality review (behavior-preserving except finding 1, which removes redundant RPC calls without changing output): - (1) evm/rpc: apply the logs↔receipts de-duplication on the aggregate of all requests, not per-request. `toRequiredData` now reports the raw `logs` need and the new exported `unionRequiredData` applies `logs &&= !receipts` once, so a multi-request config (one needs logs, another needs receipts) no longer fires a redundant eth_getLogs every stride. Regression test added (verified failing pre-fix). - (7) evm/rpc: `augmentFields` returns {fields, grew} with an in-place grew flag, deleting the second structural-diff pass (`selectionGrew`/`SELECTION_TYPES`). - (4) fallback: extract the all-down/failover preamble duplicated between `runStream` and `delegateHead` into a `selectActive()` async generator. - (5) fallback: collapse the identical freshness-gauge reset in `setActive`/ `clearActive` into `resetFreshnessGauges()`. - (6) fallback: extract the hand-rolled timeout-race and fire-and-forget iterator close into shared `withTimeout`/`safeReturn` (fallback/async.ts), reused by `headWithTimeout` and the capability probe; `sleep` now delegates to `delay`. - (2) packaging: in-package guard test asserting exports/typesVersions/barrels stay in sync (verified catching a dropped typesVersions entry). - (3) packaging: add the `./package.json` subpath export (an exports map is a hard allowlist). Bump to 0.0.3 + changefile. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
…ilot #510) Two log-hygiene leaks flagged by Copilot review: - redactUrl left the URL fragment (#...) intact; it can carry a token and rode through url.toString() into logs. Clear url.hash. - redactText matched only http(s)://, so ws:// / wss:// endpoints quoted in JSON-RPC provider errors leaked unredacted. Widen the pattern to wss?. Tests extended: fragment stripping, ws:// credential redaction, and a redactText block (previously untested). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
… (Copilot #510) A custom `probeCapability` that throws synchronously (before returning a Promise) escaped `#maybeProbeCapability` past the `#capabilityProbing[i] = true` line, so the flag was never cleared — permanently blocking future probes for that source (and it was mislabeled a liveness blip via getCachedHead's catch). Route the call through `Promise.resolve().then(() => probe(target))` so a sync throw becomes a rejection handled by the existing `.then(_, e => failSource('capability', e))` and the `.finally` always clears the flag. Regression test verified failing pre-fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
…oc (Copilot #510) - delegateHead marked head-query failures as liveness but never recorded a pass on success, so with "K consecutive liveness fails" semantics a transient blip stayed counted across intervening successes and a later blip could wrongly trip livenessFailThreshold. Call onLivenessPass() on success (mirrors getCachedHead). Regression test verified failing pre-fix (a non-consecutive second blip condemned the source and forced a failover). - Qualify the streamBoundedRanges doc: the caller's parentHash is preserved for the first block only when the stream starts within the first request range (no leading gap); a stream starting inside a gap drops it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
…pilot #510) metrics.ts re-declared MetricsSink even though @subsquid/util-internal-processor-tools exports an identical interface (its `addMetricsSink` consumes it). Import and re-export the upstream type — a single source of truth, no drift. Imported as a *type*, so it's erased at build and the VM-agnostic fallback module keeps its zero runtime dependency on processor-tools (verified: no require in the emitted metrics.js). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
…llision (Copilot #510) `const exports = pkg.exports` shadows the CommonJS module wrapper's `exports` param and would be a redeclaration SyntaxError if the file ran as CJS. Rename to `exportsMap`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2n1VHrhUa2szzVMW3mrkY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A processor today has a single data source — the Portal. When the Portal is unhealthy, indexing stops. This PR lets a processor fall back to RPC (or, more generally, to any of N preference-ordered sources) so it keeps making progress. The goal for v0 is no downtime, not perfectly flat freshness; EVM only.
Two capabilities make that possible. They ship as subpaths of a new umbrella package,
@subsquid/squid-sdk(see Packaging) rather than as standalone packages:@subsquid/squid-sdk/evm/rpc— an RPC source whose output matches the Portal'sA fallback only works if switching between sources is invisible to everything downstream — same block model, same fields, same fork semantics. The hard part is making RPC produce exactly what the Portal produces.
Rather than re-deriving that mapping (and forever chasing parity), this reuses the Portal's own decoder and the proven filtering logic, so RPC output matches by construction rather than by best effort. A processor can drop this in wherever a Portal source goes; nothing else changes. Per-network presets supply the small amount of chain config that can't be auto-detected, so the output matches the published dataset.
It depends on
@subsquid/evm-rpc/evm-normalizationonly optionally — Portal-only users pull in nothing extra.@subsquid/squid-sdk/evm/fallback(over the VM-agnostic@subsquid/squid-sdk/fallback) — the fallback itselfA meta-source that drives the lowest-index healthy source and, on failure, resumes the next one from the last committed position. The supervisor itself is VM-agnostic (exposed at
@subsquid/squid-sdk/fallback);evm/fallbackis a thin EVM binding (createEvmFallbackSource) over it. Two design choices carry the weight:{number, hash}and the sources are stateless, a fork that straddles a switch is just an ordinary reorg — it flows through the processor's existing rollback path. No new fork machinery.Health and switch state are exported as metrics so an unhealthy or lagging source is visible.
Does it actually match the Portal?
Yes — verified live against Ethereum mainnet: transactions, logs, and traces decode identically to the Portal's output, and a
[Portal, RPC]fallback both streams a range and fails over mid-stream when the primary dies. (State diffs match up to a small, documented node-version difference, not a config one.) Offline suites cover the filtering, health/failover, and per-network presets.Packaging
The RPC source and the fallback are folded into a new umbrella package,
@subsquid/squid-sdk(atmeta/squid-sdk), exposed via subpaths (evm/rpc,evm/fallback,fallback). The previously-standalone@subsquid/evm-rpc-streamand@subsquid/evm-fallback-streampackages are removed / not published — their source now lives in the umbrella. The umbrella also re-exports the core*-streamandutilpackages as subpaths (those stay published standalone). Subpaths ship both anexportsmap and atypesVersionsmap, so they type-check under a squid's classicmoduleResolution: nodeas well asnode16/nodenext/bundler; Node honorsexportsat runtime.@subsquid/evm-rpcand@subsquid/evm-normalizationare optional peer dependencies — a Portal-only or non-EVM squid installs neither and never loads the RPC stack (the EVM bindingrequiresevm/rpclazily, translating a missing peer into an actionable error).Newly published packages:
@subsquid/evm-rpc,@subsquid/evm-normalization,@subsquid/squid-sdk.rush.jsonmoves@subsquid/evm-rpc/evm-normalizationfrom thedockerversion policy tonpmso the release workflow publishes them. This does not affect the ingestion docker images — they bundle these packages viarush deploy's dependency graph, independent of version policy.Related
The Pipes SDK gets the equivalent capability in two stacked PRs (the finalized-watermark prerequisite, then the RPC source + fallback). The two SDKs share no code — only test scenarios.