Skip to content

EVM: query Ethereum, Polygon and Base for every EVM address - #17

Closed
Tarrant64 wants to merge 1 commit into
mainfrom
fix/evm-multichain-fanout
Closed

Tarrant64 wants to merge 1 commit into
mainfrom
fix/evm-multichain-fanout

Conversation

@Tarrant64

Copy link
Copy Markdown
Owner

Root cause

Wallets are stored one-row-per-(address, blockchain) (UNIQUE(user_id, address, blockchain)). utils/address.detect_blockchain always files a bare 0x... address as ethereum — only an explicit polygon:0x.../base:0x... prefix produces a different chain value. /nfts/polygon and /nfts/base (routers/nfts.py) only ever asked Alchemy about wallet rows whose own blockchain column already said that chain, so a real Polygon/Base holder whose row said ethereum was invisible to those pages.

Confirmed live (see ABCT-EVM-WALLET-COVERAGE-2026-09-28):

  • 15 of 15 tracked ethereum addresses have no matching polygon/base row.
  • Several of those 15 hold real, non-spam Polygon/Base NFTs today (Unstoppable Domains, Artificial Intelligence For Dummies, The Art of War, and others).
  • One wallet literally labeled "Stuff.io Base" is stored as ethereum and holds a real non-spam Base NFT that never shows.

Fix

Every EVM-family wallet (ethereum/polygon/base/bsc/arbitrum/avalanche) is now a candidate address for the Ethereum, Polygon, and Base NFT and balance/token fetchers, not just rows whose own column matches. New shared helpers in backend/services/alchemy_nft_utils.py:

  • EVM_ADDRESS_CHAINS — every EVM-shaped blockchain value (address donors).
  • EVM_FANOUT_TARGET_CHAINS = ('ethereum', 'polygon', 'base') — the chains actually queried per this fix (BSC/Arbitrum/Avalanche's own endpoints are unchanged/out of scope).
  • evm_fanout_wallets(wallets) — dedupes EVM-family wallets by address (case-insensitive), so an address registered under two different chain rows is only ever fetched/counted once.
  • wallet_fingerprint(addresses) — same pattern as services/solana_nft.py's existing fingerprint (from Fix NFT page showing only Ethereum: IPFS gateway fallback, show-more, Solana wallet changes #16), duplicated here as an independent copy for the EVM path.

NFTs: services/ethereum_nft.py, services/polygon.py, services/base.py now accept/build the fanned-out wallet list and add wallet-set-fingerprint cache invalidation (mirrors #16's Solana fix) — adding an EVM wallet is picked up immediately instead of waiting out the 24h in-memory / 30-day persistent cache. All 18 call sites in routers/nfts.py (every polygon/base wallet-filter line) now build the same fanned-out list consistently, which matters because the cache fingerprint compares whatever list each caller passes. Spam filtering, the persistent cache, and the #11 wipe-guard-on-provider-failure behaviour are all unchanged.

Balances/tokens: yes, the same gap exists — routers/wallets.py's balance refresh and routers/portfolio.py's summary are also gated on a wallet row's own blockchain column. New backend/services/evm_balance_fanout.py:

  • Computes fan-out candidates the same way as the NFT side, explicitly excluding any (address, target_chain) pair that already has a real DB wallet row — this is what prevents double counting when the same address is registered on two chains.
  • Fetches native balance + ERC-20 tokens via the existing single-address fetchers (services/ethereum.get_address_balance, services/polygon.get_address_info, services/base.get_address_info), with an asyncio.Semaphore(5) (same concurrency the manual bulk-refresh endpoint already uses).
  • Prices native units + tokens via the existing pricing_service (ticker match, contributes $0 and logs if unpriced — same pattern as calculate_wallet_native_assets_value).
  • Cached under the existing generic cache table for up to 1 hour, keyed by a fingerprint of the full EVM wallet set, so adding/removing an EVM wallet is picked up immediately rather than waiting out the TTL.
  • No DB schema change or migration. Nothing is written to balances/native_assets — those stay exactly one row per real wallet, owned by the background scheduler / manual refresh flow. The fan-out result is merged into the in-memory /portfolio/summary response only.

routers/portfolio.py::get_portfolio_summary merges the fan-out into each chain's total_eth/total_matic, token_count, and native_assets_value_usd (additive, after the existing per-wallet-row loop, before rounding — so it also flows into the existing portfolio_positions writer for cost-basis/history). It also adds three new, separately-visible fields per chain (fanout_extra_native, fanout_extra_value_usd, fanout_addresses_count) so the origin of the increase is auditable rather than folded silently into wallet_count (which stays "real registered wallets for this chain" — no synthetic wallet rows are added to the wallets list, to avoid confusing the wallet edit/delete UI, which is keyed by real wallet_id).

detect_blockchain and how wallets are stored are unchanged — no schema change, no data migration, per the task's constraint.

Effect on totals

  • Polygon total_matic / native_assets_value_usd: increases by the fanned-in native MATIC + ERC-20 token value from every ethereum/base-stored address that doesn't already have its own polygon row (all 15 ethereum + 2 base addresses today, per the coverage audit — 17 candidates).
  • Base total_eth / native_assets_value_usd: increases the same way from ethereum/polygon-stored addresses (17 candidates today).
  • Ethereum total_eth / native_assets_value_usd: increases from polygon/base-stored addresses that don't already have an ethereum row (4 candidates today).
  • No double counting: an address already explicitly registered on a target chain is excluded from that chain's fan-out candidates (unit-tested).
  • NFT counts (/nfts/polygon, /nfts/base, /nfts/ethereum, /nfts/all/summary) increase the same way, deduplicated by (chain, contract, tokenId) per chain's existing asset_id-keyed cache.

Call budget (Alchemy free tier)

Per the coverage audit's current wallet set (15 ethereum + 2 polygon + 2 base = 19 EVM addresses): NFT fan-out is ~19 addresses × 3 target chains = up to 57 getNFTsForOwner calls on a full (cache-miss) refresh, same as before per-chain but now also covering the previously-skipped chains — bounded by each chain's existing 24h in-memory / 30-day persistent NFT cache, refreshed early only on a wallet-set change. Balance fan-out adds up to 4 + 17 + 17 = 38 candidate (address, chain) pairs (each ~2 HTTP calls: native balance + alchemy_getTokenBalances, plus one alchemy_getTokenMetadata per token found — the same per-address cost the app already pays for real registered wallets), bounded by a new 1-hour cache independent of /portfolio/summary's own ~20-minute cache, so it does not re-hit Alchemy on every portfolio view. asyncio.Semaphore(5) caps in-flight balance fan-out requests.

What the user will see

  • The Polygon and Base NFT pages will show NFTs that are real but were previously invisible (including the "Stuff.io Base" wallet's Base NFT).
  • Portfolio Polygon/Base (and, less commonly, Ethereum) native balance and token totals will likely increase after this deploys, reflecting real holdings that were never queried before — this also flows into the historical portfolio_positions table used for cost basis, so a one-time step-up should be expected there, not a bug.
  • No wallet list changes — the Manage Wallets page is unaffected; the fanned-out coverage is total-only, with three new fields per chain in /portfolio/summary's JSON if the frontend wants to surface "N addresses without their own wallet contributed $X" later (not built in this PR — out of scope).

Tests

tests/unit/test_evm_fanout.py (14 new tests, mocked HTTP/DB/pricing, fake addresses built via string repetition so no literal address appears in source — same convention as test_alchemy_nft_free_tier.py):

  • an ethereum-stored address with Polygon NFTs appears on the Polygon fetch
  • the same address present as both an ethereum and a polygon row is fetched/counted exactly once (NFTs and balances)
  • spam is still filtered in the fanned-out fetch
  • wallet-set-change cache invalidation (adding a wallet is picked up without force_refresh)
  • balance fan-out sums native + priced tokens correctly, with no double counting
  • a provider failure on one target chain does not zero out or wipe the others

Full suite in a venv (bip_utils + full backend/requirements.txt, built from the existing abct-diag diagnostic image since this box has no python3-dev headers for a from-scratch venv): 546 passed, same 9 pre-existing failures, verified byte-identical (same 9 test names) on unmodified origin/main run in the same container — no new failures. (The task packet's stated baseline was 8; this environment reproducibly shows 9 on main itself, most likely due to a DB-migration/fixture difference in this ad hoc test container vs. wherever "8" was last measured — reported verbatim below rather than adjusted to match.)

FAILED tests/unit/test_mobile_top_holdings_coin_value.py::test_top_holdings_ada_value_excludes_other_native_tokens
FAILED tests/unit/test_mobile_top_holdings_coin_value.py::test_blockchains_list_keeps_combined_display_value
FAILED tests/unit/test_mobile_top_holdings_coin_value.py::test_non_cardano_chain_fully_itemized_gets_no_placeholder
FAILED tests/unit/test_mobile_top_holdings_coin_value.py::test_non_cardano_chain_partially_itemized_adds_placeholder_for_remainder
FAILED tests/unit/test_mobile_top_holdings_coin_value.py::test_two_chains_together_no_cross_chain_leakage
FAILED tests/unit/test_nft_image_ssrf.py::test_fetch_image_succeeds_for_a_normal_public_url
FAILED tests/unit/test_nft_toggle.py::test_instant_excludes_nfts_by_default
FAILED tests/unit/test_nft_toggle.py::test_instant_includes_nfts_when_enabled
FAILED tests/unit/test_nft_toggle.py::test_headline_and_history_move_by_the_same_nft_amount
9 failed, 546 passed, 16 warnings

sec/wallet_allowlist.txt-backed tests/unit/test_wallet_leak_check.py (26 tests, including test_real_repo_tree_is_clean over the whole repo) passes clean against this branch.

Security review

Manual pattern scan of the diff (secrets, private keys, PEM/AWS/GitHub/Stripe/Slack/JWT patterns, Blockfrost/Alchemy keys-in-URL, private IPs, home-dir paths, emails, literal 0x addresses) found nothing — no code in this diff touches credentials, and no real address appears anywhere (fake test addresses are built via string repetition, never written as a literal 42-char token, which is also what lets them pass the repo's own wallet-leak scanner without needing an allowlist entry). EXCLUDE//.gitignore compliance is untouched by this diff.

Out of scope (per task)

  • detect_blockchain / how wallets are stored — unchanged, no schema change, no migration.
  • BSC/Arbitrum/Avalanche's own dedicated endpoints — still gated on their own chain's wallet rows (their addresses ARE valid fan-out donors for Ethereum/Polygon/Base, per EVM_ADDRESS_CHAINS, but this PR does not fan Ethereum/Polygon/Base-stored addresses back into BSC/Arbitrum/Avalanche, since the user decision named Ethereum/Polygon/Base only).
  • Frontend surfacing of the new fanout_* fields — not built; the totals are already correct without it.

Task: ABCT-EVM-FANOUT-2026-09-28

🤖 Generated with Claude Code

Wallets are stored one-row-per-(address, blockchain), and detect_blockchain
always files a bare 0x address as 'ethereum' (only an explicit
polygon:0x.../base:0x... prefix gets a different chain). /nfts/polygon and
/nfts/base only ever asked about addresses whose own row already said that
chain, so a real Polygon/Base holder registered as 'ethereum' was invisible
to those pages -- confirmed live: 15 of 15 tracked ethereum addresses had no
matching polygon/base row, several hold real non-spam Polygon/Base NFTs
today, and one wallet literally labeled "... Base" was stored as ethereum.

Every EVM-family wallet (ethereum/polygon/base/bsc/arbitrum/avalanche) is
now a candidate address for Ethereum, Polygon, and Base NFT and native/
ERC-20 balance fetches, deduplicated by address so a wallet registered on
two chains is never fetched or counted twice. Keeps the spam filter,
persistent cache, and wipe-guard-on-provider-failure behaviour from #11/#16,
and extends #16's wallet-set-fingerprint cache invalidation to Ethereum/
Polygon/Base so adding an EVM wallet is picked up immediately instead of
waiting out a 24h/30-day cache.

Portfolio balance/token totals for Ethereum, Polygon, and Base now include
this fanned-out coverage too (new services/evm_balance_fanout.py), cached
separately for up to 1 hour to bound Alchemy call volume. No DB schema
change or migration; nothing is written to balances/native_assets;
detect_blockchain and how wallets are stored are unchanged.

14 new unit tests in tests/unit/test_evm_fanout.py (mocked HTTP/DB/pricing,
fake addresses built via string repetition, no live calls or real keys).
Full suite: 546 passed, 9 pre-existing failures verified identical on
unmodified origin/main in the same test environment (no new failures).

Task: ABCT-EVM-FANOUT-2026-09-28

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Tarrant64 added a commit that referenced this pull request Sep 29, 2026
Split out of PR #17 (fix/evm-multichain-fanout) after independent review
found its balance/token fan-out half wrong (native coin double-counted,
spam ERC-20s priced by raw on-chain symbol) while judging the NFT half
correct: http://192.168.50.240/data/2026-09-28/ABCT-PR17-REVIEW-2026-09-28.html

Wallets are stored one-row-per-(address, blockchain), and a bare 0x
address is always filed as 'ethereum' (utils/address.detect_blockchain
only assigns polygon/base from an explicit prefixed address). /nfts/polygon
and /nfts/base only ever asked Alchemy about addresses whose own row
already said that chain, so a real Polygon/Base holder registered as
'ethereum' was invisible. Every EVM-family wallet (ethereum/polygon/base/
bsc/arbitrum/avalanche) is now a candidate address for Ethereum, Polygon
and Base NFT fetches, deduplicated by address, with the existing spam
filter, persistent cache and wipe-guard-on-provider-failure behaviour
(#11/#16) plus the #16 wallet-set-fingerprint cache invalidation extended
to Ethereum/Polygon/Base.

Excludes services/evm_balance_fanout.py and the routers/portfolio.py
balance-total wiring entirely -- that half is being reworked separately.
Totals are unaffected by this PR.

8 unit tests in tests/unit/test_evm_fanout.py (mocked HTTP/DB, fake
addresses, no live calls or real keys).

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@Tarrant64

Copy link
Copy Markdown
Owner Author

Superseded by #18 (NFT fan-out only). The balance fan-out part was blocked in review (native coins double-counted; spam tokens priced by self-reported symbol) and is being redesigned separately.

@Tarrant64 Tarrant64 closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant