Repository navigation
Commit 714ecac
fix(evmrpc): release eth_getLogs DB-read slots when a block read panics (#4388)
After the atlantic-2 v6.7 upgrade, RPC nodes began rejecting every
`eth_getLogs` call with `server I/O saturated, rejecting new request
(semaphore: 26/32 in use)` until they were restarted (PLT-1332).
`LogFetcher.processBatch` took a `dbReadSemaphore` slot for each
uncached block and released it by hand on each exit path, so a panic in
`BlockStore.LoadBlock` kept both the slot and the `DBSemaphoreAcquired`
gauge. The worker pool recovered the panic, and `GetLogs` refuses new
requests once 80% of slots are held, so the leaks added up to an outage.
The panicking batch also sent nothing to `errChan`, so the request that
hit it could succeed with that batch's logs missing.
The part of the read that holds a slot moves into `readUncachedBlock`,
which takes the slot through `acquireDBReadSlot` and defers the release
it returns, so the slot and the gauge come back on every exit, panics
included. `processBatch` now recovers a panic and reports it on
`errChan` like any other failed read, so the request fails instead of
returning partial logs. Because the worker pool no longer sees that
panic, `reportBatchPanic` logs it with its stack and counts it in
`TasksPanicked`. Freeing the slot alone would have ended the outage but
kept the partial results. The order of the cache check, bloom check and
block fetch is unchanged, and blocks still go to `res` after the slot is
freed.
No consensus or state impact. This is the block-by-block path of
`GetLogsByFilters`, which serves `eth_getLogs`, `eth_getFilterLogs`,
`eth_getFilterChanges` and log subscriptions on nodes whose receipt
store can't answer range queries, such as the pebble backend. The panic
itself will keep happening, since v6.7 can't load blocks stored before
the v6.4 commit-hash change: ranges over those heights now fail with an
error, and each panicking batch logs one error with a stack trace, which
may be noisy on archive nodes. The hand-release pattern dates back to
#2371, so `release/v6.7` and `release/v6.6` have the same leak, and this
likely wants a backport to `release/v6.7` before mainnet upgrades.
Both tests in `filter_db_read_slot_test.go` fail on `main` (the panic
escapes `processBatch`, and a three-batch request hangs on the leaked
slot) and pass with this change, and `go test -race ./evmrpc/...`
passes.
Co-authored-by: Cursor <cursoragent@cursor.com>1 parent e822b8f commit 714ecac
2 files changed
Lines changed: 194 additions & 51 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
1500 | 1501 | | |
1501 | 1502 | | |
1502 | 1503 | | |
1503 | | - | |
| 1504 | + | |
| 1505 | + | |
1504 | 1506 | | |
1505 | | - | |
| 1507 | + | |
| 1508 | + | |
| 1509 | + | |
| 1510 | + | |
| 1511 | + | |
| 1512 | + | |
1506 | 1513 | | |
1507 | | - | |
| 1514 | + | |
1508 | 1515 | | |
1509 | 1516 | | |
1510 | 1517 | | |
| |||
1520 | 1527 | | |
1521 | 1528 | | |
1522 | 1529 | | |
1523 | | - | |
1524 | | - | |
1525 | | - | |
1526 | | - | |
1527 | | - | |
1528 | | - | |
1529 | | - | |
1530 | | - | |
1531 | | - | |
1532 | | - | |
1533 | | - | |
1534 | | - | |
1535 | | - | |
1536 | | - | |
1537 | | - | |
1538 | | - | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
1539 | 1534 | | |
1540 | 1535 | | |
| 1536 | + | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
1541 | 1541 | | |
1542 | | - | |
1543 | | - | |
1544 | | - | |
1545 | | - | |
1546 | | - | |
1547 | | - | |
1548 | | - | |
1549 | | - | |
1550 | | - | |
1551 | | - | |
| 1542 | + | |
| 1543 | + | |
| 1544 | + | |
| 1545 | + | |
| 1546 | + | |
| 1547 | + | |
| 1548 | + | |
| 1549 | + | |
1552 | 1550 | | |
1553 | | - | |
1554 | | - | |
1555 | | - | |
1556 | | - | |
1557 | | - | |
1558 | | - | |
| 1551 | + | |
| 1552 | + | |
| 1553 | + | |
| 1554 | + | |
| 1555 | + | |
| 1556 | + | |
| 1557 | + | |
| 1558 | + | |
| 1559 | + | |
| 1560 | + | |
| 1561 | + | |
| 1562 | + | |
| 1563 | + | |
| 1564 | + | |
| 1565 | + | |
| 1566 | + | |
| 1567 | + | |
| 1568 | + | |
| 1569 | + | |
| 1570 | + | |
1559 | 1571 | | |
1560 | 1572 | | |
| 1573 | + | |
| 1574 | + | |
1561 | 1575 | | |
1562 | | - | |
1563 | | - | |
1564 | | - | |
1565 | | - | |
1566 | | - | |
1567 | | - | |
1568 | | - | |
1569 | | - | |
1570 | | - | |
1571 | | - | |
| 1576 | + | |
| 1577 | + | |
| 1578 | + | |
| 1579 | + | |
| 1580 | + | |
| 1581 | + | |
| 1582 | + | |
| 1583 | + | |
| 1584 | + | |
1572 | 1585 | | |
1573 | 1586 | | |
1574 | | - | |
1575 | | - | |
1576 | | - | |
1577 | | - | |
1578 | | - | |
| 1587 | + | |
| 1588 | + | |
| 1589 | + | |
| 1590 | + | |
1579 | 1591 | | |
| 1592 | + | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
1580 | 1619 | | |
1581 | 1620 | | |
1582 | | - | |
1583 | 1621 | | |
1584 | 1622 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
0 commit comments