Skip to content

Commit 880848f

Browse files
bdchathamclaude
andauthored
perf(autobahn): reject an insert into a full producer mempool before CheckTx (#4451)
An insert into a full producer mempool ran the app CheckTx before it learned that the mempool had no room. CheckTx does ecrecover and writes the sender cache, so each such insert paid the full CheckTx cost and then failed. On a trapped giga-testnet-0 validator, about 4.4x more CheckTx attempts ran than inserts succeeded (`ok` 797/s, `pending_full` 3,476/s). That wasted CPU keeps a slow validator behind under saturation load. Linear: PLT-1371 (from PLT-1366). `State.doInsertTx` now calls `poolCanAdmit` after it gets the session mempool and before `checkTx`. That function takes the mempool lock once. It returns the error that the admission loop would return at that moment for an insert without a ticket. That is `ErrNotProducing` for a closed session, `errMempoolFull` for `TryInsertTx` when `IsFull()`, and `errPendingFull` for `InsertTx` when `len(waiters) >= maxPendingInserts()`. The errors are the same values, so `insertTx` counts them under the same `producer_inserts{result}` labels. For a valid tx, RPC callers see the same errors as before. The admission loop after CheckTx is unchanged and stays authoritative. The check also runs before the CheckTx permit wait, so a rejected insert does not hold a `checkTxSem` slot or a `tryInsertSem` slot for the CheckTx duration. Two points differ from the ticket text. The pending check omits `IsFull()`: with no ticket and `len(waiters) >= max >= 1`, `isHead` is false, so the loop rejects with `errPendingFull` whether or not the mempool is full. The closed check comes first, because the loop also checks `closed` first; without it, a closed and full session would report `full` where the loop reports `not_producing`. Error order. The order changes for a tx that is both invalid and sent to a full mempool. Before, it got its non-retryable CheckTx rejection, or `errTooLarge` from the gas checks. Now it gets the retryable `errMempoolFull` or `errPendingFull`. The client then retries a tx that can never land, and it learns that only when there is room. Under saturation this is the intended trade: CheckTx is the cost this change removes. `TestTryInsertTx_FullRejectsInvalidTxAsFull` pins the new order. Race analysis. Each early rejection is exactly the loop's answer at the moment of the check. After the check, other inserts can fill the mempool, and a prune can free it by advancing `m.first`. Thus an insert that the check rejects could have succeeded only if capacity freed during its own permit wait and CheckTx. One permit wait plus one CheckTx bounds that window. In that case the caller gets the documented, retryable mempool-full error, the same answer as with an instantaneous CheckTx. Under saturation, I expect other in-flight inserts to take the freed capacity, so throughput does not drop. A closed session never reopens, so the closed check has no race. The happy path takes one more short mempool lock (three instead of two). The `admit` and `check_tx` latency histograms get fewer samples, because early rejections skip both phases. Tests: three new tests in `checktx_limit_test.go` use a `countingApp`. They show that `TryInsertTx` on a full mempool, `InsertTx` with `MaxPendingInserts` queued, and both calls on a closed session return the expected result and do not call CheckTx. All three fail on `main`. The existing producer tests pass with no change. ``` go test -race -count=1 ./sei-tendermint/internal/autobahn/producer/... ok go test -race -count=20 ./sei-tendermint/internal/autobahn/producer/ ok gofmt -s -l / goimports -l (changed package) no output go vet ./sei-tendermint/internal/autobahn/producer/... no output staticcheck 2026.2.1 (0.8.1) ./sei-tendermint/internal/autobahn/producer/... no output golangci-lint v2.13.2 run ./sei-tendermint/internal/autobahn/producer/... 0 issues. golangci-lint v2.13.2 fmt --diff ./sei-tendermint/internal/autobahn/producer/... no output ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d4853e5 commit 880848f

2 files changed

Lines changed: 127 additions & 1 deletion

File tree

‎sei-tendermint/internal/autobahn/producer/checktx_limit_test.go‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,11 @@ import (
55
"errors"
66
"fmt"
77
"runtime"
8+
"sync/atomic"
89
"testing"
910

1011
abci "github.com/sei-protocol/sei-chain/sei-tendermint/abci/types"
12+
"github.com/sei-protocol/sei-chain/sei-tendermint/internal/autobahn/producer/metrics"
1113
"github.com/sei-protocol/sei-chain/sei-tendermint/internal/proxy"
1214
"github.com/sei-protocol/sei-chain/sei-tendermint/libs/utils"
1315
"github.com/sei-protocol/sei-chain/sei-tendermint/libs/utils/require"
@@ -202,6 +204,110 @@ func TestTryInsertTx_InFlightBounded(t *testing.T) {
202204
require.Equal(t, 2, len(env.state.UnconfirmedTxs()))
203205
}
204206

207+
// countingApp is a testApp that counts its CheckTx calls.
208+
type countingApp struct {
209+
*testApp
210+
checkTxCalls atomic.Int64
211+
}
212+
213+
func newCountingApp() *countingApp {
214+
return &countingApp{testApp: newTestApp()}
215+
}
216+
217+
func (a *countingApp) Proxy() *proxy.Proxy {
218+
return proxy.New(a)
219+
}
220+
221+
func (a *countingApp) CheckTx(ctx context.Context, req *abci.RequestCheckTxV2) *abci.ResponseCheckTxV2 {
222+
a.checkTxCalls.Add(1)
223+
return a.testApp.CheckTx(ctx, req)
224+
}
225+
226+
// TryInsertTx on a full mempool fails with errMempoolFull without running CheckTx.
227+
func TestTryInsertTx_FullSkipsCheckTx(t *testing.T) {
228+
ctx := t.Context()
229+
rng := utils.TestRng()
230+
app := newCountingApp()
231+
env := newTestEnv(rng, app.Cfg(), app.Proxy())
232+
_, err := env.fillMempool(ctx, rng, app.testApp)
233+
require.NoError(t, err)
234+
235+
calls := app.checkTxCalls.Load()
236+
_, err = env.state.TryInsertTx(ctx, env.fullTx(rng, app.testApp).encode())
237+
require.Equal(t, metrics.ResultFull, insertResult(nil, err))
238+
require.Equal(t, calls, app.checkTxCalls.Load())
239+
}
240+
241+
// On a full mempool, the capacity error wins over the app's rejection: an invalid tx
242+
// gets the retryable errMempoolFull, not the CheckTx response it would get with room.
243+
func TestTryInsertTx_FullRejectsInvalidTxAsFull(t *testing.T) {
244+
ctx := t.Context()
245+
rng := utils.TestRng()
246+
app := newCountingApp()
247+
env := newTestEnv(rng, app.Cfg(), app.Proxy())
248+
_, err := env.fillMempool(ctx, rng, app.testApp)
249+
require.NoError(t, err)
250+
tx := utils.GenBytes(rng, 1)
251+
_, err = decodeTxSpec(tx)
252+
require.Error(t, err)
253+
254+
calls := app.checkTxCalls.Load()
255+
resp, err := env.state.TryInsertTx(ctx, tx)
256+
require.Nil(t, resp)
257+
require.Equal(t, metrics.ResultFull, insertResult(nil, err))
258+
require.Equal(t, calls, app.checkTxCalls.Load())
259+
}
260+
261+
// InsertTx on a full mempool with MaxPendingInserts calls already queued fails with
262+
// errPendingFull without running CheckTx.
263+
func TestInsertTx_PendingFullSkipsCheckTx(t *testing.T) {
264+
ctx := t.Context()
265+
rng := utils.TestRng()
266+
app := newCountingApp()
267+
cfg := app.Cfg()
268+
cfg.MaxPendingInserts = 2
269+
env := newTestEnv(rng, cfg, app.Proxy())
270+
mp, err := env.fillMempool(ctx, rng, app.testApp)
271+
require.NoError(t, err)
272+
273+
require.NoError(t, utils.IgnoreCancel(scope.Run(ctx, func(ctx context.Context, s scope.Scope) error {
274+
if _, err := env.enqueueInserters(ctx, s, rng, app.testApp, mp, 2, context.Canceled); err != nil {
275+
return err
276+
}
277+
calls := app.checkTxCalls.Load()
278+
_, err := env.state.InsertTx(ctx, env.fullTx(rng, app.testApp).encode())
279+
if insertResult(nil, err) != metrics.ResultPendingFull {
280+
return fmt.Errorf("InsertTx over the bound: got %v, want errPendingFull", err)
281+
}
282+
if got := app.checkTxCalls.Load(); got != calls {
283+
return fmt.Errorf("CheckTx calls: got %d, want %d", got, calls)
284+
}
285+
s.Cancel(context.Canceled)
286+
return nil
287+
})))
288+
}
289+
290+
// Inserts into a closed session fail with ErrNotProducing without running CheckTx,
291+
// even when the mempool is also full.
292+
func TestInsertTx_ClosedSkipsCheckTx(t *testing.T) {
293+
ctx := t.Context()
294+
rng := utils.TestRng()
295+
app := newCountingApp()
296+
env := newTestEnv(rng, app.Cfg(), app.Proxy())
297+
mp, err := env.fillMempool(ctx, rng, app.testApp)
298+
require.NoError(t, err)
299+
for m, ctrl := range mp.inner.Lock() {
300+
m.close(ctrl)
301+
}
302+
303+
calls := app.checkTxCalls.Load()
304+
_, err = env.state.TryInsertTx(ctx, env.fullTx(rng, app.testApp).encode())
305+
require.ErrorIs(t, err, ErrNotProducing)
306+
_, err = env.state.InsertTx(ctx, env.fullTx(rng, app.testApp).encode())
307+
require.ErrorIs(t, err, ErrNotProducing)
308+
require.Equal(t, calls, app.checkTxCalls.Load())
309+
}
310+
205311
// An absent or zero limit resolves to at least one permit so inserts proceed.
206312
func TestConfig_MaxConcurrentCheckTxDefault(t *testing.T) {
207313
cfg := &Config{}

‎sei-tendermint/internal/autobahn/producer/mempool.go‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,23 @@ func (s *State) preReadEvmNonce(mp *mempool, addr common.Address) (utils.Option[
292292
return utils.Some(s.evmNonce(addr)), first, nil
293293
}
294294

295+
// poolCanAdmit returns nil when the pool would admit or queue an insert without a ticket right
296+
// now, and otherwise the error the admission loop would return, so such an insert skips CheckTx.
297+
// The admission loop rechecks after CheckTx.
298+
func (s *State) poolCanAdmit(mp *mempool, waitIfFull bool) error {
299+
for m := range mp.inner.Lock() {
300+
switch {
301+
case m.closed:
302+
return ErrNotProducing
303+
case !waitIfFull && m.IsFull():
304+
return errMempoolFull
305+
case waitIfFull && uint64(len(m.waiters)) >= s.cfg.maxPendingInserts():
306+
return errPendingFull
307+
}
308+
}
309+
return nil
310+
}
311+
295312
// checkTx runs the app CheckTx for tx, holding one of cfg.MaxConcurrentCheckTx permits
296313
// for the duration of the call. Waiting for a permit is cancelled with ctx.
297314
func (s *State) checkTx(ctx context.Context, tx tmtypes.Tx) (*abci.ResponseCheckTxV2, error) {
@@ -379,7 +396,7 @@ func (s *State) doInsertTx(ctx context.Context, tx tmtypes.Tx, waitIfFull bool)
379396
return nil, errTooLarge
380397
}
381398
// Reject / wait for a produce session before CheckTxSafe — IsFull and closed
382-
// are checked after, since they can change while CheckTx runs.
399+
// are checked again after, since they can change while CheckTx runs.
383400
var mp *mempool
384401
var err error
385402
if waitIfFull {
@@ -394,6 +411,9 @@ func (s *State) doInsertTx(ctx context.Context, tx tmtypes.Tx, waitIfFull bool)
394411
}
395412
mp = loaded
396413
}
414+
if err := s.poolCanAdmit(mp, waitIfFull); err != nil {
415+
return nil, err
416+
}
397417
resp, err := s.checkTx(ctx, tx)
398418
if err != nil {
399419
return nil, err

0 commit comments

Comments
 (0)