Skip to content

Commit 94967eb

Browse files
bdchathamclaude
andauthored
perf(evmonly): parallelize OCC validation and merge behind the serial acceptance barrier (#4457)
Backport: this ports #4273, the reviewable version of #4261, to main from the giga-1 PR stack #4270–#4273. The stack targeted stacked giga-1 branches and never reached main; only its first PR landed on main, as #4366. @shemnon approved #4273 at 1143167. Since that commit, the changes are main's renames (`baseAccount`, the `phaseOCC*` constants, `takeAccessSets`) and two review fixes: merge fragments are cleared before they return to the pool, and each parallel pass's look-ahead is bounded. On giga-testnet-2, `occ_validate` and `occ_merge` took about a third of the executor main loop, all of it on the calling goroutine. Most of that time went to transactions that the frontier accepts as they stand. A block of about 1,800 transactions holds less than one conflict. On `giga-1`, #4261 moved this work onto the OCC pool. It cut validate and merge from about 8.5 ms to 4.1 ms per block. That gave about 20% more blocks per second during the 95k tx/s window. #4269 reverted it for the reviewable stack, and that stack (#4273) never reached main. This PR ports #4261 and the #4273 follow-up to main (PLT-1377). Acceptance stays a serial barrier in block order. `stateAccessIndex` and `blockSTMState` split into 64 address shards. `indexResults` records the writes of every speculative result up front, and `conflictsWithin(key, sourcePrefix, txIndex)` replaces `conflictsWithAfter`. `acceptValidatedPrefix` finds the first result that the serial frontier would not accept, and `applyRange` folds the run before it into the prefix shard by shard. `validateBlockSTMFrontier` then handles only that one result, and `serialBackoff` keeps dependency chains on the calling goroutine. Each pass looks ahead at most twice as far as the previous pass accepted, and at least 2,048 results, so the work of a block's passes stays linear in its size. `changeSetIntoParallel` builds the changeset of each shard on the pool and joins the shards in address order. Its per-shard base-row cache replaces `prefetchBaseAccounts`. The port needs nothing from #4258 or #4260; the only conflicts came from the `baseAccount` and phase-constant renames on main. Non-app-hash-breaking: changesets, receipts, and tx results stay byte-identical to main. Shards are contiguous ranges of the first address byte, so shard order is canonical address order. Each address lives in one shard, so the apply order in a shard is block order. The new index can only add conflicts, from stale writes of an earlier incarnation. An extra rerun executes against the exact accepted prefix, so it gives the sequential result, and the extra reruns show only in OCC metrics. Review `stmFrontierAccepts` most closely, because it must mirror `needsSTMRerun`, and `touchedShards`, because it must cover every address that `applyOwned` and `indexResults` touch. `TestOCCRandomizedConflictingBlocksMatchSequential` checks seeded dense and sparse blocks at 3 and 8 workers against the sequential executor, down to the encoded FlatKV pairs. A seeded digest run gave identical output on main and this branch for 864 blocks, including 3,000- and 6,000-transaction blocks that cross a pass's look-ahead. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 880848f commit 94967eb

9 files changed

Lines changed: 1378 additions & 248 deletions

File tree

‎giga/evmonly/README.md‎

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -211,10 +211,21 @@ are not enabled, the executor attempts optimistic parallel execution. Initial
211211
incarnations are split into execution ranges and run through the shared OCC
212212
worker pool against the base state. Worker fan-out is clamped to the amount of
213213
available work, so small blocks do not spawn idle workers and can still split
214-
down to one transaction per range. Validation then walks transaction order,
215-
comparing each incarnation's recorded balance, nonce, code, account, and
216-
`(address, slot)` storage reads/writes against writes accepted after that
217-
incarnation's source prefix.
214+
down to one transaction per range. Validation then accepts transactions in
215+
block order, comparing each incarnation's recorded balance, nonce, code,
216+
account, and `(address, slot)` storage reads/writes against writes recorded
217+
between that incarnation's source prefix and its own index.
218+
219+
Acceptance is the serial barrier, but the work behind it is spread across the
220+
pool: every incarnation's writes are indexed by transaction index up front, a
221+
parallel pass checks the pending run of transactions against that index and
222+
reports the first one the frontier would not accept, the accepted run is folded
223+
into the prefix shard by shard (shards are contiguous address ranges), and the
224+
frontier handles only the reported transaction on the calling goroutine. Each
225+
incarnation records the set of shards it touched, so a worker skips whole
226+
results that hold nothing of its own. The merge emits the changeset one shard at
227+
a time on the pool and concatenates the shards in order, which is canonical
228+
address order; a prefix with few keys is merged on the calling goroutine instead.
218229

219230
- transactions with no dependency on newly accepted prior writes are retained
220231
and accepted in block order without rerunning
@@ -240,6 +251,20 @@ reads or writes that balance, spends funds made available by that fee credit, or
240251
mixes a normal balance write with a fee credit to the same address, it is rerun
241252
against the updated prefix.
242253

254+
The parallel acceptance helps most when a block touches many distinct
255+
addresses with few conflicts, such as plain transfers between many accounts.
256+
It helps little, or costs a few percent, when one address dominates the block.
257+
A hot contract's storage or a heavily used address sits in one shard, so one
258+
worker does its apply and merge work. Dependency chains fall back to the serial
259+
stretch. Addresses with leading zero bytes all land in the first shard.
260+
261+
Local benchmarks of validate and merge per block show the range:
262+
263+
- a conflict-free block of 1,800 transactions: 20% faster than the serial path
264+
- a sparse block of 1,800 transactions: 11% faster
265+
- a dense block of 320 transactions: 3.8% slower
266+
- a block of transfers to one hot recipient: 1.7% slower
267+
243268
Speculative execution reuses scratch `nativeStateDB` instances from an executor
244269
pool. The returned receipts, logs, read/write sets, commutative balance deltas,
245270
and changesets are detached before the scratch state DB is reset. EVM snapshots

‎giga/evmonly/account_reader_test.go‎

Lines changed: 23 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package evmonly
22

33
import (
4+
"bytes"
45
"math/big"
56
"sync/atomic"
67
"testing"
@@ -78,48 +79,36 @@ func TestSnapshotReaderFetchesCodeOnlyWhenTheAccountHasSome(t *testing.T) {
7879
require.Equal(t, []byte{0x60, 0x00}, account.Code)
7980
}
8081

81-
// The merge reads every account it is about to compare against through the pool, then the serial
82-
// comparison finds them already resolved.
83-
func TestPrefetchResolvesEveryTouchedAccountOnce(t *testing.T) {
82+
// The merge compares balance, nonce and code against the same row, and resolves that row once per
83+
// touched account across the pool's workers.
84+
func TestParallelMergeResolvesEveryTouchedAccountOnce(t *testing.T) {
8485
snapshot := newMemoryGigaSnapshot(7)
85-
addrs := make([]common.Address, 0, minPrefetchedAccounts+8)
86-
for i := range minPrefetchedAccounts + 8 {
87-
addr := common.BigToAddress(big.NewInt(int64(i) + 1))
86+
addrs := make([]common.Address, 0, 512)
87+
for i := range 512 {
88+
addr := common.BigToAddress(new(big.Int).Lsh(big.NewInt(int64(i)+1), 150))
8889
snapshot.setBalance(addr, big.NewInt(int64(i)+1))
90+
snapshot.nonces[addr] = uint64(i) //nolint:gosec // i is non-negative.
8991
addrs = append(addrs, addr)
9092
}
9193
reading := &countingAccountView{memoryGigaSnapshot: snapshot}
9294

9395
state := newBlockSTMState(gigaSnapshotStateReader{snapshot: reading})
9496
for i, addr := range addrs {
95-
state.balances[addr] = big.NewInt(int64(i) + 100)
97+
state.shard(addr).balances[addr] = big.NewInt(int64(i) + 100)
98+
state.shard(addr).nonces[addr] = uint64(i) + 1 //nolint:gosec // i is non-negative.
9699
}
97-
state.prefetchBaseAccounts(t.Context(), newOCCWorkerPool(4))
98-
99-
require.Len(t, state.prefetched, len(addrs))
100-
for i, addr := range addrs {
101-
require.Equal(t, big.NewInt(int64(i)+1), state.prefetched[addr].Balance)
102-
}
103-
104-
// The comparison that follows reads the prefetched rows rather than the view.
105-
before := reading.reads.Load()
106-
base := newBaseAccounts(state.source, state.prefetched)
107-
for _, addr := range addrs {
108-
base.balance(addr)
100+
pool := newOCCWorkerPool(4)
101+
defer pool.Close()
102+
103+
var changes StateChangeSet
104+
require.NoError(t, state.changeSetIntoParallel(t.Context(), pool, &changes))
105+
106+
require.Equal(t, int64(len(addrs)), reading.reads.Load(), "each row must be read once")
107+
require.Equal(t, state.ChangeSet(), changes, "the parallel merge must match the serial one")
108+
require.Equal(t, int64(2*len(addrs)), reading.reads.Load(), "the serial merge reads each row once too")
109+
require.Len(t, changes.Balances, len(addrs))
110+
require.Len(t, changes.Nonces, len(addrs))
111+
for i := 1; i < len(changes.Balances); i++ {
112+
require.Negative(t, bytes.Compare(changes.Balances[i-1].Address[:], changes.Balances[i].Address[:]))
109113
}
110-
require.Equal(t, before, reading.reads.Load(), "the merge must not re-read what was prefetched")
111-
}
112-
113-
// Below the threshold the pool costs more than the reads it saves, so the merge reads them itself.
114-
func TestPrefetchIsSkippedForASmallBlock(t *testing.T) {
115-
snapshot := newMemoryGigaSnapshot(7)
116-
addr := testAddress(0xd4)
117-
snapshot.setBalance(addr, big.NewInt(5))
118-
reading := &countingAccountView{memoryGigaSnapshot: snapshot}
119-
120-
state := newBlockSTMState(gigaSnapshotStateReader{snapshot: reading})
121-
state.balances[addr] = big.NewInt(6)
122-
state.prefetchBaseAccounts(t.Context(), newOCCWorkerPool(4))
123-
124-
require.Nil(t, state.prefetched)
125114
}

‎giga/evmonly/cmd/evmonly-loadtest/main_test.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -951,6 +951,10 @@ func BenchmarkExecuteTransferBlock(b *testing.B) {
951951
name: "same_sender_nonce_chain",
952952
args: []string{"--same-sender"},
953953
},
954+
{
955+
name: "erc20_single_contract",
956+
args: []string{"--workload=" + workloadERC20Transfer},
957+
},
954958
}
955959
for _, tc := range tests {
956960
b.Run(tc.name, func(b *testing.B) {
@@ -965,7 +969,7 @@ func BenchmarkExecuteTransferBlock(b *testing.B) {
965969
require.NoError(b, err)
966970

967971
state := newGeneratedState()
968-
workload, err := scenarios.NewTransferWorkload(scenarioConfig(cfg), state)
972+
workload, err := scenarios.NewWorkload(cfg.workload, scenarioConfig(cfg), state)
969973
require.NoError(b, err)
970974
request, err := workload.BuildBlock(b.Context(), 1)
971975
require.NoError(b, err)

‎giga/evmonly/executor_test.go‎

Lines changed: 62 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import (
44
"context"
55
"crypto/ecdsa"
66
"errors"
7+
"math"
78
"math/big"
89
"sync"
910
"testing"
@@ -729,6 +730,64 @@ func TestExecutorOCCConflictingTransfersMatchSequential(t *testing.T) {
729730
require.Equal(t, big.NewInt(int64(txCount*3)), occState.GetBalance(recipient))
730731
}
731732

733+
// A block large enough for the parallel validation pass and the sharded merge, mixing independent
734+
// transfers, nonce chains from one sender, and a hot recipient, must produce the sequential result
735+
// exactly: transaction results, receipts, cumulative gas and the canonical changeset.
736+
func TestExecutorOCCLargeMixedBlockMatchesSequential(t *testing.T) {
737+
chainID := big.NewInt(testChainID)
738+
hot := testAddress(0xaa)
739+
seqState := NewMemoryState()
740+
occState := NewMemoryState()
741+
var rawTxs [][]byte
742+
sign := func(key *ecdsa.PrivateKey, nonce uint64, to common.Address, value int64) {
743+
rawTxs = append(rawTxs, signLegacyTxWithGasPrice(t, key, chainID, nonce, &to, big.NewInt(value), nil, 100_000, big.NewInt(1)))
744+
}
745+
fund := func(key *ecdsa.PrivateKey) {
746+
sender := crypto.PubkeyToAddress(key.PublicKey)
747+
seqState.SetBalance(sender, big.NewInt(1_000_000_000))
748+
occState.SetBalance(sender, big.NewInt(1_000_000_000))
749+
}
750+
for i := range 300 {
751+
key, err := crypto.GenerateKey()
752+
require.NoError(t, err)
753+
fund(key)
754+
switch {
755+
case i%7 == 0:
756+
sign(key, 0, hot, 3)
757+
case i%29 == 0:
758+
for nonce := range 3 {
759+
sign(key, uint64(nonce), common.BigToAddress(big.NewInt(int64(50_000+i))), 5) //nolint:gosec // nonce is non-negative.
760+
}
761+
default:
762+
sign(key, 0, common.BigToAddress(big.NewInt(int64(50_000+i))), 7)
763+
}
764+
}
765+
// A late transaction reads the hot balance that the earlier ones credit, so it is rerun after the
766+
// parallel pass has accepted the run before it.
767+
hotKey, err := crypto.GenerateKey()
768+
require.NoError(t, err)
769+
hotSender := crypto.PubkeyToAddress(hotKey.PublicKey)
770+
seqState.SetBalance(hotSender, big.NewInt(1_000_000_000))
771+
occState.SetBalance(hotSender, big.NewInt(1_000_000_000))
772+
seqState.SetBalance(hot, big.NewInt(1))
773+
occState.SetBalance(hot, big.NewInt(1))
774+
sign(hotKey, 0, hot, 1)
775+
776+
req := BlockRequest{Context: blockContext(chainID), Txs: rawTxs}
777+
seqResult, err := NewExecutor(Config{MinGasPrice: big.NewInt(0)}, withTestState(seqState)).ExecuteBlock(t.Context(), req)
778+
require.NoError(t, err)
779+
occResult, err := NewExecutor(Config{MinGasPrice: big.NewInt(0), OCCWorkers: 4}, withTestState(occState)).ExecuteBlock(t.Context(), req)
780+
require.NoError(t, err)
781+
782+
require.True(t, occResult.OCCStats.Attempted)
783+
require.False(t, occResult.OCCStats.Fallback, occResult.OCCStats.FallbackReason)
784+
require.Greater(t, occResult.OCCStats.RerunCount, uint64(0))
785+
require.Equal(t, seqResult.GasUsed, occResult.GasUsed)
786+
require.Equal(t, seqResult.Txs, occResult.Txs)
787+
require.Equal(t, seqResult.Receipts, occResult.Receipts)
788+
require.Equal(t, seqResult.ChangeSet, occResult.ChangeSet)
789+
}
790+
732791
func TestExecutorOCCFeePayingTransfersDoNotConflictOnCoinbase(t *testing.T) {
733792
chainID := big.NewInt(testChainID)
734793
txCount := 4
@@ -2114,7 +2173,7 @@ func TestStateDBGetCodeHashTracksCodelessAccountExistenceReads(t *testing.T) {
21142173
{kind: stateAccessBalance, address: eoa}: {},
21152174
})
21162175
validation := occValidationResult{}
2117-
accepted := validateSTMResultAgainstPrefix(&validation, writes, occTxExecution{gasLimit: 1, readSet: readSet}, 0, 10, 0)
2176+
accepted := validateSTMResultAgainstPrefix(&validation, writes, occTxExecution{gasLimit: 1, readSet: readSet}, 0, 10, 0, math.MaxInt)
21182177
require.False(t, accepted)
21192178
require.Equal(t, occFallbackReasonConflict, validation.fallbackReason)
21202179
}
@@ -2226,7 +2285,7 @@ func TestValidateSTMConflictMatrix(t *testing.T) {
22262285
t.Fatalf("unknown access mode %q", access)
22272286
}
22282287
validation := occValidationResult{}
2229-
accepted := validateSTMResultAgainstPrefix(&validation, writes, result, 0, 10, 0)
2288+
accepted := validateSTMResultAgainstPrefix(&validation, writes, result, 0, 10, 0, math.MaxInt)
22302289
return accepted, validation
22312290
}
22322291

@@ -2268,7 +2327,7 @@ func TestValidateSTMConflictSourcePrefix(t *testing.T) {
22682327
readSet: map[stateAccessKey]struct{}{key: {}},
22692328
gasLimit: 1,
22702329
gasUsed: 1,
2271-
}, 0, 10, sourcePrefix)
2330+
}, 0, 10, sourcePrefix, math.MaxInt)
22722331
return accepted, validation
22732332
}
22742333
cases := []struct {

0 commit comments

Comments
 (0)