Skip to content

perf(trie): batch storage proof jobs at worker level (experiment) #20210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yongkangc wants to merge 16 commits into from
Closed

perf(trie): batch storage proof jobs at worker level (experiment) #20210

yongkangc wants to merge 16 commits into from
+1,334 −754

Conversation

@yongkangc
Copy link
Member

@yongkangc yongkangc commented Dec 9, 2025 •
edited
Loading

Problem

When proof tasks queue up in the worker pool (workers busy), each worker processes tasks one at a time. Multiple tasks for the same account result in redundant trie traversals.

Fixes #20088

Solution

Implement worker-level batching for both storage and account proofs:

  • When a worker receives a job, it drains additional queued jobs
  • Storage proof jobs for the same account are merged (prefix_sets and target_slots combined)
  • Account multiproof jobs are merged (targets and prefix_sets combined)
  • A single proof computation is performed for merged inputs
  • Results are sent to all original requestors

Changes

  • Add `BatchedStorageProof` struct to hold merged storage proof inputs
  • Add `BatchedAccountProof` struct to hold merged account multiproof inputs
  • Add `StorageWorkerBatchMetrics` and `AccountWorkerBatchMetrics` for tracking batch sizes
  • Modify `StorageProofWorker::run()` to batch consecutive storage proof jobs
  • Modify `AccountProofWorker::run()` to batch consecutive account multiproof jobs
  • Add `process_batched_storage_proof()` and `process_batched_account_multiproof()` functions
  • Add `Clone` derive to `ProofResult` for result sharing

Expected Impact

Reduced trie I/O and computation overhead when proof requests queue up, especially beneficial under high transaction throughput where state updates arrive faster than proof computation.

@yongkangc yongkangc added the C-perf A change motivated by improving speed, memory usage or disk footprint label Dec 9, 2025
@yongkangc yongkangc requested a review from Rjected as a code owner December 9, 2025 03:24
@yongkangc yongkangc added the A-trie Related to Merkle Patricia Trie implementation label Dec 9, 2025
@github-project-automation github-project-automation bot moved this to Backlog in Reth Tracker Dec 9, 2025
@yongkangc yongkangc marked this pull request as draft December 9, 2025 04:23
@yongkangc yongkangc changed the title perf(trie): batch storage proof jobs at worker level perf(trie): batch storage proof jobs at worker level (experiment) Dec 9, 2025
@yongkangc
Copy link
Member Author

yongkangc commented Dec 9, 2025

Flow Diagram: https://gist.github.com/yongkangc/a6652bd0da2f0e4e6242b38640c454eb

github-actions bot and others added 15 commits December 9, 2025 11:32
@github-actions @github-merge-queue @yongkangc
chore(deps): weekly cargo update (#20174)
e7bd168 
Co-authored-by: github-merge-queue <[email protected]>
@shekhirin @yongkangc
ci: run on ubuntu instead of reth runner (#20196)
0b07657
@klkvr @shekhirin
@mattsse @yongkangc
feat: parallelize recovery (#20169)
9265700 
Co-authored-by: Alexey Shekhirin <[email protected]>
Co-authored-by: Matthias Seitz <[email protected]>
@mattsse @shekhirin @yongkangc
fix: make inserted blocks part of fcu canonical (#20164)
3094cd6 
Co-authored-by: Alexey Shekhirin <[email protected]>
@forkfury @mattsse @yongkangc
feat(primitives-traits): add recover_transactions_ref to avoid cloning (
0cbd2ae 
#20187)

Co-authored-by: Matthias Seitz <[email protected]>
@mattsse @yongkangc
fix: trace filter range off by one (#20199)
c0d73fa
@mattsse @yongkangc
perf: avoid duplicate storage get call (#20180)
12fdca7
@joshieDo @yongkangc
fix: set merkle changesets distance minimum to 128 (#20200)
b6dca7b
@lean-apple @yongkangc
test(era): complete int tests with roundtrip mainnet era files (#20064)
a430fb6
@futreall @yongkangc
fix(rpc): return error if toBlock exceeds current head (#20202)
388507b
@0xKarl98 @yongkangc
feat: add StorageSettings for StoragesHistory in RocksDB (#20154)
88dfd15
@klkvr @yongkangc
feat: add helper method to eth validator (#20206)
bb4285c
@yongkangc
perf(trie): batch proof jobs at worker level
394b0d5 
Implement worker-level batching for both storage and account proofs to
reduce redundant trie traversals when multiple proof requests queue up.

When proof requests arrive faster than workers can process them, jobs
for the same account (storage proofs) or consecutive jobs (account proofs)
are now merged into single proof computations.
@yongkangc
perf(trie): use Arc for proof sharing and add batch validation
1a5d9a3 
- Wrap DecodedMultiProof and DecodedStorageMultiProof in Arc within ProofResult
  for O(1) cloning when sending batched results to multiple receivers
- Add debug assertions in BatchedStorageProof::merge and BatchedAccountProof::merge
  to validate that all batched jobs share the same Arc for multi_added_removed_keys
  and missed_leaves_storage_roots (critical invariants for correctness)
- Unwrap Arc at extraction sites using try_unwrap for zero-cost when sole owner
@yongkangc
fix: strengthen batch invariants and prevent blinded node starvation
9bd5a3e 
- Change debug_assert to assert for multi_added_removed_keys Arc equality
  check in BatchedStorageProof::merge, ensuring incorrect proofs are
  caught in release builds, not just debug

- Change BatchedAccountProof::merge to try_merge returning Result, properly
  handling incompatible caches by processing as separate batches instead
  of panicking

- Add MAX_DEFERRED_BLINDED_NODES (16) limit to prevent starvation of
  blinded node requests under high proof load - stops batching early when
  limit reached

- Pre-allocate deferred_blinded_nodes vectors with capacity

- Remove unnecessary clone of storage_work_tx by taking reference
@yongkangc yongkangc closed this Dec 12, 2025
@github-project-automation github-project-automation bot moved this from Backlog to Done in Reth Tracker Dec 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mediocregopher mediocregopher Awaiting requested review from mediocregopher mediocregopher is a code owner

@shekhirin shekhirin Awaiting requested review from shekhirin shekhirin is a code owner

@Rjected Rjected Awaiting requested review from Rjected Rjected is a code owner

@gakonst gakonst Awaiting requested review from gakonst gakonst will be requested when the pull request is marked ready for review gakonst is a code owner

@DaniPopes DaniPopes Awaiting requested review from DaniPopes DaniPopes will be requested when the pull request is marked ready for review DaniPopes is a code owner

@mattsse mattsse Awaiting requested review from mattsse mattsse will be requested when the pull request is marked ready for review mattsse is a code owner

@joshieDo joshieDo Awaiting requested review from joshieDo joshieDo will be requested when the pull request is marked ready for review joshieDo is a code owner

@rakita rakita Awaiting requested review from rakita rakita will be requested when the pull request is marked ready for review rakita is a code owner

@klkvr klkvr Awaiting requested review from klkvr klkvr will be requested when the pull request is marked ready for review klkvr is a code owner

@fgimenez fgimenez Awaiting requested review from fgimenez fgimenez will be requested when the pull request is marked ready for review fgimenez is a code owner

Assignees

No one assigned

Labels

A-trie Related to Merkle Patricia Trie implementation C-perf A change motivated by improving speed, memory usage or disk footprint

Projects

Reth Tracker
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

perf: Batch proof tasks at worker level

10 participants

@yongkangc @shekhirin @klkvr @mattsse @forkfury @joshieDo @lean-apple @futreall @0xKarl98