Repository navigation
Conversation
Fraccaman
force-pushed
the
fraccaman/rocksdb-max-open-files-headroom
branch
2 times, most recently
from
October 27, 2026 10:15
2eb37f7 to
cfb9a3c
Compare
RocksDB's max_open_files was set to the process NOFILE soft limit (or left unlimited when the limit doesn't fit an i32). A database with many SST files could then use up every file descriptor, and the node panicked with "Too many open files" while committing a block. Reserve a quarter of the limit (at least 1024 fds) for the rest of the node and cap "unlimited" to the kernel's default fs.nr_open.
Fraccaman
force-pushed
the
fraccaman/rocksdb-max-open-files-headroom
branch
from
October 27, 2026 10:19
cfb9a3c to
04c8d02
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Part of the fix for #5005 (
Too many open filespanics).set_max_open_filesset RocksDB'smax_open_filesto the process NOFILE soft limit. When the limit doesn't fit ani32(e.g.LimitNOFILE=infinity), it was left at RocksDB's default of-1, which means unlimited. Either way, RocksDB is allowed to keep every available file descriptor open for table files. On a DB with many SST files, nothing is left for P2P/RPC sockets, the WAL or newly created SST files, and the node panics while committing a block:This is what we saw on a mainnet RPC node with
LimitNOFILE=32000→max_open_files=32000and ~1.29M SST files. It also explains why raising the limit (even to unlimited) didn't help in #5005.This PR keeps a quarter of the soft limit (at least 1024 fds) out of RocksDB's reach, and treats "unlimited" as the kernel's default
fs.nr_open(2^20). For example, 16384 → 12288 and 32000 → 24000. Once RocksDB reachesmax_open_files, it evicts table readers from its cache instead of failing. The chosen value is now logged at startup.Not consensus breaking: it only changes a local RocksDB option.
The root cause of the large number of SST files is fixed separately (diffs compaction PRs).
Checklist before merging
breaking::labelsnamada-docsreponamada-indexerornamada-masp-indexer, a corresponding PR is opened in that repo