Skip to content

Keep fd headroom when setting RocksDB max_open_files - #5063

Open
Fraccaman wants to merge 1 commit into
mainfrom
fraccaman/rocksdb-max-open-files-headroom
Open

Fraccaman wants to merge 1 commit into
mainfrom
fraccaman/rocksdb-max-open-files-headroom

Conversation

@Fraccaman

Copy link
Copy Markdown
Collaborator

Describe your changes

Part of the fix for #5005 (Too many open files panics).

set_max_open_files set RocksDB's max_open_files to the process NOFILE soft limit. When the limit doesn't fit an i32 (e.g. LimitNOFILE=infinity), it was left at RocksDB's default of -1, which means unlimited. Either way, RocksDB is allowed to keep every available file descriptor open for table files. On a DB with many SST files, nothing is left for P2P/RPC sockets, the WAL or newly created SST files, and the node panics while committing a block:

Encountered a storage error while committing a block: ... DBError("IO error: While open a file for random read: .../db/16286206.sst: Too many open files")
Location: crates/node/src/shell/mod.rs:829

This is what we saw on a mainnet RPC node with LimitNOFILE=32000 → max_open_files=32000 and ~1.29M SST files. It also explains why raising the limit (even to unlimited) didn't help in #5005.

This PR keeps a quarter of the soft limit (at least 1024 fds) out of RocksDB's reach, and treats "unlimited" as the kernel's default fs.nr_open (2^20). For example, 16384 → 12288 and 32000 → 24000. Once RocksDB reaches max_open_files, it evicts table readers from its cache instead of failing. The chosen value is now logged at startup.

Not consensus breaking: it only changes a local RocksDB option.

The root cause of the large number of SST files is fixed separately (diffs compaction PRs).

Checklist before merging

  • If this PR has some consensus breaking changes, I added the corresponding breaking:: labels
    • This will require 2 reviewers to approve the changes
  • If this PR requires changes to the docs or specs, a corresponding PR is opened in the namada-docs repo
    • Relevant PR if applies:
  • If this PR affects services such as namada-indexer or namada-masp-indexer, a corresponding PR is opened in that repo
    • Relevant PR if applies:

@Fraccaman
Fraccaman force-pushed the fraccaman/rocksdb-max-open-files-headroom branch 2 times, most recently from 2eb37f7 to cfb9a3c Compare October 27, 2026 10:15
RocksDB's max_open_files was set to the process NOFILE soft limit (or left
unlimited when the limit doesn't fit an i32). A database with many SST files
could then use up every file descriptor, and the node panicked with
"Too many open files" while committing a block.

Reserve a quarter of the limit (at least 1024 fds) for the rest of the node and
cap "unlimited" to the kernel's default fs.nr_open.
@Fraccaman
Fraccaman force-pushed the fraccaman/rocksdb-max-open-files-headroom branch from cfb9a3c to 04c8d02 Compare October 27, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant