metaslab: Repair overlapping space map entries automatically

[jameshilliard]

Motivation and Context

Duplicate or partially overlapping operations in an on-disk metaslab
space map currently reach strict range-tree assertions during
metaslab_load(). The resulting panic or blocked loader can make an
otherwise importable pool unusable.

A production failure was observed after a zfs rewrite -rvx operation
under OpenZFS 2.4.1 and matches the failure class discussed in #13995 and
#17094. Unlike a general range-tree recovery switch, this change is
limited to replaying on-disk metaslab space maps.

The repair is intentionally conservative. An ambiguous range is never
made allocatable, which can retain space that was actually free. This
cannot recover data that was overwritten before the corruption was
detected.

Description

• Replay metaslab space maps with checked single-pass range-tree operations.
  Clean entries retain the same one-lookup path as strict replay, and the
  private quarantine tree is allocated only after damage is detected.
• When an ALLOC or FREE operation conflicts with reconstructed state,
  remove its complete range from the free tree and keep that range
  allocated for all later overlapping operations.
• Track affected entries and bytes, and log the first and last conflicting
  entries with pool, vdev, metaslab, object, TXG, and sync-pass context.
  Export aggregate repair counters through the metaslab kstat and record
  completed repairs in pool history, including the old and reconstructed
  allocation summaries.
• Update metaslab, vdev, and class allocation accounting for quarantined
  space.
• Account a metaslab with an invalid smp_alloc summary as fully allocated,
  then schedule it for repair during import and reconstruct the summary
  from the replayed map.
• Force normal metaslab condensation in the following TXG so the
  conservative state replaces the corrupt history with a canonical
  space map.
• Defer repair until pending TXG, deferred-free, or log-space-map state
  has drained. Explicitly dirty and flush the metaslab until it is
  retryable, then dispatch a repair load independently of ordinary
  metaslab preloading.
• If a space map cannot be reconstructed, quarantine only that metaslab,
  assign it zero allocator weight, disable trimming for it, and account
  all remaining capacity as unavailable after pending changes drain.
• Keep transient load errors retryable. Make preload, initialize, trim,
  log-space-map debug loading, and RAID-Z expansion handle load failures
  without asserting or reporting false completion. RAID-Z expansion
  conservatively copies a complete metaslab when its free map cannot be
  loaded.
• Add zhack metaslab corrupt for test-only injection of duplicate and
  partial ALLOC/FREE entries, chained overlaps, and invalid summaries.
  The ZTS regression covers pools with and without log space maps,
  preserves pending log-space-map state, waits for automatic condensation
  and pool-history reporting with a bounded retry, performs strict zdb -b
  replay, checks unchanged file data for every corruption class, and
  verifies that an unreadable map is quarantined.
• Add unit coverage for checked range-tree insertion and removal,
  including adjacency merging, splitting, rejection of duplicate and
  partial operations, and unchanged state after rejected operations.

Normal space_map_load() callers retain strict range-tree behavior.

How Has This Been Tested?

The following local validation passed:

• Full debug OpenZFS userspace and kernel-module build
• make unit: 90/90 tests passed
• make testscheck
• make checkstyle, including C style, ShellCheck, checkbashisms, and mandoc
• scripts/cstyle.pl on every modified C/header file
• scripts/commitcheck.sh HEAD^..HEAD
• git diff --check
• Direct ShellCheck validation of the new ZTS script

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist

• My code follows the OpenZFS code style requirements.
• No user-facing documentation changes are required; the injection
  command is a test-only zhack interface.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the complete ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain
  Signed-off-by.

[Copilot]

Pull request overview

This PR adds a conservative, automatic repair path for metaslab free-space map replay so pools don’t panic during metaslab_load() when on-disk space map entries are duplicate or partially overlapping. The approach quarantines ambiguous ranges (never making them allocatable), updates accounting accordingly, and persists the repaired canonical state via condensation, with new diagnostics and test-only injection plus regression/unit coverage.

Changes:

• Add checked range-tree operations (try_add/try_remove) and a new space_map_load_length_repair() path to conservatively replay/repair metaslab free-space maps and track repair details.
• Teach metaslab loading/accounting to handle deferred/unloadable metaslabs and to persist conservative repaired state via condensation + history/kstat reporting.
• Add zhack metaslab corrupt injection and new ZTS + unit tests to cover repair/quarantine behavior and range-tree checked operations.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/zfs-tests/tests/Makefile.am	Registers the new metaslab repair ZTS test script.
tests/zfs-tests/tests/functional/cli_root/zhack/zhack_metaslab_repair.ksh	New functional test validating repair/quarantine behavior and data preservation.
tests/unit/test_range_tree.c	New unit tests covering checked range-tree insertion/removal semantics.
tests/unit/Makefile.am	Builds and links the new test_range_tree unit test binary.
tests/unit/.gitignore	Ignores the new unit test binary.
tests/runfiles/common.run	Adds zhack_metaslab_repair to the default ZTS runfile set.
module/zfs/vdev_trim.c	Makes TRIM paths tolerate metaslab load failures (skip/abort/suspend instead of asserting).
module/zfs/vdev_raidz.c	Makes RAID-Z expansion copy conservatively when metaslab free map can’t be loaded.
module/zfs/vdev_initialize.c	Makes initialize paths tolerate metaslab load failures (abort/suspend instead of asserting).
module/zfs/space_map.c	Implements conservative repair-capable free-space map replay and repair result reporting.
module/zfs/spa_log_spacemap.c	Avoids strict debug-load/assertion paths and adjusts accounting behavior for special metaslab load states.
module/zfs/range_tree.c	Refactors add/remove internals to support non-panicking “try” operations for overlap/missing-range detection.
module/zfs/metaslab.c	Core metaslab repair/quarantine/defer logic, accounting reconciliation, new kstats, and history logging hooks.
include/sys/space_map.h	Adds space_map_load_result_t and declares space_map_load_length_repair().
include/sys/range_tree.h	Declares zfs_range_tree_try_add() / zfs_range_tree_try_remove() and documents semantics.
include/sys/metaslab_impl.h	Adds metaslab load-state tracking and retained repair metadata fields.
cmd/zhack.c	Adds zhack metaslab corrupt for test-only injection of overlap/summary corruption patterns.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tests/zfs-tests/tests/functional/cli_root/zhack/zhack_metaslab_repair.ksh:173

• mktemp -d without a template is not portable (commonly fails on BSD/macOS where a template or -t is required). This can cause the test to fail before running any assertions. Use a portable mktemp invocation with an explicit template (and consider wrapping it in log_must).

tmpdir=$(mktemp -d)

tests/unit/test_range_tree.c:144

• This declares ret after a statement, which can break builds when compiled with stricter C dialects or with -Werror=declaration-after-statement (commonly enabled in this project/toolchains). Move int ret; to the start of the block to avoid a build failure.

int
main(int argc, char **argv)
{
	zfs_btree_init();
	int ret = munit_suite_main(&range_tree_test_suite, NULL, argc, argv);
	zfs_btree_fini();
	return (ret);
}

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tests/zfs-tests/tests/functional/cli_root/zhack/zhack_metaslab_repair.ksh:136

• Use the existing ZTS convention for splitting $DISKS into an array (set -A ... $DISKS) to match other tests and avoid relying on read -A ... <<<.

	read -r -A vdevs <<< "$DISKS"

tests/zfs-tests/tests/functional/cli_root/zhack/zhack_metaslab_repair.ksh:59

• Use the existing ZTS convention for splitting $DISKS into an array. read -A ... <<< is less common across the test suite; most tests use set -A (e.g. tests/zfs-tests/tests/functional/cli_root/zdb/zdb_003_pos.ksh:53).

This issue also appears on line 136 of the same file.

	read -r -A vdevs <<< "$DISKS"