Improve performance by listing snapshots a batch at a time by whoschek · Pull Request #18850 · openzfs/zfs

whoschek · 2026-07-27T00:49:22Z

PR: This is the PR that closes #18849

Motivation

Listing large numbers of snapshots remains expensive even after the
relevant metadata is resident in the ARC. In the warm-cache benchmark
used for this change, ARC misses and physical storage I/O did not
increase during listing. Profiling attributed about 93% of the
command's runtime to ZFS_IOC_SNAPSHOT_LIST_NEXT, including about 57% to
dmu_objset_hold(). The dominant cost was repeated enumeration and
object setup, not storage latency.

ZFS_IOC_SNAPSHOT_LIST_NEXT ioctl returns at most one matching snapshot.
Listing N snapshots therefore requires N successful enumeration ioctls,
followed by a final ioctl that reports the end of iteration. Each ioctl
reacquires the parent objset and holds the snapshot's dsl_dataset_t.
When simple mode cannot supply the requested columns and sort keys, the
legacy path also instantiates the snapshot objset, gathers its
statistics and properties, and transfers an nvlist. Simple mode avoids
that full-stat work for its smaller property set, but it retains the
per-snapshot ioctl, parent-objset hold, and snapshot-dataset hold
costs. These repeated costs are especially visible for unmounted
filesystems and zvols.

Description

The patches add a projected, batch-based snapshot
iterator with a cursor, and use it for zfs list when every displayed
property and explicit sort key is one of:

createtxg
guid
name
creation
userrefs
type
numclones
inconsistent
redacted
origin
used
available
referenced aka refer
mountpoint
logicalreferenced aka lrefer
defer_destroy
objsetid
written

Add ZFS_IOC_SNAPSHOT_LIST_BATCH ioctl. Each call holds the parent objset
once and walks snapshots until it reaches either the configured result
size limit or the configured wall-clock budget. The resulting batch
contains a continuation cursor, snapshot names, and parallel uint64
arrays for the requested numeric properties. The default result batch
size limit is 1024 snapshots and the default time budget is 10ms, both
configurable as described below.

For each snapshot, the kernel reads creation TXG, creation time, and
GUID directly from the dataset bonus buffer without constructing a
dsl_dataset_t or snapshot objset. It counts user holds only when
userrefs is requested.

libzfs uses the returned batch values to construct projected snapshot
handles and continues from the returned cursor to the next batch until
enumeration is complete.

Bookmark enumeration continues to use lzc_get_bookmarks(), but requests
only the needed subset of guid, createtxg, and creation instead of
requesting every bookmark property.

If the batch ioctl or its projected arguments are unavailable before any
snapshot callback has run, libzfs falls back to the legacy iterator.
This allows new userland to work with older kernels.

Also add two corresponding tuning knobs, zfs_snapshot_list_batch_size and
zfs_snapshot_list_batch_time_us:

zfs_snapshot_list_batch_size (default is 1024): Maximum number of
snapshots returned by one projected snapshot listing batch. This
limit is a proxy for the maximum transient memory consumed by a
batch: larger values can improve listing throughput at the cost of a
larger memory footprint. Projected snapshot listing is cursor based:
a batch stops after either this many snapshots have been collected or
the zfs_snapshot_list_batch_time_us budget has been reached,
whichever occurs first. It returns the matching entries collected so
far and a cursor from which a subsequent ioctl resumes, leaving the
remaining snapshots for later batches. The default is 1024 snapshots,
and the supported range is 1 to 4096.
zfs_snapshot_list_batch_time_us (default is 10,000 microseconds): Soft
wall-clock budget for one projected snapshot listing batch. This
budget is a proxy for the maximum lock latency a batch can impose on
unrelated dataset operations. The elapsed time is checked after each
snapshot is examined by the scan. At least one snapshot is therefore
examined to ensure cursor progress, and a single metadata operation
may cause the budget to be exceeded. The default is 10,000
microseconds, and the supported range is 1 to 100,000 microseconds. A
lower value is better for latency-sensitive systems and a higher
value is better for throughput-oriented backup systems.

How Has This Been Tested?

zfs-tests.sh -v libzfs,zfs_property,zfs_bookmark,zfs_snapshot,snapshot,zfs_get, zfs_list
passed on AlmaLinux-10/aarch64.
Full OpenZFS Github CI matrix passed on all platforms.
bzfs test suite passed. It exercises the existing ZFS_ITER_SIMPLE
as well as the new ZFS_IOC_SNAPSHOT_LIST_BATCH ioctl in zfs send/recv
replication scenarios.
Added a series of additional functional tests to ZTS test suite,
described in detail in each individual commit msg.

Measurements

Some of the commits include additional measurements and details for variations of the theme, but the main gist is attached in the form of plots and table data in the ticket here: #18849, in particular this plot: #18849 (comment)

Types of Changes

Bug fix (non-breaking change which fixes an issue)
[] New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Quality assurance (non-breaking change which makes the code more robust against bugs)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair and libzfsbootenv)
Note that this PR adds a new ioctl.
Documentation (a change to man pages or other documentation)

Checklist

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

whoschek · 2026-07-30T17:30:06Z

@behlendorf @allanjude: I’d appreciate a review when you have capacity. This builds on the earlier snapshot-listing work and uses batched enumeration to remove much of the per-snapshot ioctl and repeated dataset-hold overhead.

The measurements and plots linked from #18849 (comment) show a substantial improvement for large snapshot listings. The PR includes a fallback to the legacy iterator and expanded functional coverage to preserve compatibility.

@allanjude, this extends the direction of your earlier improvements. Happy to share the exact benchmark script/setup if that would help reproduce or evaluate the results.

@amotin, would you be willing to take a look at the kernel-side iterator/ioctl and its locking/latency trade-offs in particular?

Upcoming snapshot-listing optimizations introduce alternate iterator paths whose results must remain identical to the legacy full-stat path. Add zfs_list_013_pos to establish broad regression coverage for those semantics. Compare normal listings with listings forced through the full-stat path for filesystems, clones, and volumes. Cover snapshots and bookmarks, human-readable and JSON output, each supported sort key, and all 63 nonempty subsets of createtxg, creation, guid, name, type, and userrefs. Also verify empty results, exact enumeration around 1024 entries, maximum-length names, default and explicit ordering, destroyed bookmark sources, user holds, and listing while snapshots are concurrently created, renamed, and destroyed. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

Motivation ========== Listing large numbers of snapshots remains expensive even after the relevant metadata is resident in the ARC. In the warm-cache benchmark used for this change, ARC misses and physical storage I/O did not increase during listing. Profiling attributed about 93% of the command's runtime to ZFS_IOC_SNAPSHOT_LIST_NEXT, including about 57% to dmu_objset_hold(). The dominant cost was repeated enumeration and object setup, not storage latency. ZFS_IOC_SNAPSHOT_LIST_NEXT ioctl returns at most one matching snapshot. Listing N snapshots therefore requires N successful enumeration ioctls, followed by a final ioctl that reports the end of iteration. Each ioctl reacquires the parent objset and holds the snapshot's dsl_dataset_t. When simple mode cannot supply the requested columns and sort keys, the legacy path also instantiates the snapshot objset, gathers its statistics and properties, and transfers an nvlist. Simple mode avoids that full-stat work for its smaller property set, but it retains the per-snapshot ioctl, parent-objset hold, and snapshot-dataset hold costs. These repeated costs are especially visible for unmounted filesystems and zvols. Description =========== Add a projected, batch-based snapshot iterator with a cursor, and use it for zfs list when every displayed property and explicit sort key is one of: createtxg, creation, guid, name, type, userrefs This set could be (and probably should be) extended to cover more properties in follow-on commits. The iterator also requests createtxg internally when it is needed for default ordering, sort tie-breaking, or JSON metadata. Listings that display or sort by any other property continue to use the legacy iterator. Add ZFS_IOC_SNAPSHOT_LIST_BATCH ioctl. Each call holds the parent objset once and walks snapshots until it reaches either the configured result size limit or the configured wall-clock budget. The resulting batch contains a continuation cursor, snapshot names, and parallel uint64 arrays for the requested numeric properties. The default result batch size limit is 1024 snapshots and the default time budget is 10ms, both configurable as described below. For each snapshot, the kernel reads creation TXG, creation time, and GUID directly from the dataset bonus buffer without constructing a dsl_dataset_t or snapshot objset. It counts user holds only when userrefs is requested. libzfs uses the returned batch values to construct projected snapshot handles and continues from the returned cursor to the next batch until enumeration is complete. Bookmark enumeration continues to use lzc_get_bookmarks(), but requests only the needed subset of guid, createtxg, and creation instead of requesting every bookmark property. If the batch ioctl or its projected arguments are unavailable before any snapshot callback has run, libzfs falls back to the legacy iterator. This allows new userland to work with older kernels. Add two tuning knobs, zfs_snapshot_list_batch_size and zfs_snapshot_list_batch_time_us, described below: - zfs_snapshot_list_batch_size (default is 1024): Maximum number of snapshots returned by one projected snapshot listing batch. This limit is a proxy for the maximum transient memory consumed by a batch: larger values can improve listing throughput at the cost of a larger memory footprint. Projected snapshot listing is cursor based: a batch stops after either this many snapshots have been collected or the zfs_snapshot_list_batch_time_us budget has been reached, whichever occurs first. It returns the matching entries collected so far and a cursor from which a subsequent ioctl resumes, leaving the remaining snapshots for later batches. The default is 1024 snapshots, and the supported range is 1 to 4096. - zfs_snapshot_list_batch_time_us (default is 10,000 microseconds): Soft wall-clock budget for one projected snapshot listing batch. This budget is a proxy for the maximum lock latency a batch can impose on unrelated dataset operations. The elapsed time is checked after each snapshot is examined by the scan. At least one snapshot is therefore examined to ensure cursor progress, and a single metadata operation may cause the budget to be exceeded. The default is 10,000 microseconds, and the supported range is 1 to 100,000 microseconds. A lower value is better for latency-sensitive systems and a higher value is better for throughput-oriented backup systems. To reduce latency and impact on other operations, the new kernel fast path allocates batch-sized result buffers before dmu_objset_hold () acquires dp_config_rwlock as a reader, not while holding the lock. It also respects the wall-clock budget described above. Recall that dp_config_rwlock writers already take precedence over readers, e.g. dsl_sync_task() via `zfs snapshot` and `zfs destroy` takes precedence over `zfs list -t snapshot`. We use monotonic gethrtime() wall-clock time for the budget, rather than CPU time, since sleeping on ARC I/O still blocks writers. After each examined snapshot, it checks gethrtime() - start_time >= time_budget. How Has This Been Tested? ========================= - `zfs-tests.sh -v libzfs,zfs_property,zfs_bookmark,zfs_snapshot,snapshot,zfs_get, zfs_list` passed on AlmaLinux-10/aarch64. - Full OpenZFS Github CI matrix passed on all platforms. - bzfs test suite passed. It exercises the existing ZFS_ITER_SIMPLE as well as the new ZFS_IOC_SNAPSHOT_LIST_BATCH in zfs send/recv replication scenarios. Where practical, the functional tests use legacy full-stat listing as the reference result: requesting the unsupported "available" property forces the legacy path, after which that extra column is discarded and the output is compared byte for byte. An LD_PRELOAD shim confirms that the batched ioctl was selected and injects errors that are otherwise difficult to reproduce. Automated coverage includes: - libzfs_input_check validates the required and optional ioctl arguments, every supported property projection, empty projections, unsupported properties, invalid property types, and the assigned ioctl number. - With this change, zfs_list_013_pos from the parent commit exercises the new path and compares it with full-stat listing for filesystems, clones, and volumes; snapshots and bookmarks; tabular and JSON output; every supported display and sort property; and all 63 nonempty subsets of the six projected properties. It also covers empty results, 1023/1024/1025-object boundaries, maximum-length names, destroyed bookmark sources, user holds, ordering, and concurrent snapshot creation, rename, and destruction. - zfs_list_010_pos validates both tunable ranges and forward progress with one-entry and one-microsecond limits. It compares projected and legacy output for destroyed bookmark sources, deferred-destroy snapshots, locked encrypted datasets, implicit listsnapshots output, recursive mixed filesystem/volume/snapshot/bookmark hierarchies, and nonrecursive mixed-type scoping. - zfs_list_011_pos verifies direct libzfs iteration across batch boundaries. Iterating 1025 snapshots returns every snapshot exactly once, and sorted iteration preserves creation-TXG order across two-entry batches. - zfs_list_012_pos exercises destination-buffer growth for both synthetic and kernel-reported ENOMEM, and legacy fallback for older or incompatible kernels. Fallback preserves creation-TXG filters, mixed snapshot/bookmark output, and bookmarks whose source snapshots no longer exist. Unsupported display and sort properties are also verified to bypass projected listing. - Fault injection further verifies continuation after an empty, non-EOF batch; ENOENT and ESRCH end-of-iteration handling; EINTR, bookmark, callback, and handle-allocation error propagation; and rejection of malformed parent metadata with EPROTO. A failure after the first callback is propagated without falling back or replaying callbacks. Stale parent handles are tested across destroy, rename, dataset-type replacement, and encryption-state replacement. Measurements ============ The benchmark compares the parent commit, labeled "nobatch", with this commit, labeled "batch". Each benchmark case contains eight datasets with 2,000 snapshots per dataset, for 16,000 snapshots total. One zfs list process handles each dataset. The processes run through xargs with a maximum concurrency of P, where P = 1, 2, 4, or 8. For example, the P = 4 case runs: printf 'bzfs-perf/zfs_list_snapshots_bench/fs-mounted/fs%05d\n' {0..7}|\ time xargs -r -n1 -P4 \ zfs list -H -p -t snapshot -d 1 \ -o createtxg,guid,name,creation,userrefs,type >/dev/null The benchmark covers three dataset classes: mounted filesystems, unmounted filesystems, and zvols. Each case was run once as a warmup, followed by three measured trials. Two output projections were measured: A = createtxg,guid,name B = createtxg,guid,name,creation,userrefs,type The parent commit can serve projection A through ZFS_ITER_SIMPLE. Projection B requires its full-stat path because it contains properties that simple mode does not provide. This commit serves both projections through the new batched path. Measurements were taken with the default values for the tuning knobs zfs_snapshot_list_batch_size (1024) and zfs_snapshot_list_batch_time_us (10,000 microseconds). Throughput is the 16,000 listed snapshots divided by the elapsed wall-clock time for the complete eight-command run. The table reports the median, minimum, maximum, and standard deviation across the three measured trials. Speedup is the batch median divided by the matching nobatch median for the same projection, dataset class, and concurrency. These are warm-cache measurements. They do not characterize cold-cache I/O, and the absolute throughput will depend on the host, pool, and workload. ``` zfs list -t snapshot throughput by dataset type (dataset-count=8, snapshots-per-dataset=2000, columns=A, B) speedup = batch median / nobatch median for matching columns/type/P. columns type speedup_range ------------------------------------------ ------------ ------------- createtxg,guid,name fs-mounted 6.7x - 7.2x createtxg,guid,name fs-unmounted 167x - 300x createtxg,guid,name zvol 160x - 310x createtxg,guid,name,creation,userrefs,type fs-mounted 17.6x - 23.7x createtxg,guid,name,creation,userrefs,type fs-unmounted 155x - 308x createtxg,guid,name,creation,userrefs,type zvol 164x - 293x A = createtxg,guid,name B = createtxg,guid,name,creation,userrefs,type Throughput values are snapshots/sec (median of 3 measured trials). columns type kind P median min max stddev speedup ------- ------------ ------- - ------- ------- ------- -------- -------- A fs-mounted batch 1 116269 111523 116791 2902.5 7.2x A fs-mounted nobatch 1 16177 16170 16193 12.1 1.0x A fs-mounted batch 2 183289 182574 186342 2001.2 7.1x A fs-mounted nobatch 2 25840 25368 26746 700.4 1.0x A fs-mounted batch 4 287801 277002 292365 7889.7 6.7x A fs-mounted nobatch 4 43053 42974 43503 285.5 1.0x A fs-mounted batch 8 399543 363389 401072 21328.4 6.9x A fs-mounted nobatch 8 57756 56539 60240 1886.3 1.0x A fs-unmounted batch 1 84767 84358 85412 531.7 166.9x A fs-unmounted nobatch 1 508 508 509 0.5 1.0x A fs-unmounted batch 2 128829 128238 129197 483.9 186.0x A fs-unmounted nobatch 2 692 691 695 2.2 1.0x A fs-unmounted batch 4 206310 191202 208825 9532.3 250.8x A fs-unmounted nobatch 4 823 821 823 1.2 1.0x A fs-unmounted batch 8 235112 229094 254870 13484.5 300.7x A fs-unmounted nobatch 8 782 774 785 5.6 1.0x A zvol batch 1 80861 79911 83823 2040.5 160.3x A zvol nobatch 1 505 499 506 3.5 1.0x A zvol batch 2 126076 125602 129912 2363.2 224.1x A zvol nobatch 2 563 561 567 3.3 1.0x A zvol batch 4 167032 159459 175771 8162.8 310.0x A zvol nobatch 4 539 531 539 4.7 1.0x A zvol batch 8 228337 160354 240753 43281.7 307.1x A zvol nobatch 8 743 739 758 10.2 1.0x B fs-mounted batch 1 97542 95698 98035 1231.9 17.6x B fs-mounted nobatch 1 5532 5386 5539 86.2 1.0x B fs-mounted batch 2 164220 156051 164955 4942.0 19.0x B fs-mounted nobatch 2 8638 8445 8653 116.1 1.0x B fs-mounted batch 4 259780 259331 261490 1139.1 22.4x B fs-mounted nobatch 4 11572 11557 11583 13.0 1.0x B fs-mounted batch 8 330682 318150 330785 7265.0 23.7x B fs-mounted nobatch 8 13945 13900 13966 33.8 1.0x B fs-unmounted batch 1 75601 73734 75895 1172.3 154.8x B fs-unmounted nobatch 1 488 485 489 2.4 1.0x B fs-unmounted batch 2 117110 116620 117584 482.0 185.8x B fs-unmounted nobatch 2 630 630 637 4.1 1.0x B fs-unmounted batch 4 185415 183291 188587 2665.1 244.5x B fs-unmounted nobatch 4 758 752 764 6.1 1.0x B fs-unmounted batch 8 220625 216321 236681 10730.5 308.5x B fs-unmounted nobatch 8 715 712 761 27.7 1.0x B zvol batch 1 72975 72301 74459 1104.1 164.0x B zvol nobatch 1 445 443 448 2.7 1.0x B zvol batch 2 117963 116335 120371 2030.5 192.5x B zvol nobatch 2 613 612 613 0.6 1.0x B zvol batch 4 147818 146725 150604 2000.2 286.1x B zvol nobatch 4 517 510 520 5.2 1.0x B zvol batch 8 208498 203724 222560 9792.0 292.7x B zvol nobatch 8 712 708 716 3.8 1.0x ``` Potential Improvements ====================== - Add batch support for numclones,inconsistent,redacted,origin. This would extend the set of covered ZFS properties such that it is the union of the current set and simple mode (see dc95911 and 0cee240 and openzfs#14110). - Consider adding the "written" ZFS property, too. - Consider adjusting the defaults for the tuning knobs. - Given that the snapshot listing throughput and latency of fs-mounted vs fs-unmounted vs zvol can be dramatically different, consider splitting the latency time budget tuning knob into three knobs, one for each dataset type: zfs_snapshot_list_batch_fsmounted_time_us, zfs_snapshot_list_batch_fsunmounted_time_us and zfs_snapshot_list_batch_zvol_time_us. - Consider starting the timer for the budget before dmu_objset_hold (). For now we start the timer immediately after dmu_objset_hold () returns, which improves throughput in the presence of I/O waits, especially at high concurrency; this is a throughput/latency trade-off. - Use memory more efficiently: Collect as many snapshots as can fit into the fixed size batch buffer (subject to the time budget). If snapshot names turn out to be a lot shorter than 256 bytes we could fit more snapshots into the same amount of space, or a similar number of snapshots into a smaller buffer. - The current patch is already large so might be good to separate related improvements from this commit. - Thanks for reading this far. The snapshots now come one batch at a time; this commit message came all at once. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The legacy snapshot iterator delivers each successfully collected snapshot before reporting an error encountered on a later snapshot. Preserve those semantics for batched iteration by having the kernel return the nvlist, arrays, etc for the snapshots collected before an error occured, along with the error. Have libzfs consume those snapshots before reporting the saved ioctl error. Require userspace to provide max_results and cap the kernel batch accordingly: effective_batch_size = min(zfs_snapshot_list_batch_size, max_results) Libzfs requests at most 1024 results and supplies a buffer sized to hold that many maximum-sized entries. This avoids an undersized output buffer after the iterator has advanced. Add coverage for missing and invalid max_results values and for an ioctl that returns both partial results and EIO. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

Dependent traversal could omit a snapshot's clones when a caller requested simple or batched iteration. This would return an incomplete dependency list and could cause callers to treat snapshots with clones as having no clone dependents. This appears to be a latent bug only; there are no known call sites that actually call zfs_iter_dependents_v2() with flag != 0. The bug existed already before (and is unrelated to) batch listing. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. get_numeric_property() zeroes a large zfs_cmd_t for every numeric property lookup, although only the ZFS_IOC_OBJSET_ZPLPROPS cases use it. Scope the command and result nvlist to those cases so cached properties avoid the unnecessary initialization. Throughput on the zfs list -t snapshot benchmark with fs-mounted, nprocs=1, columns=createtxg,guid,name,creation,userrefs,type: 116111 snapshots/s (1.10x parent commit) Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. Store projected creation and userrefs values directly in snapshot handles. Lazily materialize their traditional property nvlists only for callers of zfs_get_all_props(). Preserve duplication, refresh, error, and public getter behavior. Throughput on the zfs list -t snapshot benchmark with fs-mounted, nprocs=1, columns=createtxg,guid,name,creation,userrefs,type: 142242 snapshots/s (1.35x parent commit) (sort by creation) 145855 snapshots/s (1.25x parent commit) (no sort) Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. Add batch support for ZFS properties numclones,inconsistent,redacted,origin to ZFS_IOC_SNAPSHOT_LIST_BATCH ioctl. With this the new set of batch-enabled ZFS properties is: createtxg,guid,name,creation,userrefs,type,numclones,inconsistent, redacted,origin In other words, this extends the set of batch-enabled ZFS properties to the union of the current set and simple mode (see dc95911 and 0cee240 and openzfs#14110). Measurements ============ The benchmark compares the parent commit, labeled "nobatch", with this commit, labeled "batch". Each benchmark case contains eight datasets with 2,000 snapshots per dataset, for 16,000 snapshots total. One zfs list process handles each dataset. The processes run through xargs with a maximum concurrency of P, where P = 1, 2, 4, or 8. For example, the P = 4 case runs: printf 'bzfs-perf/zfs_list_snapshots_bench/fs-mounted/fs%05d\n' {0..7}|\ time xargs -r -n1 -P4 \ zfs list -H -p -t snapshot -d 1 \ -o createtxg,guid,name,creation,userrefs,type >/dev/null The benchmark covers three dataset classes: mounted filesystems, unmounted filesystems, and zvols. Each case was run once as a warmup, followed by three measured trials. Two output projections were measured: A = createtxg,guid,name,numclones,inconsistent,redacted,origin B = A + creation,userrefs,type The parent commit can serve projection A through ZFS_ITER_SIMPLE. Projection B requires its full-stat path because it contains properties that simple mode does not provide. This commit serves both projections through the new batched path. Measurements were taken with the default values for the tuning knobs zfs_snapshot_list_batch_size (1024) and zfs_snapshot_list_batch_time_us (10,000 microseconds). Throughput is the 16,000 listed snapshots divided by the elapsed wall-clock time for the complete eight-command run. The table reports the median, minimum, maximum, and standard deviation across the three measured trials. Speedup is the batch median divided by the matching nobatch median for the same projection, dataset class, and concurrency. These are warm-cache measurements. They do not characterize cold-cache I/O, and the absolute throughput will depend on the host, pool, and workload. ``` zfs list -t snapshot throughput by dataset type (dataset-count=8, snapshots-per-dataset=2000, columns=A, B) speedup = batch median / nobatch median for matching columns/type/P. columns type speedup_range -------------------------------------- ------------ ----------------- A fs-mounted 6.6x - 7.8x A fs-unmounted 150.1x - 493.7x A zvol 180.5x - 305.8x B fs-mounted 18.3x - 27.2x B fs-unmounted 160.9x - 469.1x B zvol 202.4x - 315.1x A = createtxg,guid,name,numclones,inconsistent,redacted,origin B = createtxg,guid,name,numclones,inconsistent,redacted,origin,creation, userrefs,type Throughput values are snapshots/sec (median of measured trials). columns type kind P median min max stddev speedup ------- ------------ ------- - ------- ------- ------- -------- -------- A fs-mounted batch 1 112083 109383 113101 1586.5 6.6x A fs-mounted nobatch 1 16959 16831 17021 96.7 1.0x A fs-mounted batch 2 203124 199159 208230 3512.6 7.6x A fs-mounted nobatch 2 26601 25592 26764 634.6 1.0x A fs-mounted batch 4 320891 313415 327453 6704.3 7.8x A fs-mounted nobatch 4 41211 40767 41776 505.8 1.0x A fs-mounted batch 8 429211 406910 458025 18739.9 6.7x A fs-mounted nobatch 8 63702 63341 64312 490.5 1.0x A fs-unmounted batch 1 86594 85399 88038 1115.8 150.1x A fs-unmounted nobatch 1 577 571 577 3.6 1.0x A fs-unmounted batch 2 143702 142166 144186 906.7 203.2x A fs-unmounted nobatch 2 707 704 715 5.5 1.0x A fs-unmounted batch 4 161888 156038 180211 9843.8 224.7x A fs-unmounted nobatch 4 720 717 722 2.6 1.0x A fs-unmounted batch 8 143943 133931 150567 6624.8 493.7x A fs-unmounted nobatch 8 292 288 292 2.0 1.0x A zvol batch 1 99148 96533 99648 1304.0 180.5x A zvol nobatch 1 549 541 550 5.0 1.0x A zvol batch 2 168117 166161 169977 1671.5 227.1x A zvol nobatch 2 740 737 741 2.1 1.0x A zvol batch 4 244090 230088 256700 9543.5 305.8x A zvol nobatch 4 798 785 812 13.6 1.0x A zvol batch 8 304872 298558 319975 9055.3 262.6x A zvol nobatch 8 1161 1156 1163 3.5 1.0x B fs-mounted batch 1 108465 107645 109574 705.1 18.3x B fs-mounted nobatch 1 5930 5927 5964 20.7 1.0x B fs-mounted batch 2 200408 198160 201173 1247.9 21.9x B fs-mounted nobatch 2 9169 8996 9319 161.5 1.0x B fs-mounted batch 4 315465 313459 317964 1985.1 27.2x B fs-mounted nobatch 4 11590 11582 11675 51.6 1.0x B fs-mounted batch 8 400190 377444 411715 15054.5 25.4x B fs-mounted nobatch 8 15728 15614 15785 87.3 1.0x B fs-unmounted batch 1 85088 83000 85847 1091.8 160.9x B fs-unmounted nobatch 1 529 527 530 1.1 1.0x B fs-unmounted batch 2 137488 133797 140287 2879.1 207.5x B fs-unmounted nobatch 2 663 645 678 16.6 1.0x B fs-unmounted batch 4 162928 149990 165336 6126.9 239.5x B fs-unmounted nobatch 4 680 675 689 6.8 1.0x B fs-unmounted batch 8 135788 113140 143946 11534.2 469.1x B fs-unmounted nobatch 8 289 288 290 0.6 1.0x B zvol batch 1 96312 91504 97814 2473.3 202.4x B zvol nobatch 1 476 472 477 2.4 1.0x B zvol batch 2 162365 150837 163718 5243.6 239.1x B zvol nobatch 2 679 677 680 1.5 1.0x B zvol batch 4 238931 231351 240046 3929.1 315.1x B zvol nobatch 4 758 745 763 9.5 1.0x B zvol batch 8 274810 194241 319813 50251.8 245.0x B zvol nobatch 8 1122 1119 1133 7.4 1.0x ``` Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. Add batch support for these ZFS properties to ZFS_IOC_SNAPSHOT_LIST_BATCH ioctl: used, available, referenced aka refer, mountpoint, logicalreferenced aka lrefer, defer_destroy, objsetid With this the new set of batch-enabled ZFS properties is: createtxg,guid,name,creation,userrefs,type,numclones,inconsistent, redacted,origin,used,available,referenced aka refer,mountpoint,logicalreferenced aka lrefer, defer_destroy, objsetid With this, the typical default `zfs list -t snapshot` command (where the user does not specifying any ZFS output properties) is now batch optimized too. While there, also include useful properties from the same cheap dsl_dataset_phys_t bonus-buffer read. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. Projected listing may omit a requested property from snapshot and bookmark handles only when neither object type can have a value for that property. In this case zfs list prints "-" for the property. Before this change, zfs_list_batch_flags() recognized this case only for the available, mountpoint, and origin properties. Any other native property not defined for snapshots or bookmarks, such as compression, quota, recordsize, or sharenfs, unnecessarily selected the (slow) legacy snapshot iterator. For example: zfs list -t snapshot -o name,compression pool/fs zfs list -t snapshot -o name -s quota pool/fs Both commands unnecessarily read (slow) full statistics for every snapshot. The first printed "-" for compression, and quota could not affect the second command's ordering because snapshots cannot have either property. After this change, both commands use projected batch iteration, without forwarding the inapplicable properties to the kernel. They produce the same output and ordering as before. Bookmark listings likewise request no value from the kernel for properties that bookmarks cannot have. Mixed listings also retain their existing behavior. Filesystem and volume rows show and sort by their actual property values. Snapshot and bookmark rows show "-", sort after rows that have a value, and use later sort keys or the default name order to break ties. When a property applies to neither ZFS_TYPE_SNAPSHOT nor ZFS_TYPE_BOOKMARK, continue building the batch flags without adding a projected field for that property. A property that applies to either type still selects legacy iteration unless the projected iterator explicitly supports it. User properties, user, group, and project quota properties, and written@ expressions, and -o all still select legacy iteration. If the kernel does not support the batch ioctl, libzfs still retries with the legacy iterator before delivering any results. Add functional tests that compare projected results byte for byte with forced legacy iteration for output columns and sort keys in snapshot, bookmark, and recursive mixed-type listings. Also verify empty bookmark projections, old-kernel fallback, and legacy selection for user properties, quota pseudo-properties, written@, -o all, and colored output. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

The only user-visible change is improved performance. Requesting `written`, as either an output column or sort key, currently disables projected snapshot iteration. zfs list must therefore create a full zfs_handle_t and collect all dataset statistics for every snapshot, even though it needs only one additional value. This is particularly expensive for unmounted filesystems and zvols. Extend ZFS_IOC_SNAPSHOT_LIST_BATCH and the projected iterator to return `written` without constructing dsl_dataset_t state for each snapshot. For a snapshot and its immediate predecessor, the existing calculation reduces to: current referenced bytes - predecessor referenced bytes + current snapshot deadlist used bytes Read those values directly from their MOS bonus buffers while the batch ioctl holds the pool configuration lock as a reader. The lock stabilizes predecessor linkage and snapshot deadlists against snapshot creation, destruction, deadlist merging, promotion, and other operations requiring the configuration writer lock. Validate the predecessor dataset bonus type and size, verify that its creation TXG matches ds_prev_snap_txg and precedes the current snapshot, and validate the deadlist object before reading it. Support both modern DMU_OT_DEADLIST objects and legacy DMU_OT_BPOBJ deadlists, including the minimal BPOBJ_SIZE_V0 header used by old pool versions. Snapshot enumeration uses ZAP order rather than chronological order, so the predecessor cannot safely be inferred from the previous iterator result. Reading the current deadlist and ds_prev_snap_obj costs up to two additional MOS bonus-buffer reads per included snapshot. This avoids unsafe ordering assumptions, cross-batch state, and much more expensive dsl_dataset_t construction. Validate the current deadlist before applying the creation-TXG filter, matching the legacy full-stat path's snapshot-construction error behavior. Apply the filter before reading ds_prev_snap_obj, so excluded snapshots avoid the predecessor read but still performs current-deadlist validation, again to match the legacy snapshot iterator behavior. Add parallel uint64 value and uint8 validity arrays to the batch protocol. A separate validity value is necessary because zero is valid and the full uint64 range provides no safe sentinel. If a snapshot has no predecessor or its value cannot be calculated, leave `written` absent from the projected handle. This preserves full-stat behavior instead of reporting a false zero or failing the entire batch. Libzfs requires both arrays to match the snapshot count and rejects invalid validity bytes with EPROTO. Add valid values to the normal property nvlist through zfs_batch_add_uint64_prop(). Update output-column and sort-key selection, ioctl input validation, malformed-response tests, and projected/full-stat parity coverage. Validation covers sorting in both directions, missing values, clone origins, modern and legacy deadlists, intermediate snapshot destruction, TXG filters, bookmarks, zvols, encryption, deferred destroy, batch boundaries, and concurrent snapshot mutation. Measurements ============ The warm-cache benchmark lists 16000 snapshots across eight datasets with columns name,written. One process handles each dataset. Maximum concurrency P is 1, 2, 4, or 8. Each case ran once as a warmup, followed by five measured trials. Throughput is 16000 snapshots divided by total elapsed wall-clock time. The table reports the median, minimum, maximum, and standard deviation across measured trials. Speedup is the batch median divided by the matching nobatch median for the same dataset type and concurrency. Throughput values are snapshots/sec. ``` type kind P median min max stddev speedup ------------ ------- - ------- ------- ------- -------- -------- fs-mounted batch 1 80996 76831 83127 2686.4 14.3x fs-mounted nobatch 1 5646 5556 5898 129.4 1.0x fs-mounted batch 2 137295 130187 141878 5310.3 15.3x fs-mounted nobatch 2 9002 8879 9102 96.9 1.0x fs-mounted batch 4 209338 124179 216418 38507.6 17.5x fs-mounted nobatch 4 11964 7145 12097 2158.0 1.0x fs-mounted batch 8 248571 234901 251283 6519.7 18.5x fs-mounted nobatch 8 13400 13329 13437 40.3 1.0x fs-unmounted batch 1 65289 64137 66745 993.9 122.7x fs-unmounted nobatch 1 532 523 535 5.4 1.0x fs-unmounted batch 2 99938 96684 100828 1710.8 130.1x fs-unmounted nobatch 2 768 747 781 12.8 1.0x fs-unmounted batch 4 156700 144135 162983 7227.8 178.9x fs-unmounted nobatch 4 876 866 894 11.0 1.0x fs-unmounted batch 8 160326 146566 183011 14132.1 247.4x fs-unmounted nobatch 8 648 647 655 4.5 1.0x zvol batch 1 64142 63673 64323 252.1 135.9x zvol nobatch 1 472 469 473 1.5 1.0x zvol batch 2 100945 99235 105864 2586.9 143.3x zvol nobatch 2 704 701 714 5.5 1.0x zvol batch 4 148143 143062 151424 3180.2 186.9x zvol nobatch 4 793 783 795 5.0 1.0x zvol batch 8 182457 166889 196401 12018.6 238.4x zvol nobatch 8 765 758 769 4.3 1.0x ``` The nobatch cases included quota solely to select the full-stat path. Batched `written` improved throughput by 14.3x to 247.4x across the tested dataset types and process counts. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

Why === The batched snapshot iterator is an optimization of the existing zfs list path. For snapshots whose `written` property is selected, it must preserve the pre-batch iterator's observable error behavior. The pre-batch path constructs the current snapshot before retrieving `written` such that, if the current snapshot's deadlist cannot be opened, snapshot construction fails and the listing reports the error. There, a later failure opening the predecessor occurs inside dsl_get_written() and merely leaves `written` unavailable. The same errno can therefore have different results depending on the stage at which it occurs. Said compatibility boundary is subtle and difficult to exercise reliably on a healthy pool. Without focused coverage, the two stages can accidentally be treated alike while normal `written` values still look correct. The TXG-filtered case preserves a separate compatibility aspect: snapshots excluded by the creation-TXG filter still validate the current deadlist, preserving the pre-batch snapshot-construction error behavior, but avoid the predecessor read because `written` will not be returned. This retains useful early filtering for the predecessor lookup without weakening current-snapshot error parity. What ==== Add ZTS coverage for these projected `written` lookup behaviors: * A normal lookup succeeds and returns a valid `written` value. * EIO reading the current deadlist is returned to the caller. * EIO reading the predecessor succeeds with `written` left invalid. * A lookup excluded by either TXG bound still returns an injected current-deadlist EIO, but otherwise leaves written invalid without touching an injected predecessor. For each fault-injection case, use a marker to prove whether the selected metadata read was reached. This makes both the compatibility contract and the early-filtered exception explicit. How === Create a real file-backed pool with two snapshots, export it, and import it read-only through libzpool. A small test program obtains the snapshot, deadlist, and predecessor object numbers from the pool and calls dsl_dataset_snapshot_stats() for each case. Extend the ZTS-only preload shim to intercept dmu_bonus_hold() only in an explicit test mode and for one exact MOS object. It returns EIO and writes the marker. The pool, object layout, and lookup path remain real; only the selected metadata hold is made to fail. Register the helper and scenario in the existing zfs_list test group. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

whoschek · 2026-08-05T03:25:58Z

Update:

Rebased on main
Removed these commits as they made user-visible changes beyond performance:
- zfs list: propagate recursive iterator errors (9fe5a4d)
- zfs list: allow snapshot,bookmark types without -d (e1e0ce4)
Added these commits:
- zfs list: on error return a batch containing results collected so far
- zfs list perf: improve perf of listing default snapshot properties
- zfs list perf: avoid full snapshot stats for non-snapshot native props
- zfs list perf: batch the written property for snapshots
Force pushed

The only user-visible change is improved performance.

Aggregate diff line stats between first and last commit in the entire PR, including the other commits (comment/blank lines excluded):

+---------------------+-------+---------+
| categoy             | added | removed |
+---------------------+-------+---------+
| kernel              |   542 |       0 |
| libzfs              |   582 |      10 |
| zfs CLI             |   142 |       1 |
| headers/boilerplate |    67 |       0 |
| tests               |  2987 |       1 |
| docs                |    27 |       0 |
| other               |     0 |       0 |
+---------------------+-------+---------+
| total               |  4347 |      12 |
+---------------------+-------+---------+

whoschek · 2026-08-05T03:26:55Z

Steps to reproduce / take measurements

Create a Linux VM running with native speed (hardware-assisted virtualization).
I use this script: https://github.com/whoschek/bzfs/blob/main/bzfs_testbed/lima_vm.sh like so

ZFS_REPO_HOST_DIR="$HOME/repos/zfs"       # aka git repo root dir on host machine
ZFS_REPO_MOUNT="${ZFS_REPO_MOUNT:-/zfs}"  # this is the dir where $ZFS_REPO_HOST_DIR is mounted within the guest VM
export LIMA_VM_MOUNTS="[{\"location\":\"$ZFS_REPO_HOST_DIR\",\"mountPoint\":\"$ZFS_REPO_MOUNT\",\"writable\":false}]"
LIMA_VM_NAME=zfsdev LIMA_VM_TEMPLATE=template:almalinux-10 LIMA_VM_DISK=50 LIMA_VM_MEMORY=8 LIMA_VM_CPUS=8 LIMA_ZFS_VERSION=none LIMA_NO_RUN_TESTS=true ./bzfs/bzfs_testbed/lima_vm.sh

Build zfs and run functional tests.
Run a benchmark sweep over the parameter space.
I use this benchmark tool: https://github.com/whoschek/bzfs/blob/main/bzfs_tests/zfs_list_snapshots_bench.py
in combination with the driver shell script attached here: run_zfs_list_snapshots_bench.sh

The zfs_list_016_pos test passes on AlmaLinux, FreeBSD, etc, but fails on Ubuntu. Ubuntu links libzpool with -Bsymbolic-functions, which binds its internal dmu_bonus_hold() calls locally and prevents the compatibility test preload shim from injecting the requested EIO. Link the production-built dsl_dataset libtool object directly into the helper and wrap dmu_bonus_hold() at the final link. Keep the exact MOS object target in thread-local state, and remove the obsolete preload and marker plumbing. This also works when libzpool is configured with --disable-static and leaves product code and pool data unchanged. Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek@mac.com>

whoschek · 2026-08-06T00:12:04Z

Where to start

The main change is a cursor-based snapshot iterator that returns projected metadata in size-limited, time-budgeted batches. This amortizes ioctl and parent-objset hold overhead across many snapshots while preserving the existing zfs list traversal, sorting, and output path.

A useful reading order is:

Userspace

cmd/zfs/zfs_main.c:zfs_do_list() and cmd/zfs/zfs_iter.c:zfs_list_batch_flags() decide whether the requested columns and sort keys can use projected listing. Unsupported cases retain the legacy path.
cmd/zfs/zfs_iter.c:zfs_callback() connects the existing traversal to snapshot and bookmark iteration.
lib/libzfs/libzfs_iter.c:zfs_iter_snapshots_v2() is the batch versus legacy dispatch point, including fallback for older kernels before any callback has run.
lib/libzfs/libzfs_iter.c:zfs_iter_snapshots_batch() and zfs_do_snapshot_list_batch_ioctl() implement the cursor loop, validate each response, construct projected snapshot handles, and preserve partial-result and error behavior.

Kernel

module/zfs/zfs_ioctl.c:zfs_ioc_snapshot_list_batch() holds the parent objset once and collects snapshots until it reaches the result limit or soft time budget.
module/zfs/dsl_dataset.c:dsl_dataset_snapshot_stats() reads snapshot metadata directly from dataset bonus buffers without constructing a dsl_dataset_t or snapshot objset.

For comparison, module/zfs/zfs_ioctl.c:zfs_ioc_snapshot_list_next() is the existing one-snapshot-per-ioctl path. Its related simple-mode optimization is the zc_simple branch, which calls module/zfs/dsl_dataset.c:dsl_dataset_fast_stat(); userspace constructs the corresponding reduced handle in lib/libzfs/libzfs_dataset.c:make_dataset_simple_handle_zc().

Bookmark listing remains in zfs_iter_bookmarks_v2() using lzc_get_bookmarks(). The specialized written calculation is in dsl_dataset_snapshot_written().

behlendorf added the Status: Code Review Needed Ready for review and testing label Aug 4, 2026

behlendorf self-requested a review August 4, 2026 21:44

whoschek added 11 commits August 4, 2026 21:53

robn self-assigned this Aug 5, 2026

robn self-requested a review August 5, 2026 03:10

robn removed their assignment Aug 5, 2026

whoschek force-pushed the wip/snap_list_batch29_stage1 branch from c0da08f to 420fe8b Compare August 5, 2026 03:24

Navigation Menu

Uh oh!

Improve performance by listing snapshots a batch at a time - #18850

Improve performance by listing snapshots a batch at a time#18850
whoschek wants to merge 12 commits into
openzfs:masterfrom
whoschek:wip/snap_list_batch29_stage1

whoschek commented Jul 27, 2026 •

edited

Loading

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

whoschek commented Aug 5, 2026

Uh oh!

whoschek commented Aug 5, 2026 •

edited

Loading

Uh oh!

whoschek commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

whoschek commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

How Has This Been Tested?

Measurements

Types of Changes

Checklist

Uh oh!

whoschek commented Jul 30, 2026

Uh oh!

whoschek commented Aug 5, 2026

Uh oh!

whoschek commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to reproduce / take measurements

Uh oh!

whoschek commented Aug 6, 2026

Where to start

Userspace

Kernel

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

whoschek commented Jul 27, 2026 •

edited

Loading

whoschek commented Aug 5, 2026 •

edited

Loading