zhack: add "mmp reclaim" to recover a pool stranded by MMP - #18892
zhack: add "mmp reclaim" to recover a pool stranded by MMP#18892mkhllr wants to merge 3 commits into
Conversation
| if (mmp_claim_relaxed && | ||
| lvd->vdev_not_present) | ||
| continue; | ||
| #endif |
There was a problem hiding this comment.
While you're here, can you also fix up the "for mirror 2 writes" portion of this comment at the end of mmp_claim_uberblock().
/*
* To guarantee visibility from a remote host we require a minimum
* number of good writes. For raidz/draid vdevs parity+1 writes, for
* mirrors 2 writes, and for singletons 1 write.
*/
| .Bd -literal | ||
| .No # Nm zpool Cm import Fl f Ar tank | ||
| cannot import 'tank': pool is imported on host '<unknown>' (hostid=0). | ||
| Export the pool on the other system, then run 'zpool import'. |
There was a problem hiding this comment.
Let's leave some more breadcrumbs to make it easier for anyone who encounters this corner case to find the documentation. Plumbing the error all the way back to do_import() in zpool_main.c would be awkward. But we could update spa_activity_check_claim() to return EIO when mmp_claim_uberblock() fails due to insufficient good writes. Then we can log a more useful error message to the console in spa_ld_activity_result() before it returns the expected EREMOTEIO, similar to the existing ENXIO case.
When a host fails together with the mirror legs attached to it, the surviving labels still describe those legs as present, so the MMP uberblock claim keeps demanding a write to every one of them and no later import can satisfy it. The pool cannot be imported by any host again. Add "zhack mmp reclaim", which imports once with the claim's required write count relaxed for the mirror legs this host cannot open, marks those leaves offline so that the ordinary imports which follow succeed, and exports. The relaxation is confined to userspace. mmp_claim_relaxed is declared under #ifndef _KERNEL, and module/Kbuild.in builds the module with -D_KERNEL, so the flag cannot exist in a kernel module. libzpool does not define _KERNEL and so gets the check. This follows the zfeature_checks_disable pattern zhack already uses around the same import, and is stronger, since that flag does exist in the kernel. Only the number of required writes changes. The write, the wait and the re-read of the activity check are untouched, so a competing host which shares any leg with this one is still detected and the import is refused. A live host whose legs are all invisible from here cannot be detected by any write-and-read scheme, so this stays a manual operation which assumes the peer has been fenced. Legs are forgiven only under a top-level mirror, which is where the relaxation lives, and exactly those legs are marked offline. A raidz or draid member is required as parity+1 in aggregate and never demanded individually, so an absent one does not raise the requirement and is left alone. Offline is used rather than removed because it persists unconditionally, is already excluded from the claim, and has "zpool online" as its inverse when the hardware returns. Suggested-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Michael Heller <michael.heller@gmail.com>
Six scenarios: a stranded pool is recovered and the claim then accepts it, a live host sharing a leg is still refused, both top-level vdevs are counted after a recovery, a pool without multihost is left alone, a log vdev leg is not touched, and a raidz member is not touched. Every recovery assertion re-imports as a third hostid. zhack exports cleanly under its own hostid, so importing again as the same host takes the exported-and-matching-hostid path, skips the activity check entirely, and would leave the claim unexercised and the test vacuous. The assertions read req_writes and good_writes from the claim's own dbgmsg line, which 2017622 added. That is the only observable of the claim arithmetic, at the cost of coupling the test to a debug message this change does not control. mmp_pool_destroy() used a bare "pgrep zhack", which matches any process whose name merely contains zhack. A ksh script named mmp_zhack_reclaim.ksh has comm "mmp_zhack_recla", so the helper found the running test and killed it. Match the process name exactly. Signed-off-by: Michael Heller <michael.heller@gmail.com>
e47a52d to
4fe8424
Compare
When the claim could not write to every device the config expects present, spa_activity_check_claim() replaced the EIO from mmp_claim_uberblock() with EREMOTEIO, so the console said "activity detected, aborted import" when nothing had been detected. An operator whose peer died together with its mirror legs was told another host holds the pool, which sends them looking for a host that is not there. Keep the EIO distinct and give it its own case in spa_ld_activity_result(), which reports what actually happened and points at the recovery, then returns EREMOTEIO exactly as the ENXIO case already does. What userspace sees does not change: the load info still carries MMP_STATE_ACTIVE and EREMOTEIO, so only the console message is new. The paths where the claim genuinely detects another host still return EREMOTEIO and are unaffected. Also correct the comment above the write count, which still described the fixed two writes per mirror that 8cdd9b2 replaced with one write per leg the config expects present. The ZTS case asserts both directions, since a message that stops being emitted fails silently: the shortfall must be reported, and it must not be reported as activity detected. Signed-off-by: Michael Heller <michael.heller@gmail.com>
|
Both done, thanks. The comment was mine to fix. 8cdd9b2 replaced the fixed two writes per The breadcrumbs are implemented as you laid out: keep the EIO from so an operator whose peer died together with its legs was told another host One judgement call I would rather you made than me. I read "before it which is arguably the more misleading of the two, and it is the line Two things I would rather flag than have you find. EIO here means the claim The ZTS case asserts both directions, since a console message that quietly Kept as its own commit so it can be reviewed or dropped without touching the |
2 participants
Follow-up to #18855, which @behlendorf asked for when he merged it.
The problem
#18855 made the MMP uberblock claim require a good write to every mirror leg
the pool config still expects to be present. That is what stops a host
claiming a pool under partial visibility, but it leaves one case with no way
out. When a host fails together with the mirror legs attached to it, the
surviving labels still describe those legs as healthy, so every later import
demands writes to legs that nobody can make. The claim can never be satisfied
and the pool cannot be imported again by any host.
@arturpzol hit this on a real two-node HA cluster while testing #18855. Each
mirror there is a local disk plus an iSCSI disk from the peer, so a dead node
takes its legs with it and the config still lists them present. His logs show
req_writes=6 good_writes=3, refused. That is the ordinary failover in thattopology, and the fix alone declines it.
What this adds
zhack mmp reclaim <pool>imports the pool once with the claim's requiredwrite count relaxed for the mirror legs this host cannot open, marks those
leaves offline so that the ordinary imports which follow succeed, and exports.
The pool comes back DEGRADED with those legs offline, and
zpool onlinereturns each one when the hardware does.
Design notes
Offline instead of removed. Offline persists unconditionally, is already
excluded from the claim by #18855, has
zpool onlineas a first classinverse, and does not overload hotplug semantics.
vdev_not_present, which iswhat a failed open actually sets, is not usable here: it is recomputed on
every import, so the pool would need zhack on every failover rather than once.
No existing flag covers this. On the stranded case (2-way mirror, one leg
gone with the peer)
-f,-f -o multihost=off,-f -o multihost=off -mand-fFX -o multihost=offall refuse withreq_writes=2 good_writes=1. Settingmultihost=offat import does not help, becausespa_activity_check_required()decides from the on-disk uberblock, whichstill has MMP active, before the property is applied.
Why not
ZFS_IMPORT_SKIP_MMP. That disables the whole activity check, aszdb uses it. Here the activity check has to be kept and only the write count
relaxed, so that a competing importer is still caught.
The relaxation cannot exist in the kernel.
mmp_claim_relaxedis declaredunder
#ifndef _KERNEL, andmodule/Kbuild.inbuilds the module with-D_KERNEL, so the symbol is compiled out of every kernel build. libzpooldoes not define
_KERNEL, so zhack gets it. This follows thezfeature_checks_disablepattern zhack already uses around the same import,and is stronger, since that flag does exist in the kernel.
Only mirror legs are forgiven, and exactly those are offlined. A raidz or
draid vdev is required as parity+1 in aggregate instead of one write per
member, so an absent member does not raise the requirement and is left alone.
The claim holds there while parity+1 members stay writeable; a narrower raidz
that has lost more than that is stranded and out of scope for this tool.
What this does not do
The relaxed claim still catches a competing importer that shares any
visibility with us, which is the common operator error. It cannot catch a live
peer whose legs are all invisible from here, because the claim write and the
re-read only ever touch reachable legs. That is inherent to any write and read
scheme under disjoint visibility, and no change to this code can close it.
The fencing is therefore load-bearing. This is a manual recovery that assumes
the peer has been confirmed down, and the man page says so.
Testing
New ZTS test
mmp/mmp_zhack_reclaim, six scenarios:multihostis left aloneEvery recovery assertion re-imports as a third hostid on purpose. zhack
exports cleanly under its own hostid, so importing again as the same host
takes the exported-and-matching-hostid path in
spa_activity_check_required()and skips the activity check entirely, which would leave the claim
unexercised. The assertions read
req_writesandgood_writesfrom theclaim's own dbgmsg line, which 2017622 added. That is the only observable of
the arithmetic, at the cost of coupling the test to a debug message this
change does not control.
The test commit also fixes
mmp_pool_destroy(), whose barepgrep zhackmatched the new test's own comm (
mmp_zhack_recla) and killed it.Verified on a VM harness with the loaded module's srcversion asserted equal to
the tree's, so the results cannot come from a stale
zfs.ko. Scenario 6 waschecked against a build with the mirror-only restriction removed, where it
fails as intended. The full mmp group passes, including
mmp_active_importand
mmp_concurrent_import, which share the helper that changed.One path is not exercised. When an absent leaf holds the only copy of some
data,
vdev_offline()returns EBUSY, and the tool reports the leaf, leaves itonline, warns that a later import from another host will still be refused, and
exits non-zero. I could not stage that state reliably from userspace, so I am
flagging it as untested. The messages and the exit status are static, and the
man page documents the outcome.
Documentation
@arturpzol asked in #18855 for the recovery flow to be written down.
man/man1/zhack.1documents the subcommand, the residual risk, the fencingrequirement, and the degraded aftermath.
Worth calling out separately for release notes: a pool that imported on 2.3.5
after this class of failure will refuse on 2.3.8, and the error text
(
cannot import 'tank': pool is imported on host '<unknown>' (hostid=0).)does not point anywhere useful when the truth is that the claim wanted more
writes than it got.