Actually verify md5 on the formatted lane, and only forgive it on GCP

Checked what the formatted lane was doing before claiming it was a weaker
detector, and on k8s it was not a detector at all: _run_fio_dual's job config
carries ioengine, direct, bs, iodepth, numjobs, rw, size, nrfiles -- and no
verify of any kind. It moved IO and checked nothing. Only the docker path
verified, through the --verify=md5 hardcoded inside run_fio_test, so the two
platforms were never checking the same thing and the md5 severity work applied
to one of them.

_run_fio_dual now takes verify= and verify_fatal=, both off by default so every
existing caller is untouched. The lblk formatted lane passes verify="md5".

verify_fatal follows the hardware, which is the rule you set. Where a 4K atomic
write is guaranteed a mismatch is a real defect and fails the job outright;
on GCP's pd-balanced disks it can be an artefact of run_fio_test's own
verify_backlog, so it is recorded and triaged rather than failing the run. Same
decision as _md5_severity, derived from CLUSTER_ENV in both places.

Rendered all three configs to confirm: existing callers get no verify line,
atomic hardware gets verify=md5 plus verify_fatal=1, GCP gets verify=md5
without it.

So the two lanes now genuinely differ in what they can prove, which was the
point of running both: raw crc32c is unconditional and nothing can absorb a
torn write, while md5-through-a-filesystem is trusted only where the hardware
earns it.

Author: RaunakJalan

e2e/e2e_tests/cluster_test_base.py

@@ -1275,8 +1275,19 @@ def _connect_and_mount_dual(self, lvol_name, mount_path=None,
     def _run_fio_dual(self, lvol_name, mount_path=None, log_path=None,
                       runtime=300, name=None, rw="randrw", size="1G",
                       bs="4K", iodepth=1, numjobs=2, nrfiles=8,
-                      time_based=True, **kwargs):
-        """Start FIO. Returns thread (Docker) or job_name str (K8s)."""
+                      time_based=True, verify=None, verify_fatal=False,
+                      **kwargs):
+        """Start FIO. Returns thread (Docker) or job_name str (K8s).
+
+        verify: e.g. "md5" or "crc32c". Opt-in and off by default, so existing
+            callers are unaffected. It is worth knowing that WITHOUT it the k8s
+            job does no data verification at all -- it only moves IO -- while
+            the docker path has always had --verify=md5 hardcoded inside
+            run_fio_test. The two lanes were never checking the same thing.
+        verify_fatal: fail the FIO job itself on a mismatch. Leave False where
+            a mismatch should be reported but not fail the run, which is the
+            case on hardware with no 4K atomic-write guarantee.
+        """
         fio_name = name or f"fio_{lvol_name}"
 
         if self.k8s_test:
@@ -1288,13 +1299,19 @@ def _run_fio_dual(self, lvol_name, mount_path=None, log_path=None,
             cm_name = f"fiocfg-{job_name}"
 
             time_cfg = f"time_based\nruntime={runtime}" if time_based else ""
+            verify_cfg = ""
+            if verify:
+                verify_cfg = f"verify={verify}\nverify_state_save=0\n"
+                if verify_fatal:
+                    verify_cfg += "verify_fatal=1\n"
             fio_config = (
                 f"[global]\n"
                 f"ioengine=libaio\n"
                 f"direct=1\n"
                 f"bs={bs}\n"
                 f"iodepth={iodepth}\n"
                 f"numjobs={numjobs}\n"
+                f"{verify_cfg}"
                 f"{time_cfg}\n"
                 f"\n"
                 f"[{self._k8s_normalize_name(fio_name)[:20]}]\n"

e2e/e2e_tests/lblk/test_lblk.py

@@ -453,10 +453,22 @@ def _fs_fio(self, lvol_name, mount, tag, runtime=60):
         if not self.k8s_test:
             log = f"{self.log_path}/fio_lblk_{tag}.log"
 
+        # verify=md5 explicitly. Without it the k8s FIO job moves IO and checks
+        # nothing -- only the docker path had --verify=md5, buried inside
+        # run_fio_test -- so the "formatted" lane was not a data check on k8s
+        # at all.
+        #
+        # verify_fatal follows the hardware. Where a 4K atomic write IS
+        # guaranteed a mismatch is a real defect and should fail the job
+        # outright; on GCP's pd-balanced disks it can be an artefact of
+        # run_fio_test's own verify_backlog, so the mismatch is recorded and
+        # triaged instead of failing the run. Same rule as _md5_severity, one
+        # decision expressed in both places.
+        fatal = self._md5_severity == "error"
         handle = self._run_fio_dual(
             lvol_name, mount_path=mount, log_path=log, runtime=runtime,
             name=f"lblk{tag}", rw="randrw", bs="4K", numjobs=2, nrfiles=4,
-            size="512M")
+            size="512M", verify="md5", verify_fatal=fatal)
         if hasattr(handle, "join"):
             handle.join()