Persistent device messaging (CkDevicePersistent) aborts for inter-node transfers #3970

Activity

1. lvkale commented on Sep 9, 2026

   

   lvkale

   on Sep 9, 2026

   ContributorAuthor

   More actions

   Using a pre-migration handle is undetected today, and fails silently

   Raised by Kale: forgetting the teardown/rebuild after migration is likely to be a common mistake, so does the runtime say anything useful? Checked — it does not.

   CkDevicePersistent::get validates exactly one thing:

   if (cnt < src.cnt) {
     CkAbort("CkDevicePersistent::get: Destination buffer is smaller than source buffer\n");
   }
   CkNcpyModeDevice mode = findTransferModeDevice(src.pe, CkMyPe());

   and findTransferModeDevice only bounds-checks the index (CmiEnforce((srcPe >= 0) && (srcPe <= CmiNumPes()))). Neither can tell whether the chare that owned src.pe is still there.

   The pup method makes this materially worse, because a stale handle survives migration looking entirely valid:

   void CkDevicePersistent::pup(PUP::er& p) {
     p((char*)&ptr, sizeof(ptr));
     p|cnt;
     p|pe;
     p|cb;
     p((char*)&hapi_ipc_handle, sizeof(hapi_ipc_handle));
   }

   A raw device pointer, a stale pe, and an IPC handle from the old process are all carried to the new PE, fully populated, with nothing marking them as needing recreation. The documented contract — "After the owner chare migrates, CkDevicePersistent needs to be recreated and exchanged again" — is enforced only by the reader remembering it.

   Three distinct failure modes follow, none of which produces a diagnostic:

   1. Mode misclassification. src.pe is stale, so findTransferModeDevice can return MEMCPY when the source has actually moved to another node. hapiMemcpyAsync then dereferences src.ptr — a device pointer from a different process's address space — in this one.
   2. Use-after-free. If the source chare migrated away and freed its buffer, the MEMCPY path reads freed device memory. Silent wrong answers.
   3. Stale IPC mapping. ipc_open and ipc_ptr are cached on the source descriptor, so a mapping into a process that has since freed the allocation keeps being reused.

   A cheap guard for half of it

   The pup method is exactly the moment a handle becomes stale, so one bool closes the case where the holder migrated:

   if (p.isUnpacking()) needs_reopen = true;   // in pup
   // and at the top of get/put:
   if (needs_reopen)
     CkAbort("CkDevicePersistent used after migration: it must be recreated and "
             "re-exchanged after the owning chare moves");

   One flag, one branch, and every case above turns into a message that names the contract.

   The harder half, worth naming

   That guard does not cover the case where the source migrates. Chare B holds an exchanged copy of A's descriptor; if A moves, B's copy is stale but B was never pupped, so B has no local signal at all. Catching that needs either an epoch exchanged with the handle and validated on use, or A invalidating its outstanding copies when it migrates. That is a design question rather than a guard, and it should be settled alongside the inter-node work rather than after it — inter-node persistent plus load balancing is precisely the combination where this arises.

   🤖 Generated with Claude Code