Conversation
This reverts commit e328447.
This reverts commit ceb4ad1.
…/charmplusplus/charm into hapi_portable_reconverse_support
#3930) * reconverse-darwin-arm8: compile with C++17 (Apple clang defaults to C++98) Apple clang still defaults to gnu++98; reconverse headers require C++17. Append -std=gnu++17 to CMK_CXX_FLAGS and also to CMK_NATIVE_CXX_FLAGS / CMK_SEQ_CXX_FLAGS, because conv-mach-darwin.sh snapshots those from CMK_CXX_FLAGS before this arch script runs (charmxi builds with NATIVE). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * apply c++17 to all darwin builds --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ritvik Rao <rsrao2@illinois.edu>
Ten phases of a message ring over all PEs, each ended by CkStartQD with a callback that starts the next phase — exercising QD's detect/reset/redetect cycle, which one-shot QD-at-exit usage never tests. A hang means QD state did not reset between detections. The printed qd_settle_ms doubles as a performance canary for transport idle-path latency: healthy runtimes settle in well under a millisecond at small scale; tens of milliseconds indicate the transport delays sparse single messages (measured on reconverse/LCI: ~12.7 ms per idle cross- process hop on IBV, ~5-10 ms on ofi/tcp, vs ~0.5 ms settles on classic netlrts in the same 2-process configuration). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brings HAPI GPU support (CUDA + HIP/ROCm), CUPTI-based GPU load balancing,
the portable hapi_portable.h layer, GPUDirect device RDMA, shrink/expand and
the elastic charmrun onto the reconverse-specific build.
Build-option changes on top of the branch:
- "amd" is the user-facing option for AMD/HIP GPU support, so the build
directory is reconverse-linux-x86_64-amd; "hip" is accepted as an alias.
Both set -DBUILD_HIP=1 internally (conv-mach-hip.sh, CMK_HIP).
- opt_amd added to builddir_extra so the suffix actually lands on the
build directory name.
- BUILD_CUDA / BUILD_HIP now propagate to reconverse as
RECONVERSE_ENABLE_CUDA / RECONVERSE_ENABLE_HIP, which in turn enable
LCI_USE_CUDA / LCI_USE_HIP. Charm++ and reconverse must agree on this
or the GPU-aware RDMA paths silently disappear.
Conflict/typo resolutions:
- ckhello dependency list: kept the reconverse-aware form (no ckqt or
conv-static under reconverse) and added ckrescale plus, for GPU builds,
hybridapi.
- restored 'mpi-cray' in the GDIR match; the branch had backticks around
"mpi", so the ofi-cray/mpi-cray GDIR selection never matched.
- dropped a duplicated conv-rdma.h configure_file in fetch_reconverse.
The cupti_lb_reconverse_hapi branch was only ever built as a GPU
(multicore-linux-x86_64 cuda) target, so several GPU-only constructs had
either no CMK_CUDA/CMK_HIP guard or a guard that never evaluated correctly.
CPU-only builds (reconverse-linux-x86_64):
- pup_util.C and GreedyCentralLB.C included conv-mach-cuda.h and
conv-mach-hip.h directly, which define CMK_CUDA/CMK_HIP to 1
unconditionally. Every build then tried to include hapi_portable.h,
which is only installed into include/ for GPU builds. CMK_CUDA/CMK_HIP
now come only from conv-mach-opt.h, as they should.
- LBManager.C included hapi_portable.h unguarded.
- LBObj::setGPUPupSize, GreedyRefineCentralLB::fillData and one
hapiDeviceSynchronize() in CkLocMgr touched fields/functions that only
exist under CMK_CUDA || CMK_HIP.
GPU builds (reconverse-linux-x86_64-amd):
- cmirdmautils.h gates DeviceRdmaOp/DeviceRecvType on CMK_CUDA||CMK_HIP,
but under reconverse the converse.h on the include path is reconverse's
and does not pull in conv-mach-opt.h. Translation units that include
cmirdmautils.h before charm.h (ckrdma.C) therefore got the non-GPU
version of the header, and then failed in conv-rdmadevice.h where
CMK_HIP was set. cmirdmautils.h and pup_util.C now include
conv-mach-opt.h themselves.
- LBManager.C guarded ProcessorGPUSpeed on CMK_hapi (no such macro, so
CUDA builds silently lost it); now CMK_CUDA || CMK_HIP.
- CentralLB.C had "#if CMK_CUDA || CMK_HIP && CMK_LB_USER_DATA", which
parses as CMK_CUDA || (CMK_HIP && CMK_LB_USER_DATA); parenthesised.
- BUILD_CUDA/BUILD_HIP now imply LBUSERDATA, since the GPU load balancers
read each object's device allocation size out of LB user data.
Also fixes TreeLevel.h's "assert(myObjs.size == nObjs)" (missing call
parens; only compiled in non-production builds).
…CM_PATH The Makefile pointed at a hard-coded /u/ajain18 path and /opt/rocm. Both are now overridable and default to something that works from a checkout.
…l wrote no files) Reconverse's ConverseExit calls no trace hooks, so any trace module whose cleanup depends on the converse-driven global traceClose() silently writes nothing; trace-summary's per-PE .sum/.sumd writes live in SumLogPool's destructor, reached only from that path. Make summary self-closing from its exit function, mirroring the trace-projections pattern: a broadcast closeSummaryOnPe() disables the module, writes the .sts on PE 0, frees the log pool (which writes this PE's files), and reduces into CkContinueExit. In production use since 2026-07-28 (Anvil 80M/2B FoF runs, 120-1920 PEs). Full root-cause analysis and the general reconverse trace-shutdown lesson: UIUC-PPL/charm-notes reconverse-trace-shutdown.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ritvikrao
force-pushed
the
reconverse-specific-build
branch
from
August 20, 2026 20:18
90f05d8 to
90ef159
Compare
Add -c (tokens in flight), -s (payload bytes), -r (laps/phase), -d (pre-messaging busy-wait) plus per-lap timestamps on PE 0 and a local-compute control timed each phase. These are the instruments that characterized the ~1 s-onset per-message latency cliff on Anvil (fixed per-hop adder, transport-side; see charm-notes 2026-08-01/02). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tions Neighbor exchange on a 1D ring + per-step sum reduction (value-checked) + per-step broadcast (exactly-once-checked), with seeded pseudo-random elements migrating via migrateMe() as the last action of the entry method that completes their contribution -- i.e. with the reduction in flight. All checks via CkEnforce; a watchdog dumps per-element protocol state and aborts if no reduction completes for 5 s. Finds a real bug: with many elements migrating per step (-m 1 -c 8, 8-16 elements, 4-8 PEs, single process) the run hangs nondeterministically; the state dump shows every element contributed for step N yet the step-N reduction never reaches the mainchare. Reproduced identically on classic netlrts (step 130) and reconverse (steps 29/46/50/343), so the loss is in shared Ck-layer reduction machinery under concurrent anytime migration, violating manual.rst:1385. Multi-process runs passed 500 steps but are not proven immune. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-B replaces the per-step array broadcast (and finish) with p2p sends so the test runs under +record/+replay, which break on array broadcasts (record: segfault, #3940; replay: silent stall). The hang reproduces in -B mode (4/4 seeds under randomized queues), confirming the loss is in the reduction machinery, not broadcasts. -w <secs> sets the watchdog stall threshold (0 disables, required for record/replay runs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#3939) A PE can end up POPULATED BUT OBLIGATION-FREE for the current reduction: every local element that owed redNo migrated away before contributing, while every arrival already contributed to redNo elsewhere (adj(redNo) balances the lcounts). Such a PE never starts the round: it gets no local contribution, may have no tree kids, and the parent's sendReductionStartingToKids pokes only inactiveList kids -- which this PE never joined because lcount never reached 0. The subtree never reports and the reduction hangs. Diagnosed from a deterministic record-replay of tests/charm++/anytime_bcastred with CMK_DEBUG_REDUCTIONS tracing: the stuck PE's ledger showed three ci(601) leavings and three ci(602) arrivals with redNo pinned at 601, requirement 3-3=0, and every finishReduction early-returning on !inProgress. Fix: when a migration/death event leaves the current round's local requirement already satisfied on a non-barren, not-in-progress PE, eagerly start and finish it (ships the empty result up). A migrant that later owes the round contributes through the existing LateMigrantMsg path; lcount>0 keeps barren PEs on the established inactive-list path, so migration-free programs are unaffected (adj can only go negative via migration). Validated: previously-hanging configs pass on reconverse (1x8PE seeds 7-10, 2-proc heavy x3) and on classic netlrts SMP with randomized msgq (6/6 seeds that hung before, plus 8PE/16-elem/1000-step migrate-all: 6430 migrations), with per-step reduction sums CkEnforce-checked. pingpong 19/19 and tests/charm++/anytime_migration unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…og tolerance - CkMessageRecorder::process: pack/unpack the envelope only when a CRC/checksum is actually computed. The unconditional round trip swaps the envelope out from under CkArrayBroadcaster's retained pointer, segfaulting any program using array broadcasts under +record (#3940); the logged header fields are valid on unpacked envelopes. - flushLog: fflush after write, and quiet the threshold flush, so a small +recplay-logsize yields per-record durable logs -- the main use case is recording a run that will be killed at a hang. - CkMessageReplay::getNext: treat a truncated or empty log as the end sentinel (hold further messages at the recorded frontier) instead of aborting the replay; an SMP comm thread's log is legitimately empty when the recording run was killed. Validated: array-broadcast programs record cleanly; a recorded hang (anytime-migration reduction loss, #3939) replays deterministically to the exact recorded frontier, twice. Replay of broadcast-using programs still stalls (separate defect, see #3940 discussion). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent defects, both broadcast-specific: 1. Silent stall: BocBcastMsg/ArrayBcastMsg ride the node queue and are claimed by an arbitrary rank, which performs the within-node fan-out with its own srcPe/event stamps (_processBocBcastMsg -> _sendMsgBranchWithinNode). The replay watcher's SMP shepherding branch only bounced NodeBocInitMsg/ForNodeBocMsg, so a broadcast claimed by a different rank than in the recording was buffered at the wrong rank forever: the original stranded, the fan-out the other ranks' logs expect never produced, no diagnostic printed (srcPe/event mismatches are silent). Add both types to the bounce set so the recorded winner claims it. 2. Segfault once the stall was fixed: CkMessageReplay::process, like the recorder before 3b5ed5e, round-tripped every unpacked message through CkPackMessage/CkUnpackMessage unconditionally, swapping the envelope out from under CkArrayBroadcaster's retained pointer. Gate the pack on CRC/checksum being enabled, as in the recorder. Validated: bcastred (broadcast mode) records and replays gentle, migrating, and heavy (2000 steps / 10491 migrations) runs to identical completion twice; -B p2p mode and hello unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
opt_reconverse defaulted to 1 ('fake argument for now'), so every build
-- netlrts included -- passed -DRECONVERSE=1, pulled in fetch_reconverse,
and installed reconverse's converse.h over the classic one: ck-core then
failed on _Cmi_sleepOnIdle and classic arches were unbuildable from this
branch. Default it to 0; the network==reconverse case and
--enable-reconverse still turn it on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ng (#3941) Partial fix for classic-arch buildability from this branch: buildcmake defaulted opt_reconverse=1 ('fake argument for now'), so netlrts builds were configured as reconverse builds and got reconverse's converse.h installed over the classic one. With that fixed, configure exposes the deeper state: the classic converse library build is commented out in cmake/converse.cmake (converse is now a custom target wrapping the reconverse external), so classic arches need the classic library build restored, not just header guards. That restoration is better done by landing this branch's reconverse glue on main (where the classic build is intact) than by resurrecting classic here -- see the convergence discussion. Also rights the conv-static dependency (was add_dependencies(reconverse ...) inside if(NOT RECONVERSE)). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
9e48ce9 committed debug hackery into the example (CkExit in the Hello constructor, init callback commented out), making it print 'Hello 0 created' and exit 0 -- masquerading as a runtime failure for anyone using it as a smoke test (#3941 item 2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With a branch name, the same charm commit fetches whatever reconverse's tip is that day: builds are unreproducible and can break silently when the pairing drifts (a fresh clone stopped configuring in Aug 2026 for exactly this reason -- newer reconverse wanted persistent.h). Pin the current reconverse main tip (36ab139, '+stacksize #206'); bumping the pin becomes a deliberate, reviewable one-line commit that CI tests. Development flows are unchanged: --with-fetch-reconverse-dir consumes a local checkout in place and --with-fetch-reconverse-tag still overrides. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
spanningTree.C (in charm_cxx_utils) includes charm++.h -> ckmarshall.h -> CkMarshall.decl.h, which charmxi generates under the ci-generated target; with no target-level dependency a fresh parallel build races the generation. Existing builds have been winning the race by scheduling luck; a clean tree lost it deterministically during the main+glue dry-run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds
reconverse-darwin-arm8andreconverse-linux-x86_64as new build time options for the reconverse build.