Skip to content

Record every cabled pair in a jaccl-ring hostfile - #4269

Open
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:jaccl-ring-full-topology
Open

Record every cabled pair in a jaccl-ring hostfile#4269
erwinzhang7 wants to merge 1 commit into
ml-explore:mainfrom
erwinzhang7:jaccl-ring-full-topology

Conversation

@erwinzhang7

Copy link
Copy Markdown
Contributor

configure_jaccl_ring writes None for every pair that is not a ring neighbour, including
pairs that are physically cabled:

for other in ring:
    if other not in (peer_left, peer_right):
        rdmas.append(None)

configure_jaccl a few lines above emits the full N x N matrix, so connectivity survives in a
mesh hostfile and is discarded in a ring one, before anything in C++ sees it.

The information is already available. IPConfigurator keys on every connected Thunderbolt port
and assigns addresses per cable, and its setup() brings up an interface for each pair it
holds, so the machines are cabled and configured for the whole topology while the hostfile
describes a subset of it.

Nothing changes how the group runs. MLX_JACCL_RING selects the ring, not the shape of
the matrix, and launch_jaccl only requires the matrix to be square with a null diagonal. Both
still hold, ring neighbours keep the width the ring was built with, and a hostfile for a
topology with no extra cables is byte-identical to before.

What it buys is that a link is no longer lost. A subgroup can only contain members that are
directly connected, so the pairs dropped here are the ones Group.split() would need, and the
hostfile is everything the runtime is given. Related to #3205.

The matrix construction moves into jaccl_ring_devices(ring, count, ips), a pure function over
a device map, so it is testable without a cluster. Five tests cover a plain ring, a ring with a
chord, the square-and-null-diagonal contract launch_jaccl enforces, ring ordering, and
multiple cables between neighbours.

configure_jaccl_ring wrote None for every pair that was not a ring
neighbour, including pairs that are physically cabled. configure_jaccl
emits the full matrix, so the connectivity survived in a mesh hostfile
and was discarded in a ring one, before any C++ saw it.

The information was already there. IPConfigurator keys on every connected
Thunderbolt port and assigns addresses for each, and its setup() brings
up an interface for every pair it holds, so the machines are cabled and
configured for the whole topology while the hostfile describes a subset.

Nothing about how the group runs changes. MLX_JACCL_RING selects the
ring, not the shape of the matrix, and launch_jaccl only requires the
matrix to be square with a null diagonal, both of which still hold. The
ring neighbours keep the width the ring was built with.

What it buys is that a link is no longer lost. A subgroup can only
contain members that are directly connected, so the pairs dropped here
are exactly the ones a future split would need, and the hostfile is
everything the runtime is ever given.
@zcbenz zcbenz added the await verification This pull request is non-trivial and requires a human expert to verify its correctness. label Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

await verification This pull request is non-trivial and requires a human expert to verify its correctness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants