Skip to content

Add runnable distributed examples - #4219

Merged
zcbenz merged 1 commit into
ml-explore:mainfrom
erwinzhang7:distributed-examples
Aug 13, 2026
Merged

Add runnable distributed examples#4219
zcbenz merged 1 commit into
ml-explore:mainfrom
erwinzhang7:distributed-examples

Conversation

@erwinzhang7

Copy link
Copy Markdown
Contributor

Closes #2930. The docs added in #2973 explain data and tensor parallelism, but examples/python
still has nothing you can launch, so this adds two scripts that run as they are:

mlx.launch -n 4 python examples/python/distributed_data_parallel.py
mlx.launch -n 4 python examples/python/distributed_tensor_parallel.py

Both work on any number of ranks including one, and both end on a number you can check rather
than on output you have to take on trust.

distributed_data_parallel.py trains a small MLP with SGD. The dataset is drawn from fixed
keys so every rank starts from the same data, then each rank keeps its own slice and drops the
rest. Averaging the gradients of equal slices gives the gradient of the whole dataset, so the
run is equivalent to single process training and the final loss does not move with the number of
ranks:

ranks loss
1 0.017677
2 0.017679
4 0.017677

The sixth decimal moves because the additions happen in a different order, which the docstring
says.

distributed_tensor_parallel.py shards an MLP across ranks, AllToShardedLinear for the up
projection and ShardedToAllLinear for the down projection, one all reduce per block. It runs
the full model first and prints the largest difference against the sharded one: 0.000e+00 at 1
and 2 ranks, 2.086e-07 at 4, which is float32 reassociation from summing four partial results.

Both scripts call mx.eval on every rank before the rank 0 print. Leaving that to .item()
inside the rank check means only rank 0 forces the collective and the others exit while it
waits, which surfaces as [ring] connection to a peer was lost and does not point anywhere near
the cause. There is a comment saying so in both files, since it is an easy thing to reproduce by
accident.

There is no fully sharded example because RingGroup::sum_scatter throws, so nn.fully_shard
cannot run on the ring backend and mlx.launch uses ring locally.

Verified

macOS 26.6, M5 Max, ring backend via mlx.launch at 1, 2 and 4 ranks.

@erwinzhang7
erwinzhang7 force-pushed the distributed-examples branch from d196aad to 4b8f94e Compare August 13, 2026 02:02

@zcbenz zcbenz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@zcbenz
zcbenz merged commit fd7f023 into ml-explore:main Aug 13, 2026
1 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docs] Simple examples of using mlx.distributed

2 participants