Fix _maybe_set_distributed_sampler_epoch to handle MultiDataLoader - #1078
Open
tachukao wants to merge 1 commit into
Open
Fix _maybe_set_distributed_sampler_epoch to handle MultiDataLoader#1078tachukao wants to merge 1 commit into
_maybe_set_distributed_sampler_epoch to handle MultiDataLoader#1078tachukao wants to merge 1 commit into
Conversation
Summary: `_maybe_set_distributed_sampler_epoch` only handles plain `torch.utils.data.DataLoader` instances. When the train dataloader is a `MultiDataLoader` (used by the Cascade multi-task pipeline), the `isinstance(dataloader, DataLoader)` check fails silently and `set_epoch()` is never called on any child sampler. This means `DistributedSampler.epoch` stays at 0 for the entire run, so the sampler draws the same random indices every epoch. Fix: recurse into `MultiDataLoader.individual_dataloaders` and call `_maybe_set_distributed_sampler_epoch` on each child. Reviewed By: galrotem Differential Revision: D114246690
|
@tachukao has exported this pull request. If you are a Meta employee, you can view the originating Diff in D114246690. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
_maybe_set_distributed_sampler_epochonly handles plaintorch.utils.data.DataLoaderinstances. When the train dataloader is aMultiDataLoader(used by the Cascade multi-task pipeline), theisinstance(dataloader, DataLoader)check fails silently andset_epoch()is never called on any child sampler.This means
DistributedSampler.epochstays at 0 for the entire run,so the sampler draws the same random indices every epoch.
Fix: recurse into
MultiDataLoader.individual_dataloadersand call_maybe_set_distributed_sampler_epochon each child.Reviewed By: galrotem
Differential Revision: D114246690