Repository navigation
Add --times flag to exp run/repro #5608
Description
It would be great to have a --times flag (maybe @jorgeorpinel has a better naming suggestion) that runs an experiment X many times. In other words, dvc exp run --times 5 would run the same experiment 5 times.
This benefits a couple of use cases:
- Checkpoints (see Document checkpoints without make_checkpoint() dvc.org#2292): A user may want to repeat a checkpoint experiment multiple times to run through many epochs rather than having that hard-coded into the script or requiring the user to break out of an infinite loop.
- Non-deterministic stage: A user may want to compare multiple non-deterministic experiment runs. For example, a user could read in random splits of training and test data on each experiment run to perform cross-validation.
One complication is that the first scenario is sequential and the second is parallelizable, but similar concerns are true with --queue/--run-all. Maybe this could even be merged with --queue, like allowing multiple identical experiments to be added to accomplish the same idea.
Thoughts @pmrowla?
Activity
jorgeorpinel commentedon Mar 11, 2021
Current workaround (Linux):
Questions:
As for the name, maybe
--re(peat) <number>dberenbaum commentedon Mar 12, 2021
Good point, but we do support those now with
--always-changed, right?jorgeorpinel commentedon Mar 12, 2021
Is that the use case for
stage add --always-changed?TBH I'm not sure 🤷UPDATE: Doesn't seem like that's the intended use per #2479. Something about forcing "callback stages" (without deps) in the middle of a pipeline.
pmrowla commentedon Mar 12, 2021
IMO this seems like something that should just be done by the user through shell scripting?
dberenbaum commentedon Mar 12, 2021
I see some downstream uses for this, but I can put them in an enhancement proposal with other thoughts on future experiment/checkpoint developments. If you don't mind, we can leave this here until I get to that.
jorgeorpinel commentedon Mar 20, 2021
Apparently it can be difficult to quit shell loops (e.g. a
while true) per treeverse/dvc.org#2292 (comment). That would be an argument for this.pmrowla commentedon Mar 21, 2021
That's specifically when it comes to using an infinite loop in a shell script. This flag would be for a fixed number of iterations, which isn't the same thing.