Skip to content

There's no way to depend on a file in the data_tmp directory #6

Description

@krzentner

While jobs are running they write outputs to the data_tmp directory, which are copied over to data when they are finished. This is usually fine, but ideally it would be possible to write cmds that resume another command, and that would probably require being able to check that another command has started and is not running.

Ideally, that would look something like this:

cmd('python', 'my_experiment.py', Out(log_dir),
    priority=base_priority + (0,))

cmd('python', 'my_experiment.py', Out(log_dir),
    '--resume-dir', TmpIn(log_dir),
    priority=base_priority + (1,))

When #5 is fixed, doexp will only start the resume command (and not first command) if the log_dir exists in the temporary directory. Theoretically the above example could break if the log_dir is empty (and thus cannot be resumed from), but I think we should leave that up to the user's script to fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions