Reconfigurable modules #1472

Activity

1. dmpetrov

   added

   feature requestRequesting a new featureRequesting a new feature

   on Jan 7, 2019
2. dmpetrov commented on Jan 9, 2019

   

   dmpetrov

   on Jan 9, 2019

   ContributorAuthor

   More actions

   A possible interface to reconfigurable modules is below. This example is based on https://dvc.org/doc/get-started

   # Clones a repository and pull data for reconfigurable modules with data
   $ dvc clone https://github.com/iterative/so-dataset-posts-25K
   
   $ dvc run -d prepare.py -d so-dataset-posts-25K/data.xml \
           -o data.tsv -o data-test.tsv \
           python prepare.py so-dataset-posts-25K/data.xml
   
   $ dvc clone https://github.com/iterative/text-to-bag-of-words
   
   # Run cloned module instead of:
   # dvc run -d featurization.py -d data.tsv -o matrix.pkl \
   #              python featurization.py data.tsv matrix.pkl
   # -d1 - pass a file as the first module input\dependency (since it can have a few)
   # -o1 - instatiate (create a hardlink) the first module output as a data file
   $ dvc sub text-to-bag-of-words -d1 data.tsv -o1 matrix.pkl \
                 -p columns=1,2 -p lowercase=true -p max_features=9000
   
   # Just a regular run
   $ dvc run -d train.py -d matrix.pkl \
                 -o model.pkl \
                 python train.py matrix.pkl model.pkl

   Details

   The module should not be executed dvc sub from its directory text-to-bag-of-words since a single run might be not enough. Instead, a separate module instance should be created in some directory (let say .dvc/inst/) for each separate run with a unique suffix (for example .dvc/inst/text-to-bag-of-words_8bf3cfe).

   Connection to build cache issue

   The module unique suffix can be based on the module instance config file (not in the example above) and set of params. In such a way DVC can easely identify a similar runs and can be reused as build cache #1234 for a regular runs (not modules).
3. dmpetrov commented on Jan 10, 2019

   

   dmpetrov

   on Jan 10, 2019

   ContributorAuthor

   More actions

   @kskyten I'd love to hear your feedback on this.
4. efiop

   closed this as completedon May 3, 2021
5. treeverse

   locked and limited conversation to collaborators on May 3, 2021