Skip to content

Slow saving/loading of binary files #358

Description

@boazmohar

We are dealing with a long but small volumetric data set (~50k-100k time points, ~200Kb per time point).
Currently saving this images object using tobinary() produces a large number of small files, which is slow to read, write and especially delete using our storage back-end.
I suggest to add a parameter that would group n time points in order to reduce the number of files written to disk.
A few points to think about are:

  1. What grouping to use: would a list work or we will need to stack the n dimensional data in the n+1 dimension.
  2. What would be the equivalent series implementation: a grouping factor for each axis?
  3. How to retrieve the original images or series object: change the conf.json file to include this parameter, have a similar parameter in frombinary(), maybe both.

@freeman-lab, @jwittenbach would like to hear what you think before starting to play around with this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions