Skip to content

Forward exist_ok in dataframe_to_mds - #991

Open
SiluPanda wants to merge 1 commit into
mosaicml:mainfrom
SiluPanda:fix/dataframe-to-mds-exist-ok
Open

Forward exist_ok in dataframe_to_mds#991
SiluPanda wants to merge 1 commit into
mosaicml:mainfrom
SiluPanda:fix/dataframe-to-mds-exist-ok

Conversation

@SiluPanda

Copy link
Copy Markdown

Summary

  • forward mds_kwargs["exist_ok"] to the initial CloudUploader.get() call
  • add an end-to-end Spark regression covering conversion into an existing, non-empty local output directory
  • preserve the existing default (False) when callers omit exist_ok

Fixes #871.

Validation

  • Fail-first: the new regression raised FileExistsError on the unmodified implementation because CloudUploader.get() received exist_ok=False
  • pytest -o addopts='' tests/base/converters/test_dataframe_to_mds.py (22 passed)
  • pytest -o addopts='' tests/base/converters/test_dataframe_to_mds.py::TestDataFrameToMDS::test_end_to_end_conversion_local_exist_ok (1 passed)
  • Changed-file pre-commit hooks: Ruff, YAPF, isort, syntax/whitespace/license/docformatter checks passed
  • Targeted Pyright 1.1.256: 0 errors
  • git diff --check

This contribution was implemented with AI assistance and manually reviewed and validated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

exist_ok mds_kwargs not passable to dataframe_to_mds()

1 participant