Fork test: PR #1317 - FEATURE: Added profile build for "has_no_outliers" check#1363
Open
mwojtyczka wants to merge 31 commits into
Open
Fork test: PR #1317 - FEATURE: Added profile build for "has_no_outliers" check#1363mwojtyczka wants to merge 31 commits into
mwojtyczka wants to merge 31 commits into
Conversation
…qx into feature/has_no_outliers
Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com>
Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com>
Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com>
Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com>
- Emit gate: use inclusive `<=` for outliers_ratio to match the sibling max_null_ratio / max_empty_ratio gates; update docstring + docs to "at or below". - Validate mutually-exclusive has_no_outliers_allow_columns / _deny_columns once up front (validate_profile_options in DQProfiler.profile) instead of raising per-column mid-profiling; keep the per-column check as defense-in-depth. - Degenerate MAD guard: skip when the band width is non-positive or near-zero relative to the column scale, not only when bounds are exactly equal. - Fix stale _build_profiles_for_column docstring (min_max is no longer last; order is null_or_empty -> is_in -> min_max -> has_no_outliers). - Docs: note that ratio thresholds are evaluated on the sampled data. - Add unit tests: at-threshold emission, near-degenerate MAD skip, and validate_profile_options raise/pass paths. Co-authored-by: Isaac
mwojtyczka
added
do-not-merge
The has_no_outliers profiler feature is opt-in (PROFILE_OPTION_HAS_NO_OUTLIERS defaults to False), but the integration tests were written assuming it was enabled by default. They expected has_no_outliers entries in the profile output (or asserted exactly one profile) without setting the flag, so the builder short-circuited and emitted nothing -> DQProfile list mismatches and assert 0 == 1. Pass "has_no_outliers": True in the options of every profiler test that expects the profile, so they exercise the feature explicitly. Also enable it in the negative threshold test so it actually verifies suppression instead of passing coincidentally while the feature is disabled. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1317
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).