Fork test: PR #1309 - Feature/validate upstream tables - #1367
Closed
mwojtyczka wants to merge 17 commits into
Closed
Fork test: PR #1309 - Feature/validate upstream tables#1367mwojtyczka wants to merge 17 commits into
mwojtyczka wants to merge 17 commits into
Conversation
…ng functions as _is_aggr_compare and _get_ref_df, avoiding re-implementing existing logic
…VIVAS/dqx into feature/validate_upstream_tables
- Wire aggr_matches_dataset into the all-dataset-checks coverage: add it to all_dataset_checks.yaml and both all-checks integration tests (test_apply_checks_all_checks_as_yaml already loads the YAML; test_apply_checks_all_checks_using_classes gains the class-based check plus ref_dfs), exercising the declarative/YAML round-trip like the sibling dataset checks. - Add count_distinct + group_by integration tests (match, mismatch, and a NULL group key) to lock in grouped distinct-count behaviour for this check; the NULL-key case asserts flagging behaviour rather than exact message text since the window-incompatible join is not null-safe (documented). - Add performance benchmarks: dataset-wide count comparison and the count_distinct + group_by two-stage join path. Co-authored-by: Isaac
# Conflicts: # src/databricks/labs/dqx/check_funcs.py
mwojtyczka
added
do-not-merge
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1367 +/- ##
==========================================
+ Coverage 91.55% 92.76% +1.21%
==========================================
Files 102 102
Lines 10429 10524 +95
==========================================
+ Hits 9548 9763 +215
+ Misses 881 761 -120
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
✅ 1/1 passed, 19m5s total Running from mcp #63 |
Contributor
|
✅ 821/821 passed, 42 skipped, 6h15m17s total Running from acceptance #5242 |
Contributor
|
✅ 194/194 passed, 2 skipped, 7h22m11s total Running from anomaly #1356 |
The MCP suite (tests/integration_mcp) only ran on PRs touching mcp-server/, src/databricks/labs/dqx/, or the mcp workflow — so nightly never exercised it. Add an mcp-tests job to nightly.yml mirroring the mcp.yml PR job (setup-env, prebuild-wheel, install Databricks CLI, acceptance harness with the integration_mcp codegen), minus the PR-only fork gate since nightly is schedule-triggered. No coverage upload (the MCP server/runner execute remotely). Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync from fork PR for CI testing.
Original PR: #1309
All tests, including unit and integration tests run on this PR (they are skipped for fork PRs).