Skip to content

Add is_valid_national_id row-level check#1346

Open
SreeramaYeshwanthGowd wants to merge 4 commits into
databrickslabs:mainfrom
SreeramaYeshwanthGowd:feature/1265-is-valid-national-id
Open

Add is_valid_national_id row-level check#1346
SreeramaYeshwanthGowd wants to merge 4 commits into
databrickslabs:mainfrom
SreeramaYeshwanthGowd:feature/1265-is-valid-national-id

Conversation

@SreeramaYeshwanthGowd

@SreeramaYeshwanthGowd SreeramaYeshwanthGowd commented Jul 19, 2026

Copy link
Copy Markdown

Changes

Adds a new row-level check is_valid_national_id(column, country="US") that validates national identification numbers. Validation covers format, number ranges, and obvious structural errors only; it does not verify that a number was actually issued.

The implementation follows the existing pattern-validator idiom (is_valid_email, is_valid_ipv4_address): a new DQPattern.SSN_US (an anchored, ReDoS-safe regex) plus a small _NATIONAL_ID_PATTERNS_BY_COUNTRY map keyed by ISO 3166 alpha-2 code. The map is the extension point for the multi-country requirement in the issue; US is implemented here, and other countries can be added by mapping a new DQPattern member.

Behavior:

  • Accepts the AAA-GG-SSSS form with a consistent separator (all hyphens, all single spaces, or none), so 123-45-6789, 123 45 6789 and 123456789 are all valid, but mixed separators are rejected.
  • Rejects structurally invalid ranges: area 000, 666 and 900-999 (the last covering ITINs), group 00, and serial 0000.
  • country is validated the same way as cidr_block in is_ipv4_address_in_cidr: None raises MissingParameterError, a non-string raises InvalidParameterError, and an unsupported code raises InvalidParameterError. It is case-insensitive.
  • Null column values pass without a violation, consistent with the other pattern checks.

Linked issues

Resolves #1265

Tests

  • manually tested

  • added unit tests

  • added integration tests

  • added end-to-end tests

  • added performance tests

  • Unit tests in tests/unit/test_row_checks.py cover the default, explicit and case-insensitive country alias, plus the None, non-string and unsupported-country error paths.

  • Integration tests in tests/integration/test_row_checks.py cover the valid/invalid/boundary/null matrix, column-expression input, and lowercase-country normalization.

  • is_valid_national_id is also added to tests/resources/all_row_checks.yaml and the "all checks" integration tests (test_apply_checks_all_checks_as_yaml, test_apply_checks_all_row_checks_as_yaml_with_streaming, and test_apply_checks_all_checks_using_classes) so both the declarative metadata path and the DQX classes path are exercised.

  • A performance benchmark (test_benchmark_is_valid_national_id) was added under tests/perf/, mirroring the existing per-check benchmarks such as test_benchmark_is_valid_email.

  • All test values are synthetic; no real SSNs are used.

Documentation and Demos

  • added/updated demos
  • added/updated docs
  • added/updated agent skills

docs/dqx/docs/reference/quality_checks.mdx gets a reference-table entry and YAML plus programmatic (DQRowRule) usage examples.

@SreeramaYeshwanthGowd
SreeramaYeshwanthGowd requested a review from a team as a code owner July 19, 2026 14:45
@SreeramaYeshwanthGowd
SreeramaYeshwanthGowd requested review from grusin-db and removed request for a team July 19, 2026 14:45
@CLAassistant

CLAassistant commented Jul 19, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@mwojtyczka
mwojtyczka self-requested a review July 22, 2026 08:00
Comment thread tests/resources/all_row_checks.yaml

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SreeramaYeshwanthGowd LGTM - I left one comment, can you please address this before we kick the tests and merge this

@mwojtyczka mwojtyczka added the under-review This PR is currently being reviewed by one of DQX maintainers. label Jul 22, 2026
@SreeramaYeshwanthGowd

Copy link
Copy Markdown
Author

@mwojtyczka thanks, added it. Good to kick off the tests.

@mwojtyczka mwojtyczka added Approved to Merge When PR is reviewed and approved. To be merged once all tests pass and removed under-review This PR is currently being reviewed by one of DQX maintainers. labels Jul 22, 2026
@mwojtyczka

Copy link
Copy Markdown
Contributor

@mwojtyczka thanks, added it. Good to kick off the tests.

Thank you! just started the tests, we can merge once all tests are green

@SreeramaYeshwanthGowd

Copy link
Copy Markdown
Author

@mwojtyczka the anomaly failure looks unrelated, could you please rerun it?

@mwojtyczka

Copy link
Copy Markdown
Contributor

all green now, there are other PRs that needs to be merged before, but nothing blocking this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved to Merge When PR is reviewed and approved. To be merged once all tests pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: New check to validate SSN Number

3 participants