Add is_valid_national_id row-level check#1346
Open
SreeramaYeshwanthGowd wants to merge 4 commits into
Open
Conversation
SreeramaYeshwanthGowd
requested review from
grusin-db
and removed request for
a team
July 19, 2026 14:45
mwojtyczka
self-requested a review
July 22, 2026 08:00
mwojtyczka
reviewed
Jul 22, 2026
mwojtyczka
approved these changes
Jul 22, 2026
Contributor
There was a problem hiding this comment.
@SreeramaYeshwanthGowd LGTM - I left one comment, can you please address this before we kick the tests and merge this
Author
|
@mwojtyczka thanks, added it. Good to kick off the tests. |
Contributor
Thank you! just started the tests, we can merge once all tests are green |
Author
|
@mwojtyczka the anomaly failure looks unrelated, could you please rerun it? |
Contributor
|
all green now, there are other PRs that needs to be merged before, but nothing blocking this one |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a new row-level check
is_valid_national_id(column, country="US")that validates national identification numbers. Validation covers format, number ranges, and obvious structural errors only; it does not verify that a number was actually issued.The implementation follows the existing pattern-validator idiom (
is_valid_email,is_valid_ipv4_address): a newDQPattern.SSN_US(an anchored, ReDoS-safe regex) plus a small_NATIONAL_ID_PATTERNS_BY_COUNTRYmap keyed by ISO 3166 alpha-2 code. The map is the extension point for the multi-country requirement in the issue; US is implemented here, and other countries can be added by mapping a newDQPatternmember.Behavior:
AAA-GG-SSSSform with a consistent separator (all hyphens, all single spaces, or none), so123-45-6789,123 45 6789and123456789are all valid, but mixed separators are rejected.000,666and900-999(the last covering ITINs), group00, and serial0000.countryis validated the same way ascidr_blockinis_ipv4_address_in_cidr:NoneraisesMissingParameterError, a non-string raisesInvalidParameterError, and an unsupported code raisesInvalidParameterError. It is case-insensitive.Linked issues
Resolves #1265
Tests
manually tested
added unit tests
added integration tests
added end-to-end tests
added performance tests
Unit tests in
tests/unit/test_row_checks.pycover the default, explicit and case-insensitivecountryalias, plus theNone, non-string and unsupported-country error paths.Integration tests in
tests/integration/test_row_checks.pycover the valid/invalid/boundary/null matrix, column-expression input, and lowercase-country normalization.is_valid_national_idis also added totests/resources/all_row_checks.yamland the "all checks" integration tests (test_apply_checks_all_checks_as_yaml,test_apply_checks_all_row_checks_as_yaml_with_streaming, andtest_apply_checks_all_checks_using_classes) so both the declarative metadata path and the DQX classes path are exercised.A performance benchmark (
test_benchmark_is_valid_national_id) was added undertests/perf/, mirroring the existing per-check benchmarks such astest_benchmark_is_valid_email.All test values are synthetic; no real SSNs are used.
Documentation and Demos
docs/dqx/docs/reference/quality_checks.mdxgets a reference-table entry and YAML plus programmatic (DQRowRule) usage examples.