Add valid string case check (#1335) by SyedIshmumAhnaf · Pull Request #1347 · databrickslabs/dqx

SyedIshmumAhnaf · 2026-07-19T18:41:40Z

Changes

Add a has_valid_string_case row-level check for validating consistent string casing.
Support upper, lower, title, and sentence case modes.
Cast input values to strings before normalization and comparison so the check can handle non-string columns consistently.
Validate the case argument and reject unsupported values.
Register the check for declarative rule construction.
Add unit and integration test coverage for the new behavior.
Document the check and update the DQX check-definition agent skill.

Linked issues

Resolves #1335

Tests

Documentation and Demos

added/updated demos
added/updated docs
added/updated agent skills

CLAassistant · 2026-07-19T18:41:48Z

All committers have signed the CLA.

ghanse

Overall looks good. Left a few small comments. Can you add some performance tests for this check?

Here's an example:

dqx/tests/perf/test_apply_checks.py

Lines 2144 to 2167 in 9c8fcaf

    
           @pytest.mark.parametrize( 
        
               "column", 
        
               [ 
        
                   "col1_email_standard", 
        
                   "col2_email_with_quoted_local_part", 
        
                   "col3_email_with_ip_domain", 
        
                   "col4_email_with_multi_part_domain", 
        
               ], 
        
           ) 
        
           @pytest.mark.benchmark(group="test_benchmark_is_valid_email") 
        
           def test_benchmark_is_valid_email(benchmark, ws, generated_email_df, column): 
        
               dq_engine = DQEngine(workspace_client=ws, extra_params=EXTRA_PARAMS) 
        
               checks = [ 
        
                   DQRowRule( 
        
                       name=f"{column}_is_valid_email_address", 
        
                       criticality="warn", 
        
                       check_func=check_funcs.is_valid_email, 
        
                       column=column, 
        
                   ), 
        
               ] 
        
               benchmark.group += f" {column}" 
        
               checked = dq_engine.apply_checks(generated_email_df, checks) 
        
               actual_count = benchmark(lambda: checked.count()) 
        
               assert actual_count == EXPECTED_ROWS

ghanse · 2026-07-20T12:30:06Z

 | `is_older_than_col2_for_n_days`    | Checks whether the values in one input column are at least N days older than the values in another column.                                                                                                                                                                                                                                                                                                                        | `column1`: first column to check (can be a string column name or a column expression); `column2`: second column to check (can be a string column name or a column expression); `days`: number of days; `negate`: if the condition should be negated                                                                                                                                                                                                                                                                                                  |
 | `regex_match`                      | Checks whether the values in the input column match a given regex.                                                                                                                                                                                                                                                                                                                                                                | `column`: column to check (can be a string column name or a column expression); regex: regex to check; `negate`: if the condition should be negated (true) or not                                                                                                                                                                                                                                                                                                                                                                                    |
 | `is_valid_email`                   | Checks whether the values in the input column have valid email address format.                                                                                                                                                                                                                                                                                                                                                    | `column`: column to check (can be a string column name or a column expression)                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `has_valid_string_case`            | Checks whether the values match the requested string case. `upper` and `lower` use the corresponding Spark case functions; `title` uses Spark `initcap`; `sentence` splits on literal `.` characters, cases each segment independently, and preserves delimiters and surrounding whitespace. | `column`: column to check (can be a string column name or a column expression); `case`: one of `upper`, `lower`, `title`, or `sentence` |


I think it's better to say what each case is rather than how it is implemented. For example:

...`upper` and `lower` check that all characters are uppercase or lowercase, respectively; `title` checks that the 1st character of each word is uppercase; `sentence` checks that the 1st character of each sentence is uppercase

ghanse · 2026-07-20T12:32:51Z

 ## Fallback: custom SQL

-**Search `check_funcs` first** — the built-ins cover null/empty, range, set membership, regex, referential, aggregate, uniqueness, schema, freshness, comparison, and outlier cases with typed error messages and tested edge handling. Drop down to SQL only when no built-in fits.
+**Search `check_funcs` first** - the built-ins cover null/empty, string case, range, set membership, regex, referential, aggregate, uniqueness, schema, freshness, comparison, and outlier cases with typed error messages and tested edge handling. Use `has_valid_string_case` for `upper`, `lower`, `title`, or `sentence` string-case validation. Drop down to SQL only when no built-in fits.


Let's remove the specific sentence about case validation. Details about most individual checks are intentionally left out of the skill file to prevent it from getting too large.

ghanse · 2026-07-20T12:40:05Z

+                None,
+            ],
+            [None, None, None, None, None, None],
+            [None, None, None, violation(" first SEGMENT. second SEGMENT ", "sentence", "sentence"), None, None],


I think the title case should have a violation here?

ghanse · 2026-07-20T12:45:09Z

+                lambda segment: F.concat(
+                    F.regexp_extract(segment, r"^(\s*)", 1),
+                    F.upper(F.substring(F.regexp_replace(segment, r"^\s*", ""), 1, 1)),
+                    F.lower(F.substring(F.regexp_replace(segment, r"^\s*", ""), 2, 2147483647)),


Maybe best to use an internal constant for 2147483647

ghanse · 2026-07-20T12:50:12Z

+    elif case == "lower":
+        normalized_case = F.lower(col_expr_cast)
+    elif case == "title":
+        normalized_case = F.initcap(col_expr_cast)


This will flag strings with uppercased abbreviations like "Notes From IEEE Meeting".

Maybe we should use a pattern similar to what's been used for sentence case?

ghanse · 2026-07-20T12:51:43Z

+def test_has_valid_string_case_rule_round_trip():
+    rule = DQRowRule(
+        criticality="warn",
+        check_func=has_valid_string_case,
+        column="a",
+        check_func_kwargs={"case": "lower"},
+    )
+    expected_metadata = [
+        {
+            "name": "a_has_invalid_lower_string_case",
+            "criticality": "warn",
+            "check": {
+                "function": "has_valid_string_case",
+                "arguments": {"column": "a", "case": "lower"},
+            },
+        }
+    ]
+
+    assert serialize_checks([rule]) == expected_metadata
+
+    deserialized_rules = deserialize_checks(expected_metadata)
+    assert len(deserialized_rules) == 1
+    deserialized_rule = deserialized_rules[0]
+    assert deserialized_rule.name == "a_has_invalid_lower_string_case"
+    assert deserialized_rule.check_func is has_valid_string_case
+    assert deserialized_rule.column == "a"
+    assert deserialized_rule.check_func_kwargs == {"case": "lower"}


I don't think we need a specific test for this check. We can add it to the rulesets in the existing tests.

SyedIshmumAhnaf · 2026-07-21T18:17:35Z

Thanks for the review and the helpful comments. I’ll work through the requested changes, add the performance test coverage, and update the PR accordingly.

SyedIshmumAhnaf · 2026-07-23T21:56:55Z

Hi @ghanse, I’ve incorporated the feedback from the review into PR #1347.

The updates include the requested documentation changes, revised title-case behavior for uppercase abbreviations, updated integration expectations, use of an internal substring-length constant, consolidation of the rule-construction coverage, and performance tests for all four supported case modes.

The PR is ready for another review. Thanks!

Add valid string case check (databrickslabs#1335)

141fcb7

SyedIshmumAhnaf requested a review from a team as a code owner July 19, 2026 18:41

SyedIshmumAhnaf requested review from nehamilak-db and removed request for a team July 19, 2026 18:41

SyedIshmumAhnaf mentioned this pull request Jul 19, 2026

[FEATURE]: New check to validate string case #1335

Open

1 task

ghanse self-requested a review July 20, 2026 12:25

ghanse requested changes Jul 20, 2026

View reviewed changes

ghanse added under-review This PR is currently being reviewed by one of DQX maintainers. needs-changes Changes required after review labels Jul 20, 2026

Address string case review feedback

a633efc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add valid string case check (#1335)#1347

Add valid string case check (#1335)#1347
SyedIshmumAhnaf wants to merge 2 commits into
databrickslabs:mainfrom
SyedIshmumAhnaf:feat/valid-string-case-check

SyedIshmumAhnaf commented Jul 19, 2026

Uh oh!

CLAassistant commented Jul 19, 2026 •

edited

Loading

Uh oh!

ghanse left a comment

Uh oh!

ghanse Jul 20, 2026

Uh oh!

ghanse Jul 20, 2026

Uh oh!

ghanse Jul 20, 2026

Uh oh!

ghanse Jul 20, 2026

Uh oh!

ghanse Jul 20, 2026

Uh oh!

ghanse Jul 20, 2026

Uh oh!

SyedIshmumAhnaf commented Jul 21, 2026

Uh oh!

SyedIshmumAhnaf commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	@pytest.mark.parametrize(
	"column",
	[
	"col1_email_standard",
	"col2_email_with_quoted_local_part",
	"col3_email_with_ip_domain",
	"col4_email_with_multi_part_domain",
	],
	)
	@pytest.mark.benchmark(group="test_benchmark_is_valid_email")
	def test_benchmark_is_valid_email(benchmark, ws, generated_email_df, column):
	dq_engine = DQEngine(workspace_client=ws, extra_params=EXTRA_PARAMS)
	checks = [
	DQRowRule(
	name=f"{column}_is_valid_email_address",
	criticality="warn",
	check_func=check_funcs.is_valid_email,
	column=column,
	),
	]
	benchmark.group += f" {column}"
	checked = dq_engine.apply_checks(generated_email_df, checks)
	actual_count = benchmark(lambda: checked.count())
	assert actual_count == EXPECTED_ROWS

Uh oh!

Conversation

SyedIshmumAhnaf commented Jul 19, 2026

Changes

Linked issues

Tests

Documentation and Demos

Uh oh!

CLAassistant commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghanse left a comment

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

SyedIshmumAhnaf commented Jul 21, 2026

Uh oh!

SyedIshmumAhnaf commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented Jul 19, 2026 •

edited

Loading