fix(freshness): error on NULL loaded_at instead of reporting a ~56-year staleness - #16587
Open
kiwamizamurai wants to merge 2 commits into
Open
kiwamizamurai wants to merge 2 commits into
kiwamizamurai wants to merge 2 commits into
Conversation
…Unix epoch When `MAX(loaded_at_field)` is NULL (e.g. the source table has no rows), `calculate_freshness_common` read the value with `PrimitiveArray::value(0)`, which ignores the null bitmap and returns 0 (1970-01-01). `dbt source freshness` then reported a bogus ~56 year staleness. Check `is_null(0)` before reading the value and return an explicit error for both `max_loaded_at` and `snapshotted_at`. Resolves dbt-labs#16489 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…elper Keeps `calculate_freshness_common` shorter than before (CodeScene Large Method). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #16489
Problem
When
MAX(loaded_at_field)(or aloaded_at_queryresult) is SQLNULL— e.g. the source table has no rows —calculate_freshness_commonread the value withPrimitiveArray::value(0). That accessor ignores the null bitmap and returns whatever is in the buffer slot, which is0(the Unix epoch).dbt source freshnesstherefore reported a specific-looking but meaningless staleness:Solution
Add a small helper,
extract_non_null_nanos, that casts the column to nanoseconds (as before) and checksis_null(0)before calling.value(0). If the value is NULL it returns an explicit error instead of computing an age from garbage. It is used for bothmax_loaded_atandsnapshotted_at(the latter defensively).No public types, JSON artifacts, or CLI flags change.
Behavior of the error at the call sites (unchanged code, relying on the existing per-node handling in
run_freshness):dbt source freshness: the node is reported as an error with the new message.dbt build/dbt run(freshness collected as part of state-aware orchestration): a warning is emitted and the source is treated as updated, so dependents rebuild. Previously an empty table was read as the epoch.Alternative considered: a dedicated "no data"
FreshnessStatus. I did not do it here because it would touchrun_results/ telemetry / artifacts; happy to follow up if maintainers prefer that direction.Testing
Added unit tests for the helper (NULL → error; present value → correct nanoseconds, via a
Timestamp(Microsecond, UTC)column as in the issue's reproduction).Checklist