From 82033f0e0f031870179f7c5f3010b1285556e2a9 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Sat, 1 Aug 2026 16:05:25 +0700 Subject: [PATCH 1/2] fix(relay): enforce scalar-only attribute-release claims Attribute-release profiles could accidentally release structured object/array values through a direct source-field read or a CEL expression. Hold both claim kinds to the same scalar contract the subject side already enforces: a non-scalar value is unavailable, so a required claim denies the release and an optional one is omitted, with a value-free warn naming only the profile, claim, and JSON type tag. The contract is enforced at every layer, not just the HTTP handler: evaluate_release_scalar rejects structured results as a TypeMismatch, config validation (and registryctl check, via the shared validator) rejects expressions that always produce a list or map, and the warn-once dedupe lives on the per-snapshot evaluator so a registry-wide reload re-arms it. Marked BREAKING in the changelog with a migration pointer to the Notary consultation and credential surface, per the issue triage decision. Document the contract in the OpenAPI schemas, the API guide, and the API reference. Closes #88 Signed-off-by: Jeremi Joslin --- crates/registry-relay/CHANGELOG.md | 14 + crates/registry-relay/docs/api.md | 14 + .../openapi/registry-relay.openapi.json | 14 +- .../src/api/attribute_release.rs | 273 +++++++++- crates/registry-relay/src/api/openapi.rs | 36 +- .../src/attribute_release/mod.rs | 189 ++++++- crates/registry-relay/src/config/validate.rs | 19 +- .../tests/attribute_release_api.rs | 509 ++++++++++++++++-- .../registry-relay/tests/config_entities.rs | 14 + .../docs/reference/apis/registry-relay.mdx | 6 + 10 files changed, 1018 insertions(+), 70 deletions(-) diff --git a/crates/registry-relay/CHANGELOG.md b/crates/registry-relay/CHANGELOG.md index de11ec801..a1d30efa6 100644 --- a/crates/registry-relay/CHANGELOG.md +++ b/crates/registry-relay/CHANGELOG.md @@ -2,6 +2,20 @@ ## Unreleased +- BREAKING: Attribute-release claim values are scalar-only: a string, a + number, or a boolean. The stable v0.15.0 contract accepted arbitrary JSON + claim values; a claim whose projected or computed value is an object or an + array is now treated as unavailable instead of being released, so a + required claim of that shape denies the release and an optional one is + omitted, with a value-free warning naming the profile, version, and claim. + A claim expression that always produces a list or map (a top-level literal + or a `map()`/`filter()` comprehension) is rejected at configuration + validation and by `registryctl check`. Profiles that need structured + values should model them in the Registry Notary consultation and + credential surface, where their schema and limits are explicit. The + OpenAPI contract documents the scalar-only claim bundle and top-level-only + claim selection. + ## 0.16.3 - 2026-08-01 - No user-visible Registry Relay changes. The v0.16.2 workflow stopped at an diff --git a/crates/registry-relay/docs/api.md b/crates/registry-relay/docs/api.md index 43f339f9c..28573acad 100644 --- a/crates/registry-relay/docs/api.md +++ b/crates/registry-relay/docs/api.md @@ -508,6 +508,20 @@ registry row, never a raw or hashed subject value. A `source` block only when the profile sets `response.include_source_metadata: true`; it is absent by default. +Claim values are scalar-only: a string, a number, or a boolean. A claim whose +projected or computed value is an object or an array is unavailable — an +optional claim is omitted and a required claim denies the release through the +collapsed `release.subject_denied` below — and the structured content never +reaches the body. A claim expression that always produces a structured value +(a top-level list or map literal, or a `map()`/`filter()` comprehension) is +rejected at configuration validation and by `registryctl check`, before the +route ever serves. A shape only some rows produce is caught at resolve time +and logged once per profile version and claim as +`attribute_release.claim.non_scalar_value`, a value-free operator signal +carrying the profile id, version, claim name, and JSON type tag. Structured +claim values belong to the Registry Notary consultation and credential model, +where their schema and limits are explicit. + Every denial after profile resolution collapses to one public code, so a caller cannot distinguish "no such subject" from "subject exists but was denied": diff --git a/crates/registry-relay/openapi/registry-relay.openapi.json b/crates/registry-relay/openapi/registry-relay.openapi.json index 3934e28f3..f2729188e 100644 --- a/crates/registry-relay/openapi/registry-relay.openapi.json +++ b/crates/registry-relay/openapi/registry-relay.openapi.json @@ -1184,7 +1184,7 @@ "type": "array" }, "claim_names": { - "description": "Names of all claims that may be returned by this profile.", + "description": "Top-level names of all claims that may be returned by this profile. Every released value is a scalar (string, number, or boolean).", "items": { "type": "string" }, @@ -1268,7 +1268,7 @@ "description": "Request body for resolving an attribute release profile against one subject.", "properties": { "claims": { - "description": "Optional subset of claim names to return. Absent means the profile default set; an empty array is rejected (400); duplicate or over-bound arrays are rejected (400); any explicit subset must include every required claim; any unknown claim name is denied.", + "description": "Optional subset of claim names to return. Entries are whole top-level claim names; there is no sub-selection inside a claim value. Absent means the profile default set; an empty array is rejected (400); duplicate or over-bound arrays are rejected (400); any explicit subset must include every required claim; any unknown claim name is denied.", "items": { "type": "string" }, @@ -1314,8 +1314,14 @@ "description": "Resolved attribute release claim bundle. Contains only the approved, minimised claims for the matched subject. Never includes raw source rows, subject identifiers outside released claims, or private source internals.", "properties": { "claims": { - "additionalProperties": true, - "description": "Released claim bundle. Keys are claim names; values are the projected or computed claim values.", + "additionalProperties": { + "type": [ + "string", + "number", + "boolean" + ] + }, + "description": "Released claim bundle. Keys are top-level claim names; values are scalar-only in v1 (string, number, or boolean). Structured object or array values are never released: a claim whose projected or computed value is not a scalar is treated as unavailable, so a required claim of that shape denies the release and an optional one is omitted. Claim selection is by top-level claim name only; there is no sub-selection inside a claim value.", "type": "object" }, "profile_id": { diff --git a/crates/registry-relay/src/api/attribute_release.rs b/crates/registry-relay/src/api/attribute_release.rs index 59bf8e187..ec4581beb 100644 --- a/crates/registry-relay/src/api/attribute_release.rs +++ b/crates/registry-relay/src/api/attribute_release.rs @@ -38,7 +38,7 @@ use crate::api::governed::{ require_governed_read_access, GovernedAccessError, GovernedRedactionProjection, GovernedRequestInfo, }; -use crate::attribute_release::AttributeReleaseEvaluator; +use crate::attribute_release::{AttributeReleaseError, AttributeReleaseEvaluator}; use crate::audit::AuditContextExt; use crate::auth::scopes::require_scope; use crate::auth::Principal; @@ -444,7 +444,8 @@ async fn run_resolve( // 9: project the claim bundle field-by-field. Required claim missing ⇒ // ClaimUnavailable; optional missing ⇒ omit; a claim whose source field is - // dropped by governed redaction is treated as unavailable. + // dropped by governed redaction is treated as unavailable, as is any claim + // whose value is not a scalar (see `scalar_claim_value`). // // Governed redaction is field-layer: `claim_is_redacted` gates direct // claims, while computed claims evaluate over the already-redacted row. @@ -463,7 +464,13 @@ async fn run_resolve( } continue; } - match claim_value(&route.evaluator, claim, &projection_row) { + match claim_value( + &route.evaluator, + claim, + route.profile.id.as_str(), + route.profile.version.as_str(), + &projection_row, + ) { Some(value) => { released.insert(claim.name.clone(), value); } @@ -654,23 +661,121 @@ fn redact_row(row: &Value, redaction_fields: &BTreeSet) -> Value { } /// Compute a single claim value from the projected subject row. A direct claim -/// reads its source field (absent ⇒ `None`); a computed claim evaluates its CEL -/// scalar (any failure ⇒ `None`, so a required computed claim fails closed). +/// reads its source field (absent ⇒ `None`) and is held to the scalar-only +/// claim contract by [`scalar_claim_value`]. A computed claim evaluates its +/// CEL scalar, whose evaluator enforces the same contract itself: a structured +/// result comes back as a `TypeMismatch` and is warned about here, and any +/// other failure ⇒ `None`, so a required computed claim fails closed. fn claim_value( evaluator: &AttributeReleaseEvaluator, claim: &ReleaseClaimConfig, + profile_id: &str, + profile_version: &str, row: &Value, ) -> Option { if let Some(field) = claim.source_field.as_deref() { - return match row.get(field) { - Some(Value::Null) | None => None, - Some(value) => Some(value.clone()), - }; + let value = row.get(field).cloned()?; + return scalar_claim_value(evaluator, value, profile_id, profile_version, &claim.name); } - if let Some(expression) = claim.expression.as_ref() { - return evaluator.evaluate_release_scalar(&expression.cel, row).ok(); + let expression = claim.expression.as_ref()?; + match evaluator.evaluate_release_scalar(&expression.cel, row) { + Ok(value) => Some(value), + Err(AttributeReleaseError::TypeMismatch(diagnostic)) => { + warn_non_scalar_once( + evaluator, + profile_id, + profile_version, + &claim.name, + type_mismatch_kind(&diagnostic), + ); + None + } + Err(_) => None, + } +} + +/// Accept only scalar claim values (string/number/bool), mirroring +/// [`scalar_subject_value`] on the request side. Released claim values are +/// scalar-only in v1: an object or array value is never released, whether it +/// came from a structured source column or from a CEL expression that returned +/// a list or a map. Null is equally unavailable, so a computed claim and a +/// direct claim over a null column behave identically and a claim is never +/// emitted as a literal `null`. The caller applies the usual required ⇒ +/// `ClaimUnavailable` / optional ⇒ omit handling, so this fails closed. +fn scalar_claim_value( + evaluator: &AttributeReleaseEvaluator, + value: Value, + profile_id: &str, + profile_version: &str, + claim_name: &str, +) -> Option { + match value { + Value::String(_) | Value::Number(_) | Value::Bool(_) => Some(value), + Value::Null => None, + structured => { + warn_non_scalar_once( + evaluator, + profile_id, + profile_version, + claim_name, + claim_value_kind(&structured), + ); + None + } + } +} + +/// A structured value is an operator-visible profile configuration signal, so +/// it is logged once per profile version and claim for the life of one +/// runtime snapshot's evaluator, not once per request: steady traffic over a +/// misconfigured claim must not flood the operator log, while a registry-wide +/// reload re-arms the warning instead of staying silent for the life of the +/// process. Profiles are globally identified by `(id, version)` and each +/// version carries its own claims config, so the version is part of both the +/// dedupe key and the record. The record carries structural locators only — +/// profile id, profile version, claim name, JSON type tag — never the value, +/// the row, or the expression text, matching the value-free diagnostics +/// discipline of `crate::attribute_release`. +fn warn_non_scalar_once( + evaluator: &AttributeReleaseEvaluator, + profile_id: &str, + profile_version: &str, + claim_name: &str, + kind: &str, +) { + if evaluator.first_non_scalar_sighting(profile_id, profile_version, claim_name) { + tracing::warn!( + code = "attribute_release.claim.non_scalar_value", + profile_id = profile_id, + profile_version = profile_version, + claim = claim_name, + kind = kind, + "attribute-release claim value is not a scalar and was not released" + ); + } +} + +/// Extract the JSON type tag from an evaluator `TypeMismatch` diagnostic +/// (`field=value expected=scalar kind=array`). Both sides of the contract +/// live in this crate; a diagnostic without a tag falls back to the +/// value-free label `structured`. +fn type_mismatch_kind(diagnostic: &str) -> &str { + diagnostic + .rsplit_once("kind=") + .map_or("structured", |(_, kind)| kind) +} + +/// PII-free JSON type tag for a claim value. Carries the shape only, never the +/// value itself. +fn claim_value_kind(value: &Value) -> &'static str { + match value { + Value::Null => "null", + Value::Bool(_) => "bool", + Value::Number(_) => "number", + Value::String(_) => "string", + Value::Array(_) => "array", + Value::Object(_) => "object", } - None } async fn read_subject_rows(route: &RouteState, subject_value: Value) -> Result, Error> { @@ -931,6 +1036,8 @@ fn with_audit_context(mut response: Response, route: &RouteState, audit: Resolve mod tests { use super::*; + use crate::config::ReleaseExpressionConfig; + #[test] fn subject_audit_raw_canonicalizes_every_accepted_scalar() { assert_eq!( @@ -963,6 +1070,148 @@ mod tests { assert_eq!(error.audit.pdp_audit, Some(audit)); } + fn direct_claim(name: &str, source_field: &str) -> ReleaseClaimConfig { + ReleaseClaimConfig { + name: name.to_string(), + source_field: Some(source_field.to_string()), + expression: None, + required: false, + sensitivity: None, + format: None, + locale: None, + } + } + + #[test] + fn direct_claim_releases_only_scalar_row_values() { + // The scalar-only claim contract: a direct claim clones whatever the row + // holds, so the gate is what keeps a structured source column from being + // released. Required claims turn `None` into ClaimUnavailable upstream. + let evaluator = AttributeReleaseEvaluator::new(); + let claim = direct_claim("attribute", "attribute"); + for scalar in [json!("Ada"), json!(42), json!(true)] { + let row = json!({ "attribute": scalar }); + assert_eq!( + claim_value(&evaluator, &claim, "civil_identity", "v1", &row), + Some(scalar.clone()), + "scalar claim value {scalar} must be released" + ); + } + for non_scalar in [ + json!({"region": "Wonderland"}), + json!(["a", "b"]), + json!(null), + ] { + let row = json!({ "attribute": non_scalar }); + assert_eq!( + claim_value(&evaluator, &claim, "civil_identity", "v1", &row), + None, + "non-scalar claim value {non_scalar} must be unavailable" + ); + } + // An absent field is unavailable too. + assert_eq!( + claim_value(&evaluator, &claim, "civil_identity", "v1", &json!({})), + None + ); + } + + #[test] + fn scalar_claim_value_matches_the_subject_side_scalar_rule() { + // Claim values and subject values share one definition of "scalar", so a + // profile cannot release a shape the request side would have rejected. + let evaluator = AttributeReleaseEvaluator::new(); + for value in [json!("Ada"), json!(42), json!(true)] { + assert_eq!( + scalar_claim_value( + &evaluator, + value.clone(), + "civil_identity", + "v1", + "attribute" + ) + .is_some(), + scalar_subject_value(&value).is_some(), + "claim and subject scalar rules disagree on {value}" + ); + } + for value in [json!({}), json!([]), json!(null)] { + assert!(scalar_claim_value( + &evaluator, + value.clone(), + "civil_identity", + "v1", + "attribute" + ) + .is_none()); + assert!(scalar_subject_value(&value).is_none()); + } + } + + #[test] + fn computed_claim_with_structured_result_is_unavailable() { + // The evaluator rejects a structured computed value as a TypeMismatch; + // the handler treats that claim as unavailable, identically to a + // structured direct projection, so both paths honor one contract. + let evaluator = AttributeReleaseEvaluator::new(); + let structured = ReleaseClaimConfig { + name: "names".to_string(), + source_field: None, + expression: Some(ReleaseExpressionConfig { + cel: "[source.given_name]".to_string(), + }), + required: false, + sensitivity: None, + format: None, + locale: None, + }; + let row = json!({ "given_name": "Ada" }); + assert_eq!( + claim_value(&evaluator, &structured, "civil_identity", "v1", &row), + None + ); + let scalar = ReleaseClaimConfig { + name: "given".to_string(), + source_field: None, + expression: Some(ReleaseExpressionConfig { + cel: "source.given_name".to_string(), + }), + required: false, + sensitivity: None, + format: None, + locale: None, + }; + assert_eq!( + claim_value(&evaluator, &scalar, "civil_identity", "v1", &row), + Some(json!("Ada")) + ); + } + + #[test] + fn type_mismatch_kind_extracts_the_type_tag() { + assert_eq!( + type_mismatch_kind("field=value expected=scalar kind=array"), + "array" + ); + assert_eq!( + type_mismatch_kind("field=value expected=scalar kind=object"), + "object" + ); + // A diagnostic without a kind tag stays value-free and non-empty. + assert_eq!(type_mismatch_kind("unstructured detail"), "structured"); + } + + #[test] + fn claim_value_kind_carries_only_a_type_tag() { + // The only claim-value detail that may reach a log line is its JSON type. + assert_eq!(claim_value_kind(&json!({"region": "Wonderland"})), "object"); + assert_eq!(claim_value_kind(&json!(["ada@example.test"])), "array"); + assert_eq!(claim_value_kind(&json!("Ada")), "string"); + assert_eq!(claim_value_kind(&json!(42)), "number"); + assert_eq!(claim_value_kind(&json!(true)), "bool"); + assert_eq!(claim_value_kind(&json!(null)), "null"); + } + #[test] fn every_accepted_subject_has_an_audit_raw() { // The invariant the audit-canonicalization fix guarantees: any value diff --git a/crates/registry-relay/src/api/openapi.rs b/crates/registry-relay/src/api/openapi.rs index ebb6592e2..7e48141a2 100644 --- a/crates/registry-relay/src/api/openapi.rs +++ b/crates/registry-relay/src/api/openapi.rs @@ -5299,7 +5299,9 @@ fn attribute_release_profile_schema() -> Value { }, "claim_names": { "type": "array", - "description": "Names of all claims that may be returned by this profile.", + "description": "Top-level names of all claims that may be returned by this \ + profile. Every released value is a scalar (string, number, or \ + boolean).", "items": { "type": "string" } }, "required_claims": { @@ -5349,11 +5351,12 @@ fn attribute_release_resolve_request_schema() -> Value { }, "claims": { "type": ["array", "null"], - "description": "Optional subset of claim names to return. Absent means the \ - profile default set; an empty array is rejected (400); \ - duplicate or over-bound arrays are rejected (400); any \ - explicit subset must include every required claim; any unknown \ - claim name is denied.", + "description": "Optional subset of claim names to return. Entries are whole \ + top-level claim names; there is no sub-selection inside a claim \ + value. Absent means the profile default set; an empty array is \ + rejected (400); duplicate or over-bound arrays are rejected \ + (400); any explicit subset must include every required claim; \ + any unknown claim name is denied.", "items": { "type": "string" }, "minItems": 1, "maxItems": MAX_ATTRIBUTE_RELEASE_CLAIMS, @@ -5384,9 +5387,15 @@ fn attribute_release_resolve_response_schema() -> Value { }, "claims": { "type": "object", - "description": "Released claim bundle. Keys are claim names; values are the \ - projected or computed claim values.", - "additionalProperties": true + "description": "Released claim bundle. Keys are top-level claim names; values \ + are scalar-only in v1 (string, number, or boolean). Structured \ + object or array values are never released: a claim whose \ + projected or computed value is not a scalar is treated as \ + unavailable, so a required claim of that shape denies the \ + release and an optional one is omitted. Claim selection is by \ + top-level claim name only; there is no sub-selection inside a \ + claim value.", + "additionalProperties": { "type": ["string", "number", "boolean"] } }, "source": { "type": "object", @@ -6522,6 +6531,15 @@ mod tests { ); assert_eq!(claims_schema["uniqueItems"], true); + // Released claim values are scalar-only in v1. The contract must not + // advertise structured values the runtime never releases. + assert_eq!( + schemas["AttributeReleaseResolveResponse"]["properties"]["claims"] + ["additionalProperties"]["type"], + json!(["string", "number", "boolean"]), + "released claim values must document the scalar-only contract" + ); + // Required fields on AttributeReleaseProfile schema let profile_required = &schemas["AttributeReleaseProfile"]["required"]; for field in [ diff --git a/crates/registry-relay/src/attribute_release/mod.rs b/crates/registry-relay/src/attribute_release/mod.rs index 52519f6bc..697696bc5 100644 --- a/crates/registry-relay/src/attribute_release/mod.rs +++ b/crates/registry-relay/src/attribute_release/mod.rs @@ -66,7 +66,7 @@ mod enabled { use cel::common::ast::{EntryExpr, Expr, IdedExpr}; use std::collections::{BTreeSet, HashMap}; - use std::sync::{Arc, RwLock}; + use std::sync::{Arc, Mutex, PoisonError, RwLock}; use serde_json::{json, Value}; @@ -85,6 +85,7 @@ mod enabled { pub struct AttributeReleaseEvaluator { runtime: Arc, cache: RwLock>>, + non_scalar_sightings: Mutex>, } impl std::fmt::Debug for AttributeReleaseEvaluator { @@ -108,6 +109,7 @@ mod enabled { Self { runtime: Arc::new(MappingRuntime::new(RuntimeOptions::default())), cache: RwLock::new(HashMap::new()), + non_scalar_sightings: Mutex::new(BTreeSet::new()), } } @@ -151,7 +153,10 @@ mod enabled { /// the raw [`Value`]. Fails closed: a missing or erroring expression /// returns `Err` rather than silently dropping the claim. A JSON `null` /// result is treated as a missing value (fail-closed) so an absent - /// computed claim is never silently emitted as `null`. + /// computed claim is never silently emitted as `null`. A structured + /// (array or object) result is a `TypeMismatch`: released claim values + /// are scalar-only, and the invariant is enforced here so this public + /// evaluator can never disagree with the HTTP response contract. pub fn evaluate_release_scalar( &self, cel: &str, @@ -163,9 +168,38 @@ mod enabled { "field=value kind=null".to_string(), )); } + if value.is_array() || value.is_object() { + return Err(AttributeReleaseError::TypeMismatch(scalar_type_diagnostic( + &value, + ))); + } Ok(value) } + /// Records a non-scalar claim sighting for a (profile id, version, + /// claim) triple and reports whether it is the first seen by this + /// evaluator, so the caller's warn line fires once per triple. The + /// evaluator lives exactly as long as one validated runtime snapshot: + /// a registry-wide reload builds a fresh evaluator, which re-arms the + /// warning for corrected-then-reintroduced configuration and drops + /// dedupe state for profiles that no longer exist, keeping the set + /// bounded by the active snapshot's profile-version/claim cardinality. + pub fn first_non_scalar_sighting( + &self, + profile_id: &str, + profile_version: &str, + claim_name: &str, + ) -> bool { + self.non_scalar_sightings + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(( + profile_id.to_string(), + profile_version.to_string(), + claim_name.to_string(), + )) + } + /// Synthesize the one-field document, compile it into this snapshot's /// cache, evaluate it against `record`, and read the single output field /// back. Shared by both predicate and scalar entry points so the @@ -238,15 +272,55 @@ mod enabled { } } - /// Compile-only validation hook used at config load. Fails closed: an - /// expression that does not compile is rejected before the runtime serves - /// any request. + /// Validation hook used at config load (and by `registryctl check`, which + /// calls this function for authored profiles). Fails closed: an expression + /// that does not compile, escapes the `source` authority, or is statically + /// guaranteed to produce a structured value is rejected before the runtime + /// serves any request. pub fn validate_release_expression(cel: &str) -> Result<(), AttributeReleaseError> { validate_expression_authority(cel)?; + validate_expression_shape(cel)?; let runtime = MappingRuntime::new(RuntimeOptions::default()); compile(&runtime, cel).map(|_| ()) } + /// Reject expressions whose top-level result is statically known to be + /// structured. A list literal, a map or message literal, or a `map()` / + /// `filter()` comprehension always produces a list or map, which the + /// scalar-only claim contract can never release and a release predicate + /// can never accept, so the authoring mistake surfaces at config load + /// instead of at resolve time. Shapes that depend on data (member selects, + /// conditionals, function results) cannot be decided here and stay guarded + /// by the resolve-time scalar enforcement in `evaluate_release_scalar`. + fn validate_expression_shape(cel: &str) -> Result<(), AttributeReleaseError> { + let program = cel::Program::compile(cel) + .map_err(|_| AttributeReleaseError::Compile("kind=parse".to_string()))?; + if statically_structured_result(program.expression()) { + return Err(AttributeReleaseError::Compile( + "kind=structured_shape".to_string(), + )); + } + Ok(()) + } + + /// True when the expression's result is a list or map on its face. The + /// comprehension arm matches the `map()`/`filter()` macro expansion (a + /// list-literal accumulator returned as the result) without catching the + /// bool-valued `all()`/`exists()` macros, whose accumulator is a scalar. + fn statically_structured_result(expression: &IdedExpr) -> bool { + match &expression.expr { + Expr::List(_) | Expr::Map(_) | Expr::Struct(_) => true, + Expr::Comprehension(comprehension) => { + matches!(comprehension.accu_init.expr, Expr::List(_)) + && matches!( + &comprehension.result.expr, + Expr::Ident(name) if *name == comprehension.accu_var + ) + } + _ => false, + } + } + /// Restrict release expressions to the projected `source` object. The /// Crosswalk runtime also offers a `context` object, but Relay does not /// define an attribute-release authority contract for it. Rejecting any @@ -482,15 +556,27 @@ mod enabled { /// PII-free diagnostic for a predicate that resolved to a non-boolean value. /// Carries only the JSON type tag, never the value itself. fn predicate_type_diagnostic(value: &Value) -> String { - let kind = match value { + format!("field=value expected=bool kind={}", value_kind(value)) + } + + /// PII-free diagnostic for a claim expression that produced a structured + /// value where the scalar-only contract requires a string, number, or + /// boolean. Shape only, never the value. + fn scalar_type_diagnostic(value: &Value) -> String { + format!("field=value expected=scalar kind={}", value_kind(value)) + } + + /// PII-free JSON type tag for a produced value: the shape only, never the + /// content. + fn value_kind(value: &Value) -> &'static str { + match value { Value::Null => "null", Value::Bool(_) => "bool", Value::Number(_) => "number", Value::String(_) => "string", Value::Array(_) => "array", Value::Object(_) => "object", - }; - format!("field=value expected=bool kind={kind}") + } } #[cfg(test)] @@ -527,6 +613,93 @@ mod enabled { assert_eq!(value, json!("Ada Lovelace")); } + #[test] + fn scalar_rejects_structured_values() { + // The public evaluator enforces the scalar-only release contract + // itself: an expression that produces a list or a map is a + // TypeMismatch, never a successfully returned structured value, + // so the Rust API cannot disagree with the HTTP response contract. + let record = json!({ "given_name": "Ada", "surname": "Lovelace" }); + let evaluator = AttributeReleaseEvaluator::new(); + let err = evaluator + .evaluate_release_scalar("[source.given_name, source.surname]", &record) + .expect_err("list result must be a type mismatch"); + assert!(matches!( + &err, + AttributeReleaseError::TypeMismatch(detail) if detail.contains("kind=array") + )); + let err = evaluator + .evaluate_release_scalar("{'name': source.given_name}", &record) + .expect_err("map result must be a type mismatch"); + assert!(matches!( + &err, + AttributeReleaseError::TypeMismatch(detail) if detail.contains("kind=object") + )); + } + + #[test] + fn validation_rejects_statically_structured_shapes() { + // A top-level list literal, map literal, or map()/filter() + // comprehension always produces a structured value the scalar-only + // claim contract can never release and a release predicate can + // never accept, so the authoring mistake fails at config load (and + // through `registryctl check`, which calls this validator) instead + // of resolving ambiguously in production. + for expression in [ + "[source.given_name, source.surname]", + "{'name': source.given_name}", + "source.items.map(item, item.name)", + "source.items.filter(item, item.active)", + ] { + let err = match validate_release_expression(expression) { + Err(err) => err, + Ok(()) => panic!("structured shape must be rejected: {expression}"), + }; + assert!( + matches!( + &err, + AttributeReleaseError::Compile(detail) + if detail.as_str() == "kind=structured_shape" + ), + "wrong rejection for {expression}: {err:?}" + ); + } + // Bool-valued macros and data-dependent shapes stay accepted: they + // are guarded by the resolve-time scalar enforcement instead. + for expression in [ + "source.given_name + ' ' + source.surname", + "source.items.exists(item, item.active)", + "source.active ? source.given_name : source.surname", + ] { + validate_release_expression(expression).unwrap_or_else(|_| { + panic!("scalar-shaped expression must validate: {expression}") + }); + } + } + + #[test] + fn non_scalar_sightings_reset_with_the_evaluator() { + // Warn-once dedupe lives on the evaluator, which is built per + // runtime snapshot: a registry-wide reload re-arms the warning for + // corrected-then-reintroduced configuration and drops dedupe state + // for profiles that no longer exist, instead of accumulating + // process-lived state. Profiles are globally identified by + // `(id, version)` and each version carries its own claims config, + // so a different version of the same profile id is a distinct + // signal, as is a different claim or profile. + let first = AttributeReleaseEvaluator::new(); + assert!(first.first_non_scalar_sighting("profile", "v1", "claim_a")); + assert!(!first.first_non_scalar_sighting("profile", "v1", "claim_a")); + assert!(first.first_non_scalar_sighting("profile", "v2", "claim_a")); + assert!(first.first_non_scalar_sighting("profile", "v1", "claim_b")); + + let reloaded = AttributeReleaseEvaluator::new(); + assert!( + reloaded.first_non_scalar_sighting("profile", "v1", "claim_a"), + "a fresh evaluator must re-arm the warning" + ); + } + #[test] fn invalid_cel_rejected_at_compile() { let err = validate_release_expression("source.given_name +") diff --git a/crates/registry-relay/src/config/validate.rs b/crates/registry-relay/src/config/validate.rs index f9fe18519..255f50e93 100644 --- a/crates/registry-relay/src/config/validate.rs +++ b/crates/registry-relay/src/config/validate.rs @@ -4018,22 +4018,33 @@ fn validate_release_profile_expressions( { let _ = has_expression; if let Some(conditions) = profile.release_conditions.as_ref() { - compile_release_expression(dataset, entity, profile, &conditions.expression.cel)?; + compile_release_expression( + dataset, + entity, + profile, + "release_conditions", + &conditions.expression.cel, + )?; } for claim in &profile.claims { if let Some(expression) = claim.expression.as_ref() { - compile_release_expression(dataset, entity, profile, &expression.cel)?; + compile_release_expression(dataset, entity, profile, &claim.name, &expression.cel)?; } } Ok(()) } } +/// `location` addresses the failing expression inside the profile: the claim +/// name for a claim expression, or the literal `release_conditions` for the +/// release predicate, so an operator can find the field without the value or +/// the expression text ever reaching the log. #[cfg(feature = "attribute-release")] fn compile_release_expression( dataset: &DatasetConfig, entity: &EntityConfig, profile: &AttributeReleaseProfile, + location: &str, cel: &str, ) -> Result<(), ConfigError> { if cel.is_empty() || cel.len() > 4096 { @@ -4042,6 +4053,7 @@ fn compile_release_expression( dataset_id = %dataset.id, entity = %entity.name, profile_id = %profile.id, + expression = %location, "attribute_release_profiles CEL expression must contain between one and 4096 bytes" ); return Err(ConfigError::ValidationError); @@ -4052,8 +4064,9 @@ fn compile_release_expression( dataset_id = %dataset.id, entity = %entity.name, profile_id = %profile.id, + expression = %location, error = %err, - "attribute_release_profiles CEL expression failed to compile" + "attribute_release_profiles CEL expression failed to validate" ); ConfigError::ValidationError }) diff --git a/crates/registry-relay/tests/attribute_release_api.rs b/crates/registry-relay/tests/attribute_release_api.rs index ef4d1ea6d..7ab29e1db 100644 --- a/crates/registry-relay/tests/attribute_release_api.rs +++ b/crates/registry-relay/tests/attribute_release_api.rs @@ -17,7 +17,7 @@ use axum::http::StatusCode; use axum::Extension; use axum_test::TestServer; use bytes::Bytes; -use datafusion::arrow::array::StringArray; +use datafusion::arrow::array::{ArrayRef, ListBuilder, StringArray, StringBuilder, StructArray}; use datafusion::arrow::datatypes::{DataType, Field, Schema}; use datafusion::arrow::record_batch::RecordBatch; use datafusion::datasource::MemTable; @@ -69,11 +69,49 @@ impl From for TestServerBuildError { } } +/// Optional fixture extensions. The default (no extension) reproduces the base +/// fixture exactly, so every pre-existing test keeps its original config shape. +#[derive(Default)] +struct ConfigExtras { + /// Expose two structured source columns (`address`, `contact_points`) on the + /// table and the entity, and back them with Arrow struct/list columns. The + /// config field type vocabulary is scalar-only, so the columns are declared + /// as `string`: this mirrors a real source whose physical column shape is + /// richer than the declared schema. + structured_fields: bool, + /// Extra release profiles appended to the entity's profile list. + profiles: String, +} + /// A two-row civil-registry config with one release profile. `deceased` /// drives the release-condition predicate; `given_name`/`surname` back direct /// and computed claims. The `optional_note` claim is optional and absent on the /// stored row so it is omitted from a successful release. -fn release_config(entity_api_extra: &str, include_source_metadata: bool, purpose: &str) -> String { +fn release_config( + entity_api_extra: &str, + include_source_metadata: bool, + purpose: &str, + extras: &ConfigExtras, +) -> String { + let extra_schema_fields = if extras.structured_fields { + r#" - name: address + type: string + nullable: true + - name: contact_points + type: string + nullable: true +"# + } else { + "" + }; + let extra_entity_fields = if extras.structured_fields { + r#" - name: address + - name: contact_points +"# + } else { + "" + }; + let extra_profiles = extras.profiles.as_str(); format!( r#" server: @@ -133,7 +171,7 @@ datasets: - name: deceased type: string nullable: false - entities: +{extra_schema_fields} entities: - name: person table: persons_table fields: @@ -143,7 +181,7 @@ datasets: - name: given_name - name: surname - name: deceased - access: +{extra_entity_fields} access: metadata_scope: civil_registry:metadata aggregate_scope: civil_registry:aggregate read_scope: {READ_SCOPE} @@ -181,35 +219,99 @@ datasets: required: false response: include_source_metadata: {include_source_metadata} -"# +{extra_profiles}"# ) } -/// Build a two-row table: one live subject (`NID-1`) and one deceased subject +/// Build a four-row table: one live subject (`NID-1`) and one deceased subject /// (`NID-DEAD`). `NID-DUP` is duplicated to exercise the ambiguity gate. -fn batch(schema: &Arc) -> RecordBatch { - RecordBatch::try_new( - Arc::clone(schema), - vec![ +/// +/// `structured` appends the two non-scalar source columns matched by +/// [`ConfigExtras::structured_fields`]: an `address` struct and a +/// `contact_points` string list. The Arrow schema is derived from the built +/// arrays so the struct/list child fields always line up. +fn schema_and_batch(structured: bool) -> (Arc, RecordBatch) { + let mut columns: Vec<(&str, ArrayRef)> = vec![ + ( + "person_id", Arc::new(StringArray::from(vec!["p1", "p2", "p3", "p4"])), + ), + ( + "national_id", Arc::new(StringArray::from(vec![ "NID-1", "NID-DEAD", "NID-DUP", "NID-DUP", ])), + ), + ( + "given_name", Arc::new(StringArray::from(vec!["Ada", "Grace", "Alan", "Alan"])), + ), + ( + "surname", Arc::new(StringArray::from(vec![ "Lovelace", "Hopper", "Turing", "Turing", ])), + ), + ( + "deceased", Arc::new(StringArray::from(vec!["false", "true", "false", "false"])), - ], + ), + ]; + if structured { + let address: ArrayRef = Arc::new(StructArray::from(vec![ + ( + Arc::new(Field::new("region", DataType::Utf8, true)), + Arc::new(StringArray::from(vec![ + "Wonderland", + "Elsewhere", + "Elsewhere", + "Elsewhere", + ])) as ArrayRef, + ), + ( + Arc::new(Field::new("postal_code", DataType::Utf8, true)), + Arc::new(StringArray::from(vec!["W-1", "E-1", "E-2", "E-3"])) as ArrayRef, + ), + ])); + let mut contact_points = ListBuilder::new(StringBuilder::new()); + for value in [ + "ada@example.test", + "grace@example.test", + "a@x.test", + "b@x.test", + ] { + contact_points.values().append_value(value); + contact_points.append(true); + } + columns.push(("address", address)); + columns.push(("contact_points", Arc::new(contact_points.finish()))); + } + let schema = Arc::new(Schema::new( + columns + .iter() + .map(|(name, array)| Field::new(*name, array.data_type().clone(), true)) + .collect::>(), + )); + let batch = RecordBatch::try_new( + Arc::clone(&schema), + columns.into_iter().map(|(_, array)| array).collect(), ) - .expect("batch") + .expect("batch"); + (schema, batch) } async fn try_server_with_scopes_and_extra( scopes: &[&str], entity_api_extra: &str, ) -> Result { - try_server_full(scopes, entity_api_extra, true, None).await + try_server_full( + scopes, + entity_api_extra, + true, + None, + &ConfigExtras::default(), + ) + .await } /// Like [`try_server_with_scopes_and_extra`] but with explicit control over the @@ -222,6 +324,7 @@ async fn try_server_full( entity_api_extra: &str, include_source_metadata: bool, purpose: Option<&str>, + extras: &ConfigExtras, ) -> Result { let tmp = TempDir::new().expect("tempdir"); let config_path = tmp.path().join("release.yaml"); @@ -231,6 +334,7 @@ async fn try_server_full( entity_api_extra, include_source_metadata, purpose.unwrap_or("identity"), + extras, ), ) .expect("write config"); @@ -243,21 +347,13 @@ async fn try_server_full( let ctx = Arc::new(SessionContext::new()); let dataset: DatasetId = id("civil_registry"); let resource: ResourceId = id("persons_table"); - let schema = Arc::new(Schema::new(vec![ - Field::new("person_id", DataType::Utf8, false), - Field::new("national_id", DataType::Utf8, false), - Field::new("given_name", DataType::Utf8, false), - Field::new("surname", DataType::Utf8, false), - Field::new("deceased", DataType::Utf8, false), - ])); + let (schema, batch) = schema_and_batch(extras.structured_fields); let ingest_ulid = Ulid::from_string("01J5K8M0000000000000000000").expect("ulid"); register_versioned_table( &ctx, table_name(&dataset, &resource), ingest_ulid, - Arc::new( - MemTable::try_new(Arc::clone(&schema), vec![vec![batch(&schema)]]).expect("memtable"), - ), + Arc::new(MemTable::try_new(Arc::clone(&schema), vec![vec![batch]]).expect("memtable")), ) .expect("register"); let mut snapshot = ReadinessSnapshot::default(); @@ -335,7 +431,7 @@ async fn resolve_omits_source_block_when_metadata_disabled() { // an eSignet authenticator profile), the claim bundle is still released but // the source block — which would disclose the backing dataset/entity names — // is suppressed entirely. - let server = try_server_full(&[RELEASE_SCOPE], "", false, None) + let server = try_server_full(&[RELEASE_SCOPE], "", false, None, &ConfigExtras::default()) .await .expect("test server builds"); let response = server.post(RESOLVE_PATH).json(&subject_body("NID-1")).await; @@ -578,6 +674,7 @@ async fn resolve_required_claim_missing_denies() { "#, true, Some("identity"), + &ConfigExtras::default(), ) .await .expect("test server builds"); @@ -605,6 +702,7 @@ async fn resolve_optional_claim_omitted_when_source_redacted() { "#, true, Some("identity"), + &ConfigExtras::default(), ) .await .expect("test server builds"); @@ -640,6 +738,7 @@ async fn resolve_computed_claim_cannot_read_redacted_field() { "#, true, Some("identity"), + &ConfigExtras::default(), ) .await .expect("test server builds"); @@ -679,6 +778,7 @@ async fn resolve_release_condition_cannot_read_redacted_field() { "#, true, Some("identity"), + &ConfigExtras::default(), ) .await .expect("test server builds"); @@ -692,6 +792,328 @@ async fn resolve_release_condition_cannot_read_redacted_field() { assert_eq!(response.json::()["code"], "release.subject_denied"); } +// --------------------------------------------------------------------------- +// Scalar-only claim values +// --------------------------------------------------------------------------- + +/// Profiles whose direct claims project the two structured source columns. None +/// of them declares CEL, so the read projects only the referenced fields and no +/// expression ever sees a structured input. +fn structured_direct_profiles() -> String { + format!( + r#" - id: structured-direct + version: v1 + purpose: identity + release_scope: {RELEASE_SCOPE} + subject: + source_field: national_id + id_type: NATIONAL_ID + claims: + - name: given_name + source_field: given_name + required: true + - name: address + source_field: address + required: false + - name: contact_points + source_field: contact_points + required: false + response: + include_source_metadata: false + - id: structured-direct-required-object + version: v1 + purpose: identity + release_scope: {RELEASE_SCOPE} + subject: + source_field: national_id + id_type: NATIONAL_ID + claims: + - name: address + source_field: address + required: true + response: + include_source_metadata: false + - id: structured-direct-required-array + version: v1 + purpose: identity + release_scope: {RELEASE_SCOPE} + subject: + source_field: national_id + id_type: NATIONAL_ID + claims: + - name: contact_points + source_field: contact_points + required: true + response: + include_source_metadata: false +"# + ) +} + +/// Profiles whose computed claims evaluate to non-scalar or null values at +/// resolve time. Every expression is statically scalar-shaped (member selects +/// and conditionals), so config validation accepts it; the structured source +/// columns supply the non-scalar shape only once a real row is evaluated. +/// Expressions that are structured on their face (a list or map literal) never +/// get this far: config load rejects them. +fn computed_shape_profiles() -> String { + format!( + r#" - id: computed-shapes + version: v1 + purpose: identity + release_scope: {RELEASE_SCOPE} + subject: + source_field: national_id + id_type: NATIONAL_ID + claims: + - name: given_name + source_field: given_name + required: true + - name: computed_list + expression: + cel: "source.contact_points" + required: false + - name: computed_map + expression: + cel: "source.address" + required: false + - name: computed_null + expression: + cel: "source.given_name == 'nobody' ? source.given_name : null" + required: false + - name: computed_text + expression: + cel: "source.given_name" + required: false + - name: computed_number + expression: + cel: "size(source.given_name)" + required: false + - name: computed_flag + expression: + cel: "source.deceased == 'false'" + required: false + response: + include_source_metadata: false + - id: computed-required-list + version: v1 + purpose: identity + release_scope: {RELEASE_SCOPE} + subject: + source_field: national_id + id_type: NATIONAL_ID + claims: + - name: computed_list + expression: + cel: "source.contact_points" + required: true + response: + include_source_metadata: false +"# + ) +} + +fn resolve_path(profile_id: &str) -> String { + format!("/v1/attribute-releases/{profile_id}/versions/v1/resolve") +} + +async fn structured_direct_server() -> TestServer { + try_server_full( + &[RELEASE_SCOPE], + "", + false, + None, + &ConfigExtras { + structured_fields: true, + profiles: structured_direct_profiles(), + }, + ) + .await + .expect("test server builds") +} + +async fn computed_shape_server() -> TestServer { + try_server_full( + &[RELEASE_SCOPE], + "", + false, + None, + &ConfigExtras { + structured_fields: true, + profiles: computed_shape_profiles(), + }, + ) + .await + .expect("test server builds") +} + +#[tokio::test] +async fn resolve_omits_optional_direct_claims_with_structured_values() { + // Claim values are scalar-only in v1. A direct claim whose source column + // holds an object (`address`) or an array (`contact_points`) is unavailable, + // so an optional claim of that shape is omitted and its structured content + // never reaches the body. + let server = structured_direct_server().await; + let response = server + .post(&resolve_path("structured-direct")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::OK); + let body: Value = response.json(); + let claims = body["claims"].as_object().expect("claims object"); + assert_eq!(claims["given_name"], "Ada"); + assert!( + !claims.contains_key("address"), + "object-valued direct claim must be omitted: {body}" + ); + assert!( + !claims.contains_key("contact_points"), + "array-valued direct claim must be omitted: {body}" + ); + let serialized = body.to_string(); + assert!( + !serialized.contains("Wonderland") && !serialized.contains("ada@example.test"), + "structured source content must never be released: {serialized}" + ); +} + +#[tokio::test] +async fn resolve_denies_required_direct_object_claim() { + // Required + unavailable is the ClaimUnavailable path: a collapsed + // 403 release.subject_denied, identical to any other unavailable claim. + let server = structured_direct_server().await; + let response = server + .post(&resolve_path("structured-direct-required-object")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::FORBIDDEN); + let body: Value = response.json(); + assert_eq!(body["code"], "release.subject_denied"); + assert!( + !body.to_string().contains("Wonderland"), + "denial body must not leak the structured value: {body}" + ); +} + +#[tokio::test] +async fn resolve_denies_required_direct_array_claim() { + let server = structured_direct_server().await; + let response = server + .post(&resolve_path("structured-direct-required-array")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::FORBIDDEN); + let body: Value = response.json(); + assert_eq!(body["code"], "release.subject_denied"); + assert!( + !body.to_string().contains("ada@example.test"), + "denial body must not leak the structured value: {body}" + ); +} + +#[tokio::test] +async fn resolve_omits_optional_computed_claims_with_structured_values() { + // A computed claim returns whatever its CEL yields. A list or map result is + // not a scalar, so the claim is unavailable and the optional form is omitted. + let server = computed_shape_server().await; + let response = server + .post(&resolve_path("computed-shapes")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::OK); + let body: Value = response.json(); + let claims = body["claims"].as_object().expect("claims object"); + assert!( + !claims.contains_key("computed_list"), + "array-valued computed claim must be omitted: {body}" + ); + assert!( + !claims.contains_key("computed_map"), + "object-valued computed claim must be omitted: {body}" + ); +} + +#[tokio::test] +async fn resolve_denies_required_computed_structured_claim() { + let server = computed_shape_server().await; + let response = server + .post(&resolve_path("computed-required-list")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::FORBIDDEN); + assert_eq!(response.json::()["code"], "release.subject_denied"); +} + +#[tokio::test] +async fn resolve_computed_null_claim_is_omitted_not_released_as_null() { + // A computed claim that evaluates to JSON null behaves exactly like a direct + // claim over a null column: the claim is missing, never a literal `null`. + let server = computed_shape_server().await; + let response = server + .post(&resolve_path("computed-shapes")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::OK); + let body: Value = response.json(); + let claims = body["claims"].as_object().expect("claims object"); + assert!( + !claims.contains_key("computed_null"), + "null-valued computed claim must be omitted, not released as null: {body}" + ); + assert!( + !claims.values().any(Value::is_null), + "no released claim may be a literal null: {body}" + ); +} + +#[tokio::test] +async fn resolve_releases_every_scalar_claim_shape() { + // The scalar contract is a floor, not a narrowing: string, number, and + // boolean claim values all still release. + let server = computed_shape_server().await; + let response = server + .post(&resolve_path("computed-shapes")) + .json(&subject_body("NID-1")) + .await; + response.assert_status(StatusCode::OK); + let claims = response.json::()["claims"].clone(); + assert_eq!(claims["given_name"], "Ada"); + assert_eq!(claims["computed_text"], "Ada"); + assert_eq!(claims["computed_number"], 3); + assert_eq!(claims["computed_flag"], true); +} + +#[tokio::test] +async fn resolve_claim_selection_is_by_top_level_name_only() { + // Claim selection matches whole top-level claim names. A structured claim + // cannot be selected, and no path-style sub-selection into a claim exists. + let server = structured_direct_server().await; + let selected = server + .post(&resolve_path("structured-direct")) + .json(&json!({ + "subject": { "id_type": "NATIONAL_ID", "value": "NID-1" }, + "claims": ["given_name", "address"] + })) + .await; + selected.assert_status(StatusCode::OK); + let claims = selected.json::()["claims"].clone(); + assert_eq!(claims["given_name"], "Ada"); + assert!(claims.get("address").is_none()); + assert!(claims.get("contact_points").is_none()); + + // A dotted path into a claim is not a claim name: it is an unknown claim and + // is denied like any other. + let path_style = server + .post(&resolve_path("structured-direct")) + .json(&json!({ + "subject": { "id_type": "NATIONAL_ID", "value": "NID-1" }, + "claims": ["given_name", "address.region"] + })) + .await; + path_style.assert_status(StatusCode::FORBIDDEN); + assert_eq!(path_style.json::()["code"], "release.subject_denied"); +} + // --------------------------------------------------------------------------- // Scope / purpose deny-before-read // --------------------------------------------------------------------------- @@ -700,9 +1122,15 @@ async fn resolve_release_condition_cannot_read_redacted_field() { async fn resolve_purpose_bound_profile_accepts_matching_purpose() { // A purpose-bound profile (purpose set, entity NOT otherwise governing // purposes) resolves when the data-purpose header equals the profile purpose. - let server = try_server_full(&[RELEASE_SCOPE], "", true, Some("identity")) - .await - .expect("test server builds"); + let server = try_server_full( + &[RELEASE_SCOPE], + "", + true, + Some("identity"), + &ConfigExtras::default(), + ) + .await + .expect("test server builds"); let response = server .post(RESOLVE_PATH) .add_header("data-purpose", "identity") @@ -717,9 +1145,15 @@ async fn resolve_purpose_bound_profile_missing_header_is_purpose_required() { // Without a backing governed_policy the entity would not require purpose, but // the profile purpose binding does: a missing data-purpose header is rejected // before the read with 400 auth.purpose_required. - let server = try_server_full(&[RELEASE_SCOPE], "", true, Some("identity")) - .await - .expect("test server builds"); + let server = try_server_full( + &[RELEASE_SCOPE], + "", + true, + Some("identity"), + &ConfigExtras::default(), + ) + .await + .expect("test server builds"); let response = server.post(RESOLVE_PATH).json(&subject_body("NID-1")).await; response.assert_status(StatusCode::BAD_REQUEST); assert_eq!(response.json::()["code"], "auth.purpose_required"); @@ -729,9 +1163,15 @@ async fn resolve_purpose_bound_profile_missing_header_is_purpose_required() { async fn resolve_purpose_bound_profile_wrong_purpose_is_denied() { // A data-purpose that does not equal the profile purpose is denied before the // read with 403 auth.purpose_denied. - let server = try_server_full(&[RELEASE_SCOPE], "", true, Some("identity")) - .await - .expect("test server builds"); + let server = try_server_full( + &[RELEASE_SCOPE], + "", + true, + Some("identity"), + &ConfigExtras::default(), + ) + .await + .expect("test server builds"); let response = server .post(RESOLVE_PATH) .add_header("data-purpose", "marketing") @@ -766,6 +1206,7 @@ async fn resolve_missing_purpose_denies_before_read() { " require_purpose_header: true\n", true, Some("identity"), + &ConfigExtras::default(), ) .await .expect("test server builds"); @@ -784,7 +1225,7 @@ fn config_accepts_hyphenated_profile_id_and_dotted_claim_name() { "REGISTRY_RELAY_TEST_AUDIT_HASH_SECRET", "relay-release-audit-secret-32-bytes", ); - let yaml = release_config("", false, "identity") + let yaml = release_config("", false, "identity", &ConfigExtras::default()) .replace("id: civil_identity", "id: esignet-civil-userinfo") .replace("name: optional_note", "name: address.region"); let tmp = TempDir::new().expect("tempdir"); diff --git a/crates/registry-relay/tests/config_entities.rs b/crates/registry-relay/tests/config_entities.rs index e538e31f8..495da170e 100644 --- a/crates/registry-relay/tests/config_entities.rs +++ b/crates/registry-relay/tests/config_entities.rs @@ -781,6 +781,20 @@ fn release_profile_rejects_invalid_cel_expression() { assert_eq!(err, "config.validation_error"); } +#[cfg(feature = "attribute-release")] +#[test] +fn release_profile_rejects_statically_structured_cel_shape() { + // A top-level list literal always produces a non-scalar value, which the + // scalar-only claim contract can never release, so the profile fails at + // config load instead of resolving ambiguously in production. + let profile = valid_release_profile().replace( + " - name: municipality\n source_field: municipality_code\n", + " - name: municipality\n expression:\n cel: \"[source.municipality_code]\"\n", + ); + let err = load_release_dataset(&profile).expect_err("structured CEL shape rejected"); + assert_eq!(err, "config.validation_error"); +} + #[cfg(feature = "attribute-release")] #[test] fn release_profile_cel_may_reference_only_projected_source() { diff --git a/docs/site/src/content/docs/reference/apis/registry-relay.mdx b/docs/site/src/content/docs/reference/apis/registry-relay.mdx index bce032043..f02ac8dad 100644 --- a/docs/site/src/content/docs/reference/apis/registry-relay.mdx +++ b/docs/site/src/content/docs/reference/apis/registry-relay.mdx @@ -69,6 +69,12 @@ For setup steps, scope strings, and credential configuration, see Their source and all-feature tests remain available. Governed attribute release is stable and included in the canonical binary. Relay does not issue response credentials or host issuer DID documents. +- **Structured attribute-release claim values.** Released claim values are scalar-only in v1: a + string, a number, or a boolean. A claim whose projected or computed value is an object or an + array is treated as unavailable, an expression that always produces one is rejected at + configuration validation, and claim selection matches whole top-level claim names. Structured + and nested claim values are out of scope for 1.0; model them in the Registry Notary + consultation and credential surface, where their schema and limits are explicit. - **Link-free Records metadata.** Stable link-free OGC Records bodies under `/metadata/ogc/records` are portable metadata. They are separate from the experimental live adapter under `/ogc/v1/records`. From 79037b33a320c3f0a9b7d55351d76aa41b389885 Mon Sep 17 00:00:00 2001 From: Jeremi Joslin Date: Sat, 1 Aug 2026 16:53:17 +0700 Subject: [PATCH 2/2] feat(relay): surface untracked privacy budget on sensitive aggregates A dataset classified personal, confidential, or secret that declares a dataset-level access.aggregate_only_execution aggregate now raises the relay.aggregates.privacy_budget_untracked deployment finding, bound finding_warn under hosted_lab, production, and evidence_grade. Only dataset-level declarations count: the aggregate query routes resolve aggregates through dataset.aggregates alone, so table-level and entity-level declarations have no runtime route and raise no finding. Aggregate routes apply per-result minimum cell-size suppression but track no longitudinal query budget; this is the documented accepted limitation from the known-limitations page, so the gate warns everywhere rather than blocking startup or readiness. The docs present leaving the warning active as a legitimate steady state, with a deployment waiver as the optional dated acknowledgement, since waivers carry a mandatory expiry and the limitation is not fixable. Boot is loud: an active finding emits a dedicated deployment.privacy_budget_untracked warn line; a waived one reports through the existing deployment.gate_waived line. The finding reaches the posture endpoint and registryctl doctor through the existing catalog plumbing. Signed-off-by: Jeremi Joslin --- crates/registry-relay/CHANGELOG.md | 7 + crates/registry-relay/docs/configuration.md | 7 +- crates/registry-relay/src/api/admin.rs | 56 ++++ crates/registry-relay/src/config/validate.rs | 157 ++++++++++ crates/registry-relay/src/deployment/mod.rs | 294 +++++++++++++++++- .../docs/security/hardening-checklist.mdx | 9 + 6 files changed, 528 insertions(+), 2 deletions(-) diff --git a/crates/registry-relay/CHANGELOG.md b/crates/registry-relay/CHANGELOG.md index a1d30efa6..626476a46 100644 --- a/crates/registry-relay/CHANGELOG.md +++ b/crates/registry-relay/CHANGELOG.md @@ -15,6 +15,13 @@ credential surface, where their schema and limits are explicit. The OpenAPI contract documents the scalar-only claim bundle and top-level-only claim selection. +- A dataset classified `personal`, `confidential`, or `secret` with a + dataset-level `access.aggregate_only_execution` aggregate now raises the + `relay.aggregates.privacy_budget_untracked` deployment finding (warn at + every bound profile) and a dedicated boot-log warning, surfacing that + aggregate routes track no longitudinal privacy budget. The warning may be + left active as an accepted-limitation signal or acknowledged with a + deployment waiver naming the finding. ## 0.16.3 - 2026-08-01 diff --git a/crates/registry-relay/docs/configuration.md b/crates/registry-relay/docs/configuration.md index ac4588fdf..cf41fa9b7 100644 --- a/crates/registry-relay/docs/configuration.md +++ b/crates/registry-relay/docs/configuration.md @@ -1403,16 +1403,19 @@ Waiver references and summaries are visible only in the restricted posture tier; | `relay.audit.retention_local_only` | (not bound) | warn | startup_fail | | `relay.audit.shipping_unverified` | (not bound) | warn | startup_fail | | `relay.audit.shipping_stale` | (not bound) | error | readiness_fail | +| `relay.aggregates.privacy_budget_untracked` | warn | warn | warn | `relay.audit.retention_local_only` fires when the audit sink is a local rotating `file` sink and `evidence.audit_offhost_shipping` is not declared: a local rotating file caps retention, and an attacker with host access can destroy the audit trail. `stdout` sinks are exempt (retention is the orchestrator's log pipeline's concern) and `syslog` sinks are exempt (forwarding is the syslog daemon's own surface). `relay.audit.shipping_unverified` and `relay.audit.shipping_stale` read the ack cursor's observed health. `shipping_unverified` fires when any shipping target (`stdout`, `syslog`, or an attested local `file` sink) lacks `evidence.audit_ack_cursor_path`. It warns under `production` and refuses startup under `evidence_grade`, because a missing observation capability cannot heal at runtime. `shipping_stale` fires when a cursor is configured but is missing, unreadable, malformed, too old, too slow to read, or names a `last_acked_hash` other than the live keyed audit-chain tail. It fails readiness under `evidence_grade` and recovers when the trusted shipper advances a fresh cursor to the current tail. Neither hard gate is waivable. Runtime tail equality establishes that the claimed watermark belongs to this chain and the local backlog is zero; the unsigned local cursor is not cryptographic proof of remote receipt. Offline `doctor` cannot bind to a live chain and therefore reports a fresh cursor as `unverified`, never `ok`; an evidence-grade offline check consequently reports the hard shipping gate. The signed-bundle acceptance audit advances the tail before Relay serves requests, so the shipper must run independently of application readiness and acknowledge that boot record before `/ready` can return 200. Remediation: configure the cursor maintained by the off-host shipper, restore shipping, adjust `evidence.audit_ack_max_age_secs` if the cadence is legitimately slower, or repair a path or watermark mismatch. Removing the cursor does not satisfy `evidence_grade`. +`relay.aggregates.privacy_budget_untracked` fires when a dataset classified `personal`, `confidential`, or `secret` has a dataset-level aggregate with `access.aggregate_only_execution: true`. Only dataset-level declarations count: they are the only ones the aggregate query routes serve, so table-level and entity-level declarations raise no finding. Aggregate routes apply per-result minimum cell-size suppression (`disclosure_control.min_cell_size`), but track no longitudinal query budget: `query_budget.tracked` is always false. This is a documented, accepted limitation, not a fixable misconfiguration, so it warns at every bound profile, including `evidence_grade`, rather than blocking startup or readiness. Leaving the finding active is a legitimate steady state that keeps the posture honest; a deployment waiver naming the finding records a dated operator acknowledgement instead, at the cost of re-issuing the waiver whenever it expires. See "Aggregates are not privacy-budgeted" in [Known limitations and non-guarantees](https://docs.registrystack.org/explanation/known-limitations/). + The current deployment profile, its findings, and active waivers are reported under `deployment` in the operations posture (`GET /admin/v1/posture`). ### Boot-time visibility -Reduced posture is loud at boot, not only visible on the posture surface. Every config load warns once per waiver-suppressed finding (`deployment.gate_waived`, with the finding id, reference, optional summary, and expiry), once per expired waiver (`deployment.waiver_expired`), and once when the profile is undeclared (`deployment.profile_undeclared`). The serve path additionally writes one operational audit record per waived gate at boot, once the audit pipeline exists: event `deployment.gate_waived` at audit path `/__events/deployment.gate_waived`, with `error_code` set to the gate id. That minimized audit record does not copy waiver metadata. +Reduced posture is loud at boot, not only visible on the posture surface. Every config load warns once per waiver-suppressed finding (`deployment.gate_waived`, with the finding id, reference, optional summary, and expiry), once per expired waiver (`deployment.waiver_expired`), and once when the profile is undeclared (`deployment.profile_undeclared`). A sensitive dataset's aggregate-only-execution aggregate additionally warns once per config load with `deployment.privacy_budget_untracked`, naming the finding id; a waived occurrence is reported through the generic `deployment.gate_waived` line instead, not a duplicate line. The serve path additionally writes one operational audit record per waived gate at boot, once the audit pipeline exists: event `deployment.gate_waived` at audit path `/__events/deployment.gate_waived`, with `error_code` set to the gate id. That minimized audit record does not copy waiver metadata. This boot-time audit write inherits `audit.write_policy` (see below). Under `fail_closed` (the default), a failed write aborts startup. Under `availability_first`, the failure is logged (`audit.operational_event_write_failed`) and startup continues, so the durable record is best-effort; the per-gate boot log warnings above remain the guaranteed floor. @@ -1730,6 +1733,8 @@ aggregates: Supported aggregate functions include the configured V1 set used by tests and examples, such as `count`, `sum`, and `avg`. The runtime config key remains `indicators` for compatibility; public aggregate APIs expose these configured series as measures. `temporal_field` is optional; when present, native aggregate `temporal.from` and `temporal.to` are translated into the declared range-capable allowed filter for that source-entity field. Dataset measure and dimension discovery is derived from these aggregate declarations, so keep ids stable and labels consumer-friendly. Keep disclosure thresholds explicit and reviewable. +`disclosure_control` and `access.aggregate_only_execution` bound per-result minimum cell-size suppression on a single query; they do not track a longitudinal query budget across repeated aggregate queries. On a `personal`, `confidential`, or `secret` dataset, a dataset-level `aggregate_only_execution` aggregate raises the `relay.aggregates.privacy_budget_untracked` deployment finding described in the findings catalog under "Deployment profile" above; see "Aggregates are not privacy-budgeted" in [Known limitations and non-guarantees](https://docs.registrystack.org/explanation/known-limitations/) for the underlying limitation. + ### Spatial EDR aggregates Spatial EDR exposure is opt-in. Requires `--features ogcapi-edr`. diff --git a/crates/registry-relay/src/api/admin.rs b/crates/registry-relay/src/api/admin.rs index dc2a2156e..398dcc4d7 100644 --- a/crates/registry-relay/src/api/admin.rs +++ b/crates/registry-relay/src/api/admin.rs @@ -1153,6 +1153,62 @@ datasets: [] assert!(ids.contains(&"relay.ingress.rate_limit_missing")); } + /// A sensitive dataset with an aggregate-only-execution aggregate surfaces + /// `relay.aggregates.privacy_budget_untracked` on the posture surface at + /// `finding_warn`, the same generic path every other catalog gate takes. + #[test] + fn deployment_summary_reports_untracked_privacy_budget_finding() { + let yaml = r#" +server: + bind: "127.0.0.1:8080" +catalog: + title: "Test Registry" + base_url: "https://data.example.test" + publisher: "Test Ministry" +auth: + mode: api_key + api_keys: [] +audit: + sink: stdout +datasets: + - id: sensitive_ds + title: "Sensitive Dataset" + description: "desc" + owner: "owner" + sensitivity: personal + access_rights: restricted + update_frequency: daily + tables: + - id: t1 + source: + type: file + path: "data/t1.csv" + refresh: + mode: manual + schema: + fields: [] + aggregates: + - id: agg1 + description: "test aggregate" + source_entity: record + disclosure_control: + min_group_size: 2 + access: + aggregate_only_execution: true +"#; + let mut config = parse_minimal_config(yaml); + config.deployment.profile = Some(DeploymentProfile::HostedLab); + let summary = deployment_summary(&config, ConfigSource::LocalFile); + let finding = summary["findings"] + .as_array() + .expect("findings array") + .iter() + .find(|finding| finding["id"] == "relay.aggregates.privacy_budget_untracked") + .expect("privacy-budget finding is reported"); + assert_eq!(finding["severity"], "finding_warn"); + assert_eq!(finding["status"], "active"); + } + /// The full posture document is schema-valid for every declared profile and /// for the undeclared default. `evidence_grade` from a local file would /// trip a startup gate at load time, so its posture is exercised with a diff --git a/crates/registry-relay/src/config/validate.rs b/crates/registry-relay/src/config/validate.rs index 255f50e93..4c1b0976b 100644 --- a/crates/registry-relay/src/config/validate.rs +++ b/crates/registry-relay/src/config/validate.rs @@ -584,6 +584,13 @@ fn log_deployment_boot_findings(evaluation: &crate::deployment::GateEvaluation) code = "deployment.profile_undeclared", "deployment profile is undeclared; no profile gates bind" ); + } else if finding.id == "relay.aggregates.privacy_budget_untracked" { + tracing::warn!( + code = "deployment.privacy_budget_untracked", + finding = %finding.id, + action = "leave the warning active, or record a dated acknowledgement with a deployment waiver naming this finding; see explanation/known-limitations, \"Aggregates are not privacy-budgeted\"", + "aggregate routes on a sensitive dataset apply per-result minimum cell-size suppression but track no longitudinal privacy budget" + ); } } } @@ -4739,6 +4746,96 @@ datasets: [] serde_saphyr::from_str(&deployment_config_yaml(extra)).expect("config parses") } + /// A dataset with `sensitivity` classification carrying one dataset-level + /// aggregate with `access.aggregate_only_execution: true` (the only + /// aggregate level the query routes serve), plus `extra` (typically a + /// `deployment:` stanza) appended at the top level. + fn sensitive_aggregate_dataset_config_yaml(sensitivity: &str, extra: &str) -> String { + format!( + r#" +server: + bind: "127.0.0.1:8080" +catalog: + title: "Test Registry" + base_url: "https://data.example.test" + publisher: "Test Ministry" +auth: + mode: api_key + api_keys: [] +audit: + sink: stdout +datasets: + - id: sensitive_ds + title: "Sensitive Dataset" + description: "desc" + owner: "owner" + sensitivity: {sensitivity} + access_rights: restricted + update_frequency: daily + tables: + - id: t1 + source: + type: file + path: "data/t1.csv" + refresh: + mode: manual + primary_key: record_id + schema: + strict: true + fields: + - name: record_id + type: string + nullable: false + - name: region_code + type: string + nullable: true + aggregates: + - id: agg1 + title: "Records by region" + description: "test aggregate" + source_entity: record + default_group_by: + - region_code + dimensions: + - id: region_code + label: Region + field: region_code + indicators: + - id: record_count + label: Records + function: count + column: id + unit_measure: records + disclosure_control: + min_group_size: 2 + suppression: omit + access: + aggregate_only_execution: true + entities: + - name: record + table: t1 + fields: + - name: id + from: record_id + - name: region_code + access: + metadata_scope: "sensitive_ds:metadata" + aggregate_scope: "sensitive_ds:aggregate" + read_scope: "sensitive_ds:rows" + api: + default_limit: 10 + max_limit: 100 + require_purpose_header: true +{extra} +"# + ) + } + + fn parse_sensitive_aggregate_dataset_config(sensitivity: &str, extra: &str) -> Config { + serde_saphyr::from_str(&sensitive_aggregate_dataset_config_yaml(sensitivity, extra)) + .expect("config parses") + } + fn consultation_deployment_config_yaml(extra: &str) -> String { format!( r#" @@ -5347,6 +5444,66 @@ deployment: ); } + #[test] + fn sensitive_aggregate_only_execution_is_loud_in_the_boot_log() { + let config = parse_sensitive_aggregate_dataset_config( + "personal", + "deployment:\n profile: production", + ); + let (result, rendered) = run_with_captured_logs(&config); + result.expect( + "an aggregate-only-execution finding on a sensitive dataset must not block startup", + ); + assert!( + rendered.contains("deployment.privacy_budget_untracked"), + "expected deployment.privacy_budget_untracked in boot log: {rendered}" + ); + assert!( + rendered.contains("relay.aggregates.privacy_budget_untracked"), + "expected the finding id in boot log: {rendered}" + ); + assert!( + rendered.contains("known-limitations"), + "expected a known-limitations reference in boot log: {rendered}" + ); + } + + #[test] + fn public_sensitivity_keeps_the_privacy_budget_boot_log_quiet() { + let config = parse_sensitive_aggregate_dataset_config( + "public", + "deployment:\n profile: production", + ); + let (result, rendered) = run_with_captured_logs(&config); + result.expect("a public-sensitivity dataset must not block startup"); + assert!( + !rendered.contains("deployment.privacy_budget_untracked"), + "no privacy-budget line expected for a public-sensitivity dataset: {rendered}" + ); + } + + #[test] + fn waived_privacy_budget_finding_shows_the_generic_gate_waived_line_only() { + let config = parse_sensitive_aggregate_dataset_config( + "personal", + "deployment:\n profile: production\n waivers:\n - finding: relay.aggregates.privacy_budget_untracked\n reference: OPS-TEST-PRIVACY-BUDGET\n expires: \"2999-01-01\"", + ); + let (result, rendered) = run_with_captured_logs(&config); + result.expect("a waived privacy-budget finding must not block startup"); + assert!( + rendered.contains("deployment.gate_waived"), + "expected deployment.gate_waived in boot log: {rendered}" + ); + assert!( + rendered.contains("relay.aggregates.privacy_budget_untracked"), + "expected the waived finding id in boot log: {rendered}" + ); + assert!( + !rendered.contains("deployment.privacy_budget_untracked"), + "the dedicated privacy-budget line must not appear once waived: {rendered}" + ); + } + #[test] fn evidence_grade_via_signed_bundle_source_boots() { // The same evidence_grade config that a local file rejects must validate diff --git a/crates/registry-relay/src/deployment/mod.rs b/crates/registry-relay/src/deployment/mod.rs index cc69faa40..87d1c7a54 100644 --- a/crates/registry-relay/src/deployment/mod.rs +++ b/crates/registry-relay/src/deployment/mod.rs @@ -35,7 +35,9 @@ use registry_platform_ops::{ DeploymentWaiver, GateSeverity, DEFAULT_AUDIT_ACK_MAX_AGE, }; -use crate::config::{AuditSinkConfig, AuthMode, Config}; +use crate::config::{ + AggregateConfig, AuditSinkConfig, AuthMode, Config, DatasetConfig, Sensitivity, +}; const AUDIT_ACK_CURSOR_READ_TIMEOUT: Duration = Duration::from_millis(500); static AUDIT_ACK_CURSOR_READ_PERMIT: OnceLock> = OnceLock::new(); @@ -103,6 +105,18 @@ pub struct DeploymentFacts { /// The cursor is fresh and its watermark equals the current keyed chain /// tail (health `ok`). False for every other observation. pub audit_ack_health_ok: bool, + /// A dataset classified `personal`, `confidential`, or `secret` has a + /// dataset-level aggregate configured with + /// `access.aggregate_only_execution: true`. Dataset-level declarations + /// are the only ones the aggregate query routes serve; table-level and + /// entity-level declarations have no runtime route and do not raise the + /// finding. Aggregate routes apply per-result minimum cell-size + /// suppression (`disclosure_control.min_cell_size`) but track no + /// longitudinal query budget: `query_budget.tracked` is always false. + /// This is a documented, + /// accepted limitation, not a fixable misconfiguration; see "Aggregates + /// are not privacy-budgeted" in the known-limitations page. + pub sensitive_aggregate_only_execution: bool, } /// One gate in the relay catalog. @@ -243,6 +257,21 @@ const GATES: &[Gate] = &[ production: Some(FindingError), evidence_grade: Some(ReadinessFail), }, + Gate { + id: "relay.aggregates.privacy_budget_untracked", + condition: |facts| facts.sensitive_aggregate_only_execution, + // Relay aggregate routes apply per-result minimum cell-size + // suppression but track no longitudinal query budget across repeated aggregate + // queries. This is a documented, accepted limitation ("Aggregates are + // not privacy-budgeted" in the known-limitations page), not a fixable + // misconfiguration, so it warns at every bound profile rather than + // blocking startup or readiness even at evidence_grade. A deployment + // waiver naming this finding records the operator's acknowledgement + // of the limitation. + hosted_lab: Some(FindingWarn), + production: Some(FindingWarn), + evidence_grade: Some(FindingWarn), + }, ]; /// Outcome of evaluating the catalog against one profile. @@ -535,9 +564,36 @@ pub fn facts_from_config_with_ack_observation( ) || config.deployment.evidence.audit_offhost_shipping, audit_ack_cursor_configured: config.deployment.evidence.audit_ack_cursor_path.is_some(), audit_ack_health_ok: observation.health == AckHealth::Ok, + sensitive_aggregate_only_execution: sensitive_aggregate_only_execution(config), } } +/// A dataset classified `personal`, `confidential`, or `secret` has a +/// dataset-level aggregate-only-execution aggregate. Only dataset-level +/// declarations are considered: the aggregate query routes resolve aggregates +/// through `AggregateQueryEngine::aggregate_config`, which reads +/// `dataset.aggregates` alone, so table-level and entity-level declarations +/// have no runtime route and expose no privacy-budget surface. +fn sensitive_aggregate_only_execution(config: &Config) -> bool { + config.datasets.iter().any(|dataset| { + matches!( + dataset.sensitivity, + Sensitivity::Personal | Sensitivity::Confidential | Sensitivity::Secret + ) && dataset_has_aggregate_only_execution(dataset) + }) +} + +fn dataset_has_aggregate_only_execution(dataset: &DatasetConfig) -> bool { + dataset.aggregates.iter().any(aggregate_only_execution) +} + +fn aggregate_only_execution(aggregate: &AggregateConfig) -> bool { + aggregate + .access + .as_ref() + .is_some_and(|access| access.aggregate_only_execution) +} + /// Read the operator's audit off-host ack cursor and evaluate its freshness. /// /// The cursor path and freshness window are the operator declarations in @@ -696,6 +752,7 @@ mod tests { audit_shipping_target_configured: false, audit_ack_cursor_configured: false, audit_ack_health_ok: true, + sensitive_aggregate_only_execution: false, } } @@ -786,6 +843,7 @@ mod tests { audit_shipping_target_configured: true, audit_ack_cursor_configured: true, audit_ack_health_ok: false, + sensitive_aggregate_only_execution: true, }; let evaluation = evaluate(None, &facts, &[], TODAY); assert_eq!(finding_ids(&evaluation), vec![PROFILE_UNDECLARED]); @@ -1443,6 +1501,240 @@ datasets: [] assert!(!facts_from_config(&config, ConfigSource::SignedBundleEndpoint).config_unsigned); } + /// Builds a config with one `datasets:` entry parsed from `dataset_yaml` + /// (a single already-indented list item), reusing `minimal_config`'s + /// server/catalog/auth/audit scaffolding. + fn config_with_dataset(dataset_yaml: &str) -> Config { + let yaml = format!( + r#" +server: + bind: "127.0.0.1:8080" +catalog: + title: "Test Registry" + base_url: "https://data.example.test" + publisher: "Test Ministry" +auth: + mode: api_key + api_keys: [] +audit: + sink: stdout +datasets: +{dataset_yaml} +"# + ); + serde_saphyr::from_str(&yaml).expect("config parses") + } + + fn dataset_with_dataset_level_aggregate( + sensitivity: &str, + aggregate_only_execution: bool, + ) -> String { + format!( + r#" - id: ds1 + title: "DS" + description: "desc" + owner: "owner" + sensitivity: {sensitivity} + access_rights: restricted + update_frequency: daily + aggregates: + - id: agg1 + description: "test aggregate" + disclosure_control: {{}} + access: + aggregate_only_execution: {aggregate_only_execution} +"# + ) + } + + #[test] + fn sensitive_aggregate_only_execution_fires_for_personal_confidential_secret() { + for sensitivity in ["personal", "confidential", "secret"] { + let config = + config_with_dataset(&dataset_with_dataset_level_aggregate(sensitivity, true)); + assert!( + facts_from_config(&config, ConfigSource::LocalFile) + .sensitive_aggregate_only_execution, + "expected the fact to fire for sensitivity {sensitivity}" + ); + } + } + + #[test] + fn sensitive_aggregate_only_execution_does_not_fire_for_public_or_internal() { + for sensitivity in ["public", "internal"] { + let config = + config_with_dataset(&dataset_with_dataset_level_aggregate(sensitivity, true)); + assert!( + !facts_from_config(&config, ConfigSource::LocalFile) + .sensitive_aggregate_only_execution, + "did not expect the fact to fire for sensitivity {sensitivity}" + ); + } + } + + #[test] + fn sensitive_aggregate_only_execution_requires_the_access_flag() { + let config = config_with_dataset(&dataset_with_dataset_level_aggregate("personal", false)); + assert!( + !facts_from_config(&config, ConfigSource::LocalFile).sensitive_aggregate_only_execution + ); + } + + #[test] + fn sensitive_aggregate_only_execution_false_when_dataset_has_no_aggregates() { + let config = config_with_dataset( + r#" - id: ds1 + title: "DS" + description: "desc" + owner: "owner" + sensitivity: personal + access_rights: restricted + update_frequency: daily +"#, + ); + assert!( + !facts_from_config(&config, ConfigSource::LocalFile).sensitive_aggregate_only_execution + ); + } + + /// Table-level aggregate declarations parse and validate, but the + /// aggregate query routes resolve only `dataset.aggregates` + /// (`AggregateQueryEngine::aggregate_config`), so an unqueryable + /// declaration must not prompt an operator to waive a finding about a + /// route that does not exist. + #[test] + fn sensitive_aggregate_only_execution_ignores_table_level_aggregate() { + let config = config_with_dataset( + r#" - id: ds1 + title: "DS" + description: "desc" + owner: "owner" + sensitivity: confidential + access_rights: restricted + update_frequency: daily + tables: + - id: t1 + source: + type: file + path: "data/t1.csv" + schema: + fields: [] + aggregates: + - id: agg1 + description: "test aggregate" + disclosure_control: {} + access: + aggregate_only_execution: true +"#, + ); + assert!( + !facts_from_config(&config, ConfigSource::LocalFile).sensitive_aggregate_only_execution + ); + } + + /// Entity-level aggregate declarations have no runtime query route either; + /// see the table-level test above. + #[test] + fn sensitive_aggregate_only_execution_ignores_entity_level_aggregate() { + let config = config_with_dataset( + r#" - id: ds1 + title: "DS" + description: "desc" + owner: "owner" + sensitivity: secret + access_rights: restricted + update_frequency: daily + entities: + - name: e1 + table: t1 + access: + metadata_scope: "ds1:metadata" + aggregate_scope: "ds1:aggregate" + read_scope: "ds1:rows" + api: + default_limit: 10 + max_limit: 100 + aggregates: + - id: agg1 + description: "test aggregate" + disclosure_control: {} + access: + aggregate_only_execution: true +"#, + ); + assert!( + !facts_from_config(&config, ConfigSource::LocalFile).sensitive_aggregate_only_execution + ); + } + + #[test] + fn sensitive_aggregate_only_execution_escalates_as_finding_warn_everywhere() { + let facts = DeploymentFacts { + sensitive_aggregate_only_execution: true, + ..clean_facts() + }; + let id = "relay.aggregates.privacy_budget_untracked"; + for profile in [ + DeploymentProfile::HostedLab, + DeploymentProfile::Production, + DeploymentProfile::EvidenceGrade, + ] { + let evaluation = evaluate(Some(profile), &facts, &[], TODAY); + assert_eq!( + finding(&evaluation, id).severity, + FindingWarn, + "unexpected severity under {profile:?}" + ); + } + } + + #[test] + fn sensitive_aggregate_only_execution_silent_when_clean() { + let id = "relay.aggregates.privacy_budget_untracked"; + for profile in [ + DeploymentProfile::HostedLab, + DeploymentProfile::Production, + DeploymentProfile::EvidenceGrade, + ] { + let evaluation = evaluate(Some(profile), &clean_facts(), &[], TODAY); + assert!( + !finding_ids(&evaluation).contains(&id.to_string()), + "unexpected finding under {profile:?}" + ); + } + } + + #[test] + fn sensitive_aggregate_only_execution_unbound_under_local() { + let facts = DeploymentFacts { + sensitive_aggregate_only_execution: true, + ..clean_facts() + }; + let evaluation = evaluate(Some(DeploymentProfile::Local), &facts, &[], TODAY); + assert!(evaluation.findings.is_empty()); + } + + #[test] + fn sensitive_aggregate_only_execution_waiver_suppresses_finding() { + let facts = DeploymentFacts { + sensitive_aggregate_only_execution: true, + ..clean_facts() + }; + let id = "relay.aggregates.privacy_budget_untracked"; + let waivers = [WaiverInput { + finding: id.to_string(), + reference: "OPS-TEST-PRIVACY-BUDGET".to_string(), + summary: Some("Accepted risk: no longitudinal privacy budget".to_string()), + expires: FUTURE.to_string(), + }]; + let evaluation = evaluate(Some(DeploymentProfile::Production), &facts, &waivers, TODAY); + assert_eq!( + finding(&evaluation, id).status, + DeploymentFindingStatus::Waived + ); + } + #[test] fn gate_severity_for_profile_resolves_binding() { // The retention gate warns under production and is a startup failure diff --git a/docs/site/src/content/docs/security/hardening-checklist.mdx b/docs/site/src/content/docs/security/hardening-checklist.mdx index ee9cb974a..9234d42ee 100644 --- a/docs/site/src/content/docs/security/hardening-checklist.mdx +++ b/docs/site/src/content/docs/security/hardening-checklist.mdx @@ -150,6 +150,15 @@ or the [Registry Notary operator configuration reference](../../products/registr summary only when operators need more context, and keep credentials and private keys out of both fields. `startup_fail` and `readiness_fail` gates are never waivable; a waiver naming one is rejected. Waiver references and summaries appear only in the restricted posture tier. +- Expect `relay.aggregates.privacy_budget_untracked` to warn under `hosted_lab`, `production`, + and `evidence_grade` (the `local` profile binds no gates) when a `personal`, `confidential`, + or `secret` dataset carries a dataset-level aggregate configured with + `access.aggregate_only_execution: true`. Aggregate routes apply per-result minimum cell-size + suppression but track no longitudinal query budget; this is a documented, accepted limitation, + not a misconfiguration to fix. Leaving the warning active is a legitimate steady state and + keeps the posture honest. Record a deployment waiver naming the finding only if your review + process needs a dated acknowledgement, and expect to re-issue it: every waiver carries a + mandatory expiry. See [Known limitations and non-guarantees](../../explanation/known-limitations/). - Use `deployment.evidence.*` flags (for example `ingress_rate_limit`, `api_key_rotation`) only to assert controls that live outside the process and cannot be observed by it. Each flag defaults to `false`.