Contrib data handling

mpcontribs-api/README.md

@@ -1,3 +1,83 @@
 - https://contribs-api.materialsproject.org
 - https://ml-api.materialsproject.org
 - https://lightsources-api.materialsproject.org
+
+## Contribution `data`
+
+A contribution's `data` is a hierarchical JSON object (max nesting depth 7). Lists are allowed, and
+dictionaries inside them have their keys coerced and validated like any other key. The API applies
+two write-time transforms that are important for clients to understand, because **the data you read
+back can differ from the data you submitted**.
+
+### Key coercion to `snake_case`
+
+Every dictionary key in `data` is coerced to `snake_case` on write:
+
+- casing is lowercased,
+- `camelCase` / `PascalCase` boundaries are split (`bandGap` → `band_gap`),
+- any run of spaces, hyphens, or punctuation collapses to a single underscore.
+
+So `"bandGap"`, `"Band Gap"`, and `"band-gap"` are all stored as `"band_gap"`. Keys must be ASCII
+and must not reduce to an empty string (e.g. `"***"` is rejected). Colliding keys after coercion
+(`"Band Gap"` and `"band_gap"` in the same object) are rejected.
+
+### Annotated keys: units and conditions
+
+A **top-level** key may carry an annotation in parentheses:
+
+```
+"name (unit, condition1=value1, condition2=value2, ...)"
+```
+
+- The single token without an `=` is the **unit** (`eV`, `S/cm`, `K`, …). Units are stored verbatim
+  (not coerced) so they round-trip through [Pint](https://pint.readthedocs.io/).
+- Each `k=v` token is a **condition**; condition names are coerced to `snake_case` like other keys.
+- `name` may be a dotted path (`"transport.conductivity (S/cm)"`) to nest the value.
+- A key with no parentheses is a plain, fully backward-compatible key.
+
+Recognized units are canonicalized to SI base units on write. Magnitudes may carry uncertainty as
+`"4.2(3)"`, `"4.2+/-0.3"`, or `"4.2±0.3"`.
+
+A value submitted with a unit annotation is stored as an **annotated leaf**:
+
+| field                       | meaning                                                                      |
+| --------------------------- | ---------------------------------------------------------------------------- |
+| `value` / `unit`            | SI-canonical form (or submitted form if the unit is unrecognized/dimensionless) |
+| `input_value` / `input_unit`| the magnitude and unit exactly as submitted                                  |
+| `error`                     | SI-propagated standard deviation (only when an uncertainty was given)        |
+| `display`                   | human-readable rendering of the submitted magnitude/unit, e.g. `"4.2+/-0.3 eV"` |
+
+### Pivoting on conditions
+
+If any key carries conditions, a single submitted contribution is **expanded into one contribution
+per distinct condition signature**. Each resulting contribution stores its conditions as ordinary
+columns, plus its own measurements, plus every condition-less ("broadcast") column, and is stamped
+with a server-computed `condition_key` identity. For example:
+
+```jsonc
+// submitted
+{
+  "conductivity (S/cm, T=300K)": 1.2,
+  "conductivity (S/cm, T=400K)": 3.4,
+  "sample": "A"          // broadcast to every pivoted row
+}
+// stored: two contributions, one per temperature, each with `t`, `conductivity`, and `sample`
+```
+
+Submissions that pivot on conditions may **not** include components (structures/tables/attachments):
+insert the pivoted contributions first, then attach components to the rows you want.
+
+**PATCH semantics.** `PATCH /contributions/{id}` runs the same expansion on its `data`:
+
+- Condition-less `data` patches the addressed contribution in place (units are still annotated); its
+  `condition_key` is unchanged.
+- `data` that carries conditions *fans out*: each condition signature updates the existing pivoted
+  row (under the same `project`/`identifier`/`version`) that already has the matching `condition_key`.
+  A PATCH never creates new rows or rewrites a `condition_key`, so a signature that matches no stored
+  row is rejected. Non-`data` fields on the patch apply to every row it touches. The response is the
+  list of updated contributions.
+
+> The write path (key grammar, coercion, and validation) lives in
+> `src/mpcontribs_api/domains/_shared/types.py`; expansion/pivoting lives in
+> `src/mpcontribs_api/domains/contributions/pivot.py`; unit handling lives in
+> `src/mpcontribs_api/domains/_shared/units.py`.

mpcontribs-api/main.py

@@ -33,8 +33,8 @@ def start(program):
     cluster = labels[f"{prefix}.cluster"].split("/", 1)[1]
     family = labels[f"{prefix}.task-definition-family"]
     version = int(labels[f"{prefix}.task-definition-version"])
-    task_arns = client.list_tasks(cluster=cluster, family=family).get("taskArns", [])
-    tasks = client.describe_tasks(cluster=cluster, tasks=task_arns).get("tasks", [])
+    task_arns = client.list_tasks(cluster=cluster, family=family).get("taskArns", [])  # pyright: ignore[reportAttributeAccessIssue] - boto3 dynamically dispatches, this is valid with ECS
+    tasks = client.describe_tasks(cluster=cluster, tasks=task_arns).get("tasks", [])  # pyright: ignore[reportAttributeAccessIssue] - boto3 dynamically dispatches, this is valid with ECS
     ntasks = 0
 
     for task in tasks:

mpcontribs-api/pyproject.toml

@@ -50,6 +50,8 @@ dependencies = [
     "structlog>=25.5.0",
     "supervisor>=4.3.0",
     "types-aiobotocore[s3]>=3.7.0",
+    "pint>=0.25.3",
+    "uncertainties>=3.2.3",
 ]
 
 [project.urls]
@@ -65,6 +67,7 @@ dev = [
     "pytest-xdist>=3.8.0",
     "httpx2>=0.28.0",
     "pytest-asyncio>=1.4.0",
+    "hypothesis>=6.100.0",
 ]
 
 [tool.pytest.ini_options]

mpcontribs-api/src/mpcontribs_api/_openapi.py

@@ -1,3 +1,84 @@
+# Client-facing documentation for the Contribution ``data`` field. Referenced as the OpenAPI schema
+# description on the input/output models (see ``contributions.models``) so it renders in the docs UI
+# and generated clients. Keep these in sync with the behavior in ``contributions.pivot`` and
+# ``_shared.units``.
+
+CONTRIBUTION_DATA_INPUT_DESCRIPTION = """\
+Hierarchical, JSON-object data for the contribution. Nesting deeper than 7 levels is rejected. Lists
+are allowed; any dictionaries inside them have their keys coerced and validated like every other key.
+
+**Key coercion (important):** every dictionary key is coerced to `snake_case` on write. Casing is
+lowercased, `camelCase`/`PascalCase` boundaries are split, and any run of spaces/hyphens/punctuation
+collapses to a single underscore. So `"bandGap"`, `"Band Gap"`, and `"band-gap"` are all stored as
+`"band_gap"`, and the **keys you read back may differ from the keys you submitted**. Keys must be
+ASCII and must not reduce to an empty string (e.g. `"***"` is rejected).
+
+**Reserved keys:** `value`, `unit`, `input_value`, `input_unit`, `error`, `input_error`,
+`precision`, and `display` are reserved for the stored value-leaf shape and may **not** be used as
+data keys (a key that coerces to any of these is rejected).
+
+**Values become structured leaves.** Any value that reads as a number is stored as a *quantity leaf*
+(see the response schema) capturing its value, unit, uncertainty, and precision; anything that does
+not (a word like `"cubic"`, a boolean, a list) is stored as-is. Scalars **inside lists** are left
+as-is (a list is treated as array data, not a column of measurements).
+
+**Two ways to give a unit (both accepted, they converge):**
+
+1. In the **key**, via an annotation in parentheses:
+   `"name (unit, condition1=value1, condition2=value2, ...)"` — with the value a bare number
+   (`{"bandgap (eV)": 5}`).
+2. In the **value** string itself (`{"bandgap": "5 eV"}`).
+
+Annotation rules: the single token without an `=` is the **unit** (e.g. `eV`, `S/cm`, `K`), left
+verbatim so it round-trips through Pint; each `k=v` token is a **condition** (names coerced to
+`snake_case`); `name` may be a dotted path (`"transport.conductivity (S/cm)"`) to nest the value; a
+key with no parentheses is a plain key. If a unit is given in **both** the key and the value and they
+differ, the **key's unit wins** — the value is converted into it (a dimensional mismatch is rejected).
+
+Recognized units are canonicalized to SI base units on write (the submitted form is preserved too —
+see the response schema); an unrecognized unit is kept verbatim. Magnitudes may carry uncertainty as
+`"4.2(3)"`, `"4.2+/-0.3"`, or `"4.2±0.3"`. Precision (significant figures) is captured from a string
+magnitude such as `"1.000"`; a bare JSON number carries no trailing-zero information.
+
+The server does **not** produce a formatted `display` string — the response returns the structured
+fields (`input_value`/`input_unit`/`input_error`/`precision`) so clients render values however they
+prefer. (`display` remains a reserved key for backward compatibility with older stored leaves.)
+
+**Pivoting on conditions:** if any key carries conditions, the single submission is *expanded* into
+one contribution per distinct condition signature. Each resulting contribution stores its conditions
+as ordinary columns plus its own measurements plus every condition-less ("broadcast") column, and
+gets a server-computed `condition_key` identity. Submissions that pivot on conditions may **not**
+include components (structures/tables/attachments) — insert the pivoted contributions first, then
+attach components.
+
+**On PATCH:** the same expansion runs. Condition-less data patches the addressed contribution in
+place (units still annotated). Data carrying conditions *fans out*: each signature updates the
+existing pivoted row that already has the matching `condition_key` (a PATCH never creates rows or
+changes a row's `condition_key`), so a signature with no matching stored row is rejected.
+"""
+
+CONTRIBUTION_DATA_OUTPUT_DESCRIPTION = """\
+Hierarchical contribution data. Keys are stored in `snake_case` (see the write schema for the
+coercion rules), so they may differ from the keys originally submitted.
+
+Any value that reads as a number is stored as a **quantity leaf** object:
+
+- `value` / `unit`: the SI-canonical magnitude and unit (or the submitted form when the unit is
+  unrecognized or dimensionless)
+- `input_value` / `input_unit`: the magnitude and unit as submitted (omitted for a bare, unitless,
+  exact number that ``value`` already fully describes)
+- `error`: the (SI-propagated) standard deviation — present only when the magnitude carried an
+  uncertainty; `input_error` is the same in the submitted unit
+- `precision`: the number of significant figures the submission carried — present only when it is
+  derivable (a string magnitude such as `"1.000"`)
+
+There is no server-rendered `display` string: `input_value`, `input_unit`, `input_error`, and
+`precision` reproduce the submitted form exactly, so a client can format the value however it likes.
+
+Values that are not numeric (words, booleans, lists) keep whatever JSON shape they were submitted
+with (after `snake_case` key coercion).
+"""
+
 openapi_tags = [
     {
         "name": "projects",
@@ -15,7 +96,10 @@
         "Each contribution uses `mp-id` or composition as identifier to associate its data with the according entries "
         "on MP. Only admins or users on the project can create, update or delete contributions, and while unpublished, "
         "retrieve its data or view it on the Portal. Contribution components (tables, structures, and attachments) are "
-        "deleted along with a contribution.",
+        "deleted along with a contribution. **Note:** `data` keys are coerced to `snake_case` on write and may carry "
+        "unit/condition annotations of the form `name (unit, cond=value, ...)`; keys with conditions cause the "
+        "submission to pivot into one contribution per condition signature. See the `data` field on the request and "
+        "response schemas for the full grammar and the annotated-value shape.",
     },
     {
         "name": "structures",

mpcontribs-api/src/mpcontribs_api/app.py

@@ -68,6 +68,7 @@ async def _setup_mongo(app: FastAPI, settings: Settings, stack: AsyncExitStack)
             Structure,
             Table,
         ],
+        allow_index_dropping=True,  # allow old indices to be dropped from MongoDB when not specified in this package
     )
 
 
@@ -132,8 +133,6 @@ def create_app(settings: Settings | None = None) -> FastAPI:
     )
 
     # Reject oversized request bodies before they're buffered into memory. Added before
-    # RequestContextMiddleware: Starlette inserts each added middleware at the front of the stack,
-    # so the later-added RequestContextMiddleware stays outermost and still access-logs rejections.
     app.add_middleware(BodySizeLimitMiddleware, max_bytes=settings.mongo.max_request_bytes)
     # Add request context to the logger
     app.add_middleware(RequestContextMiddleware)

mpcontribs-api/src/mpcontribs_api/config.py

@@ -157,6 +157,27 @@ def _clamp_concurrency(self):
         return self
 
 
+class MPContribsSettings(BaseModel):
+    max_contrib_data_depth: int = Field(
+        default=7, description="The max number of levels allowed in a Contribution's data dictionary."
+    )
+    max_columns: int = Field(
+        default=160,
+        description="The maximum allowed number of columns for a contribution (len(Contribution.data)), "
+        "which also gets reflected in Project.columns",
+    )
+    max_components: int = Field(
+        default=10,
+        description="The maximum allowed number of a single Component type (Structure, Table, Attachment) on a single "
+        "Contribution",
+    )
+    float_precision: int = Field(
+        default=6,
+        description="The precision with which to store floats in MongoDB. "
+        "Primarily used to handle Contribution.data values.",
+    )
+
+
 class Settings(BaseSettings):
     model_config = SettingsConfigDict(
         env_file=".env",
@@ -170,6 +191,9 @@ class Settings(BaseSettings):
     # requires uri and db_name
     mongo: MongoSettings
 
+    # MPContribs_mpcontribs__*
+    mpcontribs: MPContribsSettings = Field(default_factory=MPContribsSettings)
+
     # MPContribs_aws__*
     aws: AwsSettings = Field(default_factory=AwsSettings)
 

mpcontribs-api/src/mpcontribs_api/dependencies.py

@@ -72,3 +72,14 @@ def require_user(user: UserDep) -> User:
         raise AuthenticationError("authentication required")
     return user
 
+
+# AuthedDep = Annotated[User, Depends(require_user)]
+
+
+# def require_role(role: str):
+#     def checker(user: AuthedDep) -> User:
+#         if not user.has_role(role):
+#             raise PermissionError(required_role=role)
+#         return user
+
+#     return Annotated[User, Depends(checker)]

mpcontribs-api/src/mpcontribs_api/domains/_shared/bulk.py

@@ -27,11 +27,27 @@ class BulkWriteSummary[T](BaseModel):
     failed: list[BulkFailure]
 
 
-class BulkDeleteSummary(BaseModel):
+class BulkDeleteSummary[T](BaseModel):
     num_deleted: int
     num_children_deleted: int
 
 
+class BulkUpdateSummary(BaseModel):
+    """Result of a filtered bulk update.
+
+    No per-item result, reflects MongoDB's bulk update response.
+
+    Attributes:
+        matched: documents the (scoped) filter matched
+        modified: documents whose stored value actually changed
+        projects: the projects the update touched, so the caller can see its blast radius
+    """
+
+    matched: int
+    modified: int
+    projects: list[str]
+
+
 def bulk_failure_from_exception(index: int, identifier: dict[str, Any] | None, exc: BaseException) -> BulkFailure:
     """Translate any exception into a BulkFailure entry.
 

mpcontribs-api/src/mpcontribs_api/domains/_shared/filters.py

@@ -3,6 +3,28 @@
 
 from fastapi_filter.contrib.beanie import Filter
 
+from mpcontribs_api.domains._shared.types import nfc_normalize
+
+
+def _normalize_query_values(value: Any) -> Any:
+    """Recursively NFC-normalize every string in a built query condition value.
+
+    fastapi-filter wraps a filter's value into operator dicts before it reaches us (``{"$ne": v}``,
+    ``{"$in": [...]}``, ``{"$regex": ".*v.*", "$options": "i"}``), so a query term can be nested a
+    few levels deep. Normalizing every string — bare, inside operator dicts, and inside ``$in``/
+    ``$nin`` lists — means a term typed with a canonically-equivalent spelling (e.g. the OHM SIGN
+    U+2126 vs the Greek omega) matches NFC-normalized stored data. NFC is a no-op on ASCII, so ids,
+    md5 hex, and regex metacharacters are untouched. Dict *keys* (field names, ``$``-operators) are
+    ASCII and left as-is.
+    """
+    if isinstance(value, str):
+        return nfc_normalize(value)
+    if isinstance(value, Mapping):
+        return {key: _normalize_query_values(sub) for key, sub in value.items()}
+    if isinstance(value, list):
+        return [_normalize_query_values(item) for item in value]
+    return value
+
 
 class BaseFilter(Filter):
     """Base filter that bridges Beanie's ``_id`` alias and fastapi-filter's raw field names.
@@ -13,11 +35,24 @@ class BaseFilter(Filter):
     nothing, while a direct ``Document.id == x`` lookup (which Beanie resolves to ``_id``) succeeds.
     Remapping the ``id`` key to ``_id`` here keeps the two read paths consistent.
 
+    String values are also NFC-normalized (see :func:`_normalize_query_values`) so queries match the
+    NFC-normalized units, labels, and other display strings stored on the write path.
+
+    Store/query normalization invariant: this NFC pass is a global catch-all and only lines up with
+    fields stored at NFC or lighter. A field normalized *more* aggressively on write (e.g. a
+    ``SearchStr`` identifier that is NFKC-folded and casefolded, or an ``NFKCStr`` name) must declare
+    that same normalization type on its filter field so the query value is folded identically —
+    otherwise an exact/``__in``/``__neq`` lookup silently misses. See ``ContributionFilter.identifier``
+    (SearchStr) and the component ``name`` filters (NFKCStr) for the pattern.
+
     Domain filters should subclass this instead of fastapi-filter's ``Filter`` directly.
     """
 
     def _get_filter_conditions(self, nesting_depth: int = 1) -> list[tuple[Mapping[str, Any], Mapping[str, Any]]]:
         return [
-            ({"_id" if key == "id" else key: value for key, value in condition.items()}, options)
+            (
+                {("_id" if key == "id" else key): _normalize_query_values(value) for key, value in condition.items()},
+                options,
+            )
             for condition, options in super()._get_filter_conditions(nesting_depth)
         ]

mpcontribs-api/src/mpcontribs_api/domains/_shared/models.py

@@ -9,7 +9,7 @@
 from pymongo.results import DeleteResult
 
 from mpcontribs_api import pagination
-from mpcontribs_api.domains._shared.types import MD5Hash
+from mpcontribs_api.domains._shared.types import MD5Hash, NFKCStr
 from mpcontribs_api.projection import SparseFieldsModel
 
 
@@ -84,7 +84,8 @@ class ComponentIn(BaseModel):
     fields for their resource.
     """
 
-    name: str
+    # NFKC-normalized so compatibility-equivalent spellings of a name collapse to one stored form.
+    name: NFKCStr
 
 
 class Component(BaseDocumentWithInput[PydanticObjectId]):
@@ -95,7 +96,7 @@ class Component(BaseDocumentWithInput[PydanticObjectId]):
     never define a component's content identity.
     """
 
-    name: str
+    name: NFKCStr
     # Server-computed; the placeholder default is overwritten by ``_recompute_md5`` on validation.
     md5: MD5Hash = Field(default="0" * 32)