Update Tika MIME data to the 4.0.0 release - #177

Repins script/download_tika_data.rb from the 2026-05-11 snapshot (c2972478) to the commit the 4.0.0 release tag dereferences to (514e1b3d), regenerates lib/marcel/tables.rb via rake update, and classifies every resulting behavior change.

Classified behavior diff

Upstream's data diff since our pin is confined to five types.

(a) Additive

• application/vnd.android.axml — new type with working magic (RES_XML_TYPE header 03 00 08 00 + RES_STRING_POOL_TYPE at offset 8). Previously application/octet-stream. Fixture added.
• application/pdf — a new priority-40 regex fallback for %PDF headers behind long print-job-ticket comment preambles (TIKA-4782). The generator's regex subset can't express it, so it joins the pinned unsupported-rule manifest (59 → 60 rules; sha + warning-line counts updated). PDF detection is unchanged.

Inert (dead before, dead after)

• application/pkcs7-signature (DER branches), application/pkcs7-mime, application/x-pkcs12 — Tika rebuilt these DER magics on typeless <match value="0x..."> elements. Tika defaults an absent type to string and hex-decodes; our generator only decodes when type="string" is explicit, so both the old rules (five 0x3080–0x3084 branches) and the new ones generate as inert literal-text matchers. No behavior change either direction. The PEM -----BEGIN PKCS7 branch still works.

(b) The one live flip: application/timestamped-data

The pre-4.0 magic was a working byte match on 30 80 06 0B 2A 86 48 86 F7 (type="string", hex-decoded). The 4.0.0 rewrite nests the OID under a typeless 0x30 parent, which — via the generator gap above — would have silently killed the content detection marcel 2.0 shipped.

data/custom.xml now carries upstream's rule with the match types spelled out: same shape and priority 60, requiring the full id-ct-timestampedData OID (1.2.840.113549.1.9.16.1.31) scanned across the DER length forms (offset="2:6"). Net effect is a correctness fix in both directions, each pinned by a test:

• definite long-form encodings (30 82 …) are now detected — previously only the indefinite 30 80 form matched;
• sibling CMS content types in the 1.2.840.113549.1.9.16.1.* arc (compressedData .9, authData .2, …) are no longer misdetected as timestamped-data, which the loose 9-byte prefix allowed. Upstream's own comments confirm the intent: the masked .9.16.1.* magic belongs to pkcs7-mime, with timestamped-data winning only on its exact OID at higher priority.

(c) Canonical names / aliases / extensions

None. The tables diff outside MAGIC is empty.

Verification

• Full rake green (668 runs), including tables:check and the pinned unsupported-rule manifest.
• rake update is idempotent — re-running leaves the tree unchanged.
• New fixtures: timestamped-data in both DER length forms, AXML; new regression test for the CMS-arc false positive.

Follow-up (not this PR)

Teach the generator to decode typeless matches the way Tika does — or count them as unsupported — instead of emitting silent literal-text matchers. That would enliven the CMS/DER magics carried dead in the tables today (~14 entries), each needing individual review.

Review notes

• The upstream typeless timestamped-data matcher carried in the tables is near-inert rather than fully dead: a file beginning with the literal ASCII text 0x30 and carrying the binary TSD OID within bytes 2–6 would still match. Any such collision necessarily contains the full id-ct-timestampedData OID, so the detection it produces is the same one the live rule gives real TSD files; not worth adding suppression machinery to the generator for. Goes away with the typeless-decode follow-up.
• The definite-length fixture initially used a non-minimal length encoding (BER, not DER); rebuilt with a >255-byte payload so the 30 82 long form is legitimately required.