Skip to content

Decode typeless Tika matches and honor masked rule offsets - #179

Merged
jeremy merged 2 commits into
mainfrom
generator-typeless-matches
Aug 24, 2026
Merged

Decode typeless Tika matches and honor masked rule offsets#179
jeremy merged 2 commits into
mainfrom
generator-typeless-matches

Conversation

@jeremy

@jeremy jeremy commented Aug 24, 2026

Copy link
Copy Markdown
Member

Follow-up to #177, which documented that Tika 4.0.0's typeless <match> elements generate as inert literal-text matchers. This fixes that defect, a second latent defect in mask handling, and the broadening hazard that fixing them exposed.

The defects

  1. Typeless matches were never decoded. Tika defaults an absent type to string, hex-decoding 0x… values and unescaping backslash escapes. The generator only decoded when type="string" was explicit, so <match value="0x30" offset="0"> matched the four ASCII characters 0x30, not the DER SEQUENCE tag byte.
  2. The mask branch dropped the match's own offset. Masked segments anchored at the mask position alone, so whole-byte-masked rules at non-zero offsets (pkcs7 OIDs at 2:6, pkcs12 at 7:15) probed offset 0.
  3. Orphaned parents broadened their rules. A match with children requires some child to match alongside it. When every child was unsupported, the parent was emitted alone and matched far more than the rule it encoded. Fixing 1+2 without addressing this would have classified nearly every DER SEQUENCE as application/pkcs7-signature. Parents that lose all children are now skipped recursively; supported siblings survive.

Inventory (data/tika.xml; custom.xml has zero typeless nodes)

32 typeless <match> nodes = 26 value-bearing + 6 structural minShouldMatch containers (no value; handling pinned unchanged — they emit as [offset, nil, children] exactly as before).

Of the 26 value-bearing: 21 decode-changing, 5 already-identical plain literals (jigdo, sas-xport, spss-sav, jQuery/React comment probes — verified byte-identical generation).

Whole-byte-masked rules at non-zero offsets: pkcs7-mime 2:6 (single FF-run → range slides, now expressible), pkcs7-signature 2:6 (two segments → inexpressible, skipped), x-pkcs12 7:15 (single run → expressible), three x509-cert;format=der 10:50 rules (one two-segment → skipped; two single-run → expressible but discarded with their already-unsupported 0xFFF8-masked parent). All fixed-offset masked rules sit at offset 0 and generate unchanged (Ogg family, outlook-pst, Type 1 fonts, RIFF/AIFF/WebP/AVI).

Classification matrix — newly live (12 rules, 11 types)

Type Rule Verdict
application/onenote;format=one 16-byte GUID {7B5C52E4-D88C-4DA7-AEB1-5378D02996D3} Safe; highly discriminating. Probe test added
application/onenote;format=onetoc2 16-byte GUID {43FF2FA1-…} Safe; probe test added
application/pkcs7-mime 0x30 @0 + 12-byte smime-ct arc OID prefix @2:6 Tika's deliberate coarse family label for 1.2.840.113549.1.9.16.1.*. Sibling CMS types (compressedData…) now resolve here instead of octet-stream; timestamped-data (priority 60 > 50) still wins its exact OID — both pinned in tests
application/timestamped-data upstream rule, decoded Generates identically to the custom.xml override carried since #177; override removed, output verified byte-identical
application/x-pkcs12 0x30 @00x02010330 @2:6 → SignedData OID @7:15 Realistic DER layout probed in test; no priority collisions (checked TSD/pkcs7-mime ordering)
application/x-mach-o-object 0xCFFAEDFE @0 + filetype 1 @12 Completes the existing 4-branch rule (3 branches already live). Priority 55 sorts ahead of bare application/x-mach-o (50) — .o files gain the specific verdict
application/x-atari-floppy-disk-image 0x9602 @0 + flags @4 + zeroed serial @11 10 matched bytes total; probe test added
application/x-ms-compress-szdd 9-byte SZDD\x88\xF0'3A Highly discriminating; probe test
application/x-zim ZIM\x04 @0 4 distinctive bytes; probe test
text/vtt UTF-16BE BOM + WEBVTT (×2, upstream duplicate) and UTF-8 BOM + WEBVTT UTF-8-BOM'd VTT gains detection (probe test). The FEFF+ASCII rules mirror Tika verbatim; near-unmatchable but harmless

Classification matrix — retired bare parents (propagation)

Type Today Verdict
application/pkcs7-signature DER branch dead literal 0x30 parent Skipped whole (2-segment mask @range). PEM sibling survives alone — pinned in test. Negative tests: bare SEQUENCE and certificate-shaped DER stay octet-stream
application/x-touhou live: any t1 prefix matched Removed — active misdetection fixed; negative test
image/x-tga live: 4 header-shaped bytes @1, priority 90 Removed — collision-prone rule new in 2.0 (1.x had no TGA magic); back to extension-only, negative test
application/sereal;version=1/2/3 live: bare =srl/=órl Removed — version byte unverifiable, v2 was mislabeled v1; niche format, extension mapping remains. Only true detection loss in this PR
audio/ac3 p50 live bare 2-byte syncword Parent dropped; detection preserved via Tika's deliberate childless p40 syncword rule (test)
audio/eac3 dead: shadowed by identical ac3 bytes, never won Removed, zero change
audio/mpeg p50 ID3 branch live Dropped, but ID3-tagged MP3 detection preserved via Tika's deliberate childless priority-10 ID3 fallback — pinned in test; mp3 fixtures (all ID3-prefixed) still pass
audio/x-aac ID3 branch never won (mpeg sorted first) Removed, zero change
application/x-fat-diskimage dead literal \xEB text Removed, zero change (leaf regex unsupported; bare EB xx 90 would have matched every DOS/NTFS boot sector)
application/marc, image/vnd.dxf;format=ascii already dead (regex ancestors) Manifest bookkeeping only

Unsupported-rule manifest

60 → 77: +15 newly recorded orphaned parents, +2 multi-segment range masks, and 2 regex parents reclassified as no-supported-children (17 children-kind entries in all — same skip either way for the reclassified pair). Count, sha, and warning-line pins updated deliberately. Derived mask-segment offsets are revalidated against MAX_MAGIC_OFFSET after shifting, and the sereal retirement and all-zero-mask skip are pinned by tests (review round 1).

Verification

  • Full rake (suite + tables:check) green: 677 runs, 1715 assertions, 0 failures; rake update idempotent
  • Read-budget test holds unchanged (jpeg/png/gif/pdf)
  • Generator tests: typeless decode, structural containers, mask anchoring (fixed + range), recursive orphan propagation, sibling retention
  • Runtime probes for every newly-live rule; negatives for every removal that changes behavior

jeremy added 2 commits August 24, 2026 08:38
Tika defaults an absent match type to "string", hex-decoding 0x… values
and unescaping backslash escapes. The generator only decoded when
type="string" was explicit, so 26 value-bearing typeless matches were
emitted as inert literal-text matchers ("0x30" matched the four ASCII
characters, not the DER SEQUENCE tag). Treat typeless value-bearing
matches as strings; the six structural minShouldMatch containers carry
no value and pass through unchanged.

Separately, the mask branch dropped the match's own offset: every
masked segment anchored at the mask position alone, so whole-byte-
masked rules at non-zero offsets (pkcs7 at 2:6, pkcs12 at 7:15) probed
offset 0. Segments now shift by a fixed offset, a single segment slides
with a range offset, and multi-segment masks at range offsets are
recorded as unsupported: the segments can't stay mutually aligned while
the range slides.

Skipping those rules exposed a broadening hazard: a match with children
requires some child to match alongside it, so a parent that lost every
child to an unsupported rule was emitted alone and matched far more
than the rule it came from (the pkcs7-signature DER rule would have
classified nearly every DER SEQUENCE as a signature; the bare parents
already shipping matched any "t1" prefix as a Touhou replay and TGA-
shaped bytes at priority 90). Such parents are now skipped recursively;
supported siblings survive.

Newly live magics: OneNote sections/TOCs, PKCS#7 (mime family label),
PKCS#12, Atari ST floppy images, 64-bit little-endian Mach-O objects,
SZDD compression, ZIM archives, and BOM-prefixed WebVTT. Detection
retired with the over-broad bare parents: Sereal (versions were
misidentified anyway), TGA content sniffing (new in 2.0, absent from
1.x), and the never-matching E-AC-3/AAC/FAT/marc leftovers — AC-3 and
ID3-tagged MP3 keep Tika's deliberate bare fallback rules.

The timestamped-data override in data/custom.xml existed only to paper
over the typeless decode gap; the upstream rule now generates
identically (verified byte-identical) and the override is removed.

Unsupported-rule manifest: 60 -> 77 (12 orphaned parents, 2 multi-
segment range masks, 3 reclassified).
Shifting a mask segment by its match's offset could push the emitted
offset past MAX_MAGIC_OFFSET after the original offset had already
passed validation; derived segment offsets and range endpoints are now
revalidated, with boundary tests for both forms.

Pin the deliberate Sereal retirement (v1/v2/v3 content stays
octet-stream, .srl still resolves by extension) and the all-zero-mask
skip with its orphaned-parent propagation.

Correct the unsupported-manifest arithmetic: 17 children-kind entries
(15 newly recorded orphaned parents + 2 reclassified regex parents)
plus 2 multi-segment range masks explain 60 -> 77. The pkcs12 probe's
trailing OID is id-data, not SignedData.
@jeremy
jeremy merged commit 731f570 into main Aug 24, 2026
20 checks passed
@jeremy
jeremy deleted the generator-typeless-matches branch August 24, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant