Decode typeless Tika matches and honor masked rule offsets - #179
Merged
Conversation
Tika defaults an absent match type to "string", hex-decoding 0x… values
and unescaping backslash escapes. The generator only decoded when
type="string" was explicit, so 26 value-bearing typeless matches were
emitted as inert literal-text matchers ("0x30" matched the four ASCII
characters, not the DER SEQUENCE tag). Treat typeless value-bearing
matches as strings; the six structural minShouldMatch containers carry
no value and pass through unchanged.
Separately, the mask branch dropped the match's own offset: every
masked segment anchored at the mask position alone, so whole-byte-
masked rules at non-zero offsets (pkcs7 at 2:6, pkcs12 at 7:15) probed
offset 0. Segments now shift by a fixed offset, a single segment slides
with a range offset, and multi-segment masks at range offsets are
recorded as unsupported: the segments can't stay mutually aligned while
the range slides.
Skipping those rules exposed a broadening hazard: a match with children
requires some child to match alongside it, so a parent that lost every
child to an unsupported rule was emitted alone and matched far more
than the rule it came from (the pkcs7-signature DER rule would have
classified nearly every DER SEQUENCE as a signature; the bare parents
already shipping matched any "t1" prefix as a Touhou replay and TGA-
shaped bytes at priority 90). Such parents are now skipped recursively;
supported siblings survive.
Newly live magics: OneNote sections/TOCs, PKCS#7 (mime family label),
PKCS#12, Atari ST floppy images, 64-bit little-endian Mach-O objects,
SZDD compression, ZIM archives, and BOM-prefixed WebVTT. Detection
retired with the over-broad bare parents: Sereal (versions were
misidentified anyway), TGA content sniffing (new in 2.0, absent from
1.x), and the never-matching E-AC-3/AAC/FAT/marc leftovers — AC-3 and
ID3-tagged MP3 keep Tika's deliberate bare fallback rules.
The timestamped-data override in data/custom.xml existed only to paper
over the typeless decode gap; the upstream rule now generates
identically (verified byte-identical) and the override is removed.
Unsupported-rule manifest: 60 -> 77 (12 orphaned parents, 2 multi-
segment range masks, 3 reclassified).
Shifting a mask segment by its match's offset could push the emitted offset past MAX_MAGIC_OFFSET after the original offset had already passed validation; derived segment offsets and range endpoints are now revalidated, with boundary tests for both forms. Pin the deliberate Sereal retirement (v1/v2/v3 content stays octet-stream, .srl still resolves by extension) and the all-zero-mask skip with its orphaned-parent propagation. Correct the unsupported-manifest arithmetic: 17 children-kind entries (15 newly recorded orphaned parents + 2 reclassified regex parents) plus 2 multi-segment range masks explain 60 -> 77. The pkcs12 probe's trailing OID is id-data, not SignedData.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #177, which documented that Tika 4.0.0's typeless
<match>elements generate as inert literal-text matchers. This fixes that defect, a second latent defect in mask handling, and the broadening hazard that fixing them exposed.The defects
typetostring, hex-decoding0x…values and unescaping backslash escapes. The generator only decoded whentype="string"was explicit, so<match value="0x30" offset="0">matched the four ASCII characters0x30, not the DER SEQUENCE tag byte.2:6, pkcs12 at7:15) probed offset 0.application/pkcs7-signature. Parents that lose all children are now skipped recursively; supported siblings survive.Inventory (data/tika.xml; custom.xml has zero typeless nodes)
32 typeless
<match>nodes = 26 value-bearing + 6 structuralminShouldMatchcontainers (no value; handling pinned unchanged — they emit as[offset, nil, children]exactly as before).Of the 26 value-bearing: 21 decode-changing, 5 already-identical plain literals (jigdo, sas-xport, spss-sav, jQuery/React comment probes — verified byte-identical generation).
Whole-byte-masked rules at non-zero offsets:
pkcs7-mime 2:6(single FF-run → range slides, now expressible),pkcs7-signature 2:6(two segments → inexpressible, skipped),x-pkcs12 7:15(single run → expressible), threex509-cert;format=der 10:50rules (one two-segment → skipped; two single-run → expressible but discarded with their already-unsupported0xFFF8-masked parent). All fixed-offset masked rules sit at offset 0 and generate unchanged (Ogg family, outlook-pst, Type 1 fonts, RIFF/AIFF/WebP/AVI).Classification matrix — newly live (12 rules, 11 types)
application/onenote;format=one{7B5C52E4-D88C-4DA7-AEB1-5378D02996D3}application/onenote;format=onetoc2{43FF2FA1-…}application/pkcs7-mime0x30@0 + 12-byte smime-ct arc OID prefix @2:61.2.840.113549.1.9.16.1.*. Sibling CMS types (compressedData…) now resolve here instead of octet-stream; timestamped-data (priority 60 > 50) still wins its exact OID — both pinned in testsapplication/timestamped-dataapplication/x-pkcs120x30@0 →0x02010330@2:6 → SignedData OID @7:15application/x-mach-o-object0xCFFAEDFE@0 + filetype 1 @12application/x-mach-o(50) — .o files gain the specific verdictapplication/x-atari-floppy-disk-image0x9602@0 + flags @4 + zeroed serial @11application/x-ms-compress-szddSZDD\x88\xF0'3Aapplication/x-zimZIM\x04@0text/vttWEBVTT(×2, upstream duplicate) and UTF-8 BOM +WEBVTTClassification matrix — retired bare parents (propagation)
application/pkcs7-signatureDER branch0x30parentapplication/x-touhout1prefix matchedimage/x-tgaapplication/sereal;version=1/2/3=srl/=órlaudio/ac3p50audio/eac3audio/mpegp50ID3branchID3fallback — pinned in test; mp3 fixtures (all ID3-prefixed) still passaudio/x-aacID3branchapplication/x-fat-diskimage\xEBtextEB xx 90would have matched every DOS/NTFS boot sector)application/marc,image/vnd.dxf;format=asciiUnsupported-rule manifest
60 → 77: +15 newly recorded orphaned parents, +2 multi-segment range masks, and 2 regex parents reclassified as no-supported-children (17 children-kind entries in all — same skip either way for the reclassified pair). Count, sha, and warning-line pins updated deliberately. Derived mask-segment offsets are revalidated against
MAX_MAGIC_OFFSETafter shifting, and the sereal retirement and all-zero-mask skip are pinned by tests (review round 1).Verification
rake(suite +tables:check) green: 677 runs, 1715 assertions, 0 failures;rake updateidempotent