From cee790c383549320ca652729783fd768c1f5617e Mon Sep 17 00:00:00 2001 From: Jeremy Daer Date: Sat, 22 Aug 2026 12:18:34 -0700 Subject: [PATCH 1/2] Update Tika MIME data to the 4.0.0 release Repin script/download_tika_data.rb from the 2026-05-11 snapshot (c2972478) to the commit the 4.0.0 tag dereferences to (514e1b3d) and regenerate lib/marcel/tables.rb with rake update. Upstream's diff is confined to five types, classified as follows: * application/vnd.android.axml: new type with working magic (RES_XML_TYPE header + string-pool chunk). Additive; previously application/octet-stream. * application/pdf: a new priority-40 regex fallback for %PDF headers behind long print-job-ticket comment preambles (TIKA-4782). The generator's regex subset can't express it, so it joins the pinned unsupported-rule manifest (59 -> 60 rules, sha and warning-line counts updated in generate_tables.rb and generate_tables_test.rb). PDF detection is unchanged. * application/pkcs7-signature, application/pkcs7-mime, application/x-pkcs12: Tika rebuilt these DER magics on typeless elements. Tika defaults an absent type to "string" and hex-decodes the value, but the generator only decodes when type="string" is explicit, so both the old rules (five 0x3080-0x3084 branches) and the new ones generate as inert literal-text matchers. No behavior change: dead before, dead after. The PEM pkcs7-signature branch still works. * application/timestamped-data: the one live behavior change. The pre-4.0 magic was a working byte match on 30 80 06 0B 2A 86 48 86 F7; the 4.0.0 rewrite nests the OID under a typeless 0x30 match, which would have silently killed content detection marcel 2.0 shipped. data/custom.xml now carries upstream's rule with the match types spelled out: same shape and priority, requiring the full id-ct-timestampedData OID (.9.16.1.31) across the DER length forms. Net effect is a fix in both directions: definite long-form encodings are now detected (previously only indefinite 30 80), and sibling CMS content types in the 1.2.840.113549.1.9.16.1.* arc (compressedData, authData, ...) are no longer misdetected as timestamped-data, which the loose 9-byte prefix allowed. New fixtures pin timestamped-data (both DER length forms) and AXML detection, and a regression test pins the CMS false-positive fix. Follow-up worth its own change: teach the generator to decode typeless matches the way Tika does (or count them as unsupported) instead of emitting silent literal-text matchers; that would enliven the CMS/DER magics carried dead in the tables today. --- data/custom.xml | 16 ++++ data/tika.xml | 86 +++++++++++++----- lib/marcel/tables.rb | 8 +- script/download_tika_data.rb | 4 +- script/generate_tables.rb | 8 +- .../application/timestamped-data/definite.tsd | Bin 0 -> 39 bytes .../timestamped-data/indefinite.tsd | Bin 0 -> 39 bytes .../application/vnd.android.axml/axml.xml | Bin 0 -> 36 bytes test/generate_tables_test.rb | 4 +- test/magic_test.rb | 11 +++ 10 files changed, 105 insertions(+), 32 deletions(-) create mode 100644 test/fixtures/magic/application/timestamped-data/definite.tsd create mode 100644 test/fixtures/magic/application/timestamped-data/indefinite.tsd create mode 100644 test/fixtures/magic/application/vnd.android.axml/axml.xml diff --git a/data/custom.xml b/data/custom.xml index 1df40ef..767b538 100644 --- a/data/custom.xml +++ b/data/custom.xml @@ -243,4 +243,20 @@ + + + + + + + + diff --git a/data/tika.xml b/data/tika.xml index 85ab267..81caa38 100644 --- a/data/tika.xml +++ b/data/tika.xml @@ -1,5 +1,5 @@ - + + + + + + + @@ -841,6 +856,16 @@ + + + + @@ -879,6 +904,15 @@ + + + + + + @@ -886,34 +920,28 @@ - - + + - - - - - - - - - - - - + mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="2:6"/> - - + + + + + @@ -4853,6 +4881,20 @@ + + + + + + + + diff --git a/lib/marcel/tables.rb b/lib/marcel/tables.rb index 2743303..dd69e2c 100644 --- a/lib/marcel/tables.rb +++ b/lib/marcel/tables.rb @@ -2799,6 +2799,8 @@ module Marcel ['image/svg+xml', [[0, b['\n".freeze TIKA_PREAMBLE = /\A(?:<\?xml[^\n]*\?>\n)?/.freeze diff --git a/script/generate_tables.rb b/script/generate_tables.rb index 2186289..9770787 100755 --- a/script/generate_tables.rb +++ b/script/generate_tables.rb @@ -180,11 +180,11 @@ def self.namespace_uri(value, source) end class UnsupportedRules - # Tika currently contains 59 unsupported XML rules: 58 magic matches plus one root-XML - # rule. Their pretty-printed warnings span 113 physical lines, so pin the canonical rule + # Tika currently contains 60 unsupported XML rules: 59 magic matches plus one root-XML + # rule. Their pretty-printed warnings span 114 physical lines, so pin the canonical rule # set rather than stderr layout. - EXPECTED_COUNT = 59 - EXPECTED_SHA256 = "f42a5b9b0462e1c467e068b114c33706f28394d52da76111ae3271cdc5e606ce" + EXPECTED_COUNT = 60 + EXPECTED_SHA256 = "61d54e5e5b543d3d3721e4d056e43e2f928fb08b36e94c8b1da88a416a18bb48" def initialize @signatures = [] diff --git a/test/fixtures/magic/application/timestamped-data/definite.tsd b/test/fixtures/magic/application/timestamped-data/definite.tsd new file mode 100644 index 0000000000000000000000000000000000000000..a7821b7b451b57080b32874ec8dbc01d77e8ba22 GIT binary patch literal 39 ucmXqLVo+w|)@t);`_9YADZnVdK!io8Br`X)xFj*RAT>oHsj?)sSPuZoVheu& literal 0 HcmV?d00001 diff --git a/test/fixtures/magic/application/timestamped-data/indefinite.tsd b/test/fixtures/magic/application/timestamped-data/indefinite.tsd new file mode 100644 index 0000000000000000000000000000000000000000..26e49222d979219739037edd83321b89b6a9a51b GIT binary patch literal 39 ncmXqLVB^+m^Jx3d%g8CfD8HbAMW7(DGABPVMHfmdG5`Sp-vSEM literal 0 HcmV?d00001 diff --git a/test/fixtures/magic/application/vnd.android.axml/axml.xml b/test/fixtures/magic/application/vnd.android.axml/axml.xml new file mode 100644 index 0000000000000000000000000000000000000000..d98b1157b9b40cb506fd6e0c870b1e2d86050bfd GIT binary patch literal 36 VcmZQ(;9yW;U|?WmkO4yM000ZL0BHaK literal 0 HcmV?d00001 diff --git a/test/generate_tables_test.rb b/test/generate_tables_test.rb index 88a7a03..202a3e3 100644 --- a/test/generate_tables_test.rb +++ b/test/generate_tables_test.rb @@ -238,8 +238,8 @@ class Marcel::GenerateTablesTest < Marcel::TestCase assert status.success?, errors warning_lines = errors.lines - assert_equal "Skipped 59 unsupported magic rules\n", warning_lines.pop - assert_equal 113, warning_lines.size + assert_equal "Skipped 60 unsupported magic rules\n", warning_lines.pop + assert_equal 114, warning_lines.size assert File.exist?(tables_path) end end diff --git a/test/magic_test.rb b/test/magic_test.rb index 8eb4f2e..7babe00 100644 --- a/test/magic_test.rb +++ b/test/magic_test.rb @@ -37,6 +37,17 @@ def read(*) end end + # Before Tika 4.0.0, the timestamped-data magic matched any 11-byte OID under the + # 1.2.840.113549 arc, so sibling CMS content types (compressedData, authData, ...) in + # 1.2.840.113549.1.9.16.1.* were misdetected as application/timestamped-data. The 4.0.0 + # rules (carried in data/custom.xml with explicit match types) require the full + # id-ct-timestampedData OID ending in .31. + test "other CMS content types are not misdetected as timestamped-data" do + compressed_data = "\x30\x80\x06\x0B\x2A\x86\x48\x86\xF7\x0D\x01\x09\x10\x01\x09\xA0\x80".b + + assert_equal "application/octet-stream", Marcel::MimeType.for(compressed_data) + end + test "add and remove type" do Marcel::Magic.add('application/x-my-thing', extensions: 'mtg', parents: 'application/json') Marcel::Magic.remove('application/x-my-thing') From b48cd17cae1d6e4f33f40b1eaed92fe546e500f5 Mon Sep 17 00:00:00 2001 From: Jeremy Daer Date: Sat, 22 Aug 2026 12:25:13 -0700 Subject: [PATCH 2/2] Make the definite-length TSD fixture valid DER The fixture encoded a 35-byte body with the two-byte long form (30 82 00 23), which DER forbids: lengths must use the shortest encoding. Grow the payload past 255 bytes so the 30 82 header is legitimately required, keeping the OID at offset 4 that the fixture exists to exercise. --- .../application/timestamped-data/definite.tsd | Bin 39 -> 325 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/test/fixtures/magic/application/timestamped-data/definite.tsd b/test/fixtures/magic/application/timestamped-data/definite.tsd index a7821b7b451b57080b32874ec8dbc01d77e8ba22..c0a270ed6e0059dc9a3319b2076d58112b56ae69 100644 GIT binary patch literal 325 zcmXqLVsvEV)@t);`_9YADZnVdpo!6drHN6eBr`X)xFj*RAT>oHsj?)sSWjV82uDZ& E05vFg!ToHsj?)sSPuZoVheu&