Skip to content

Update Tika MIME data to the 4.0.0 release #177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremy merged 2 commits into from
Aug 22, 2026
Merged

Update Tika MIME data to the 4.0.0 release #177

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cee790c
Update Tika MIME data to the 4.0.0 release
jeremy Aug 22, 2026
b48cd17
Make the definite-length TSD fixture valid DER
jeremy Aug 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions data/custom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,4 +243,20 @@

<sub-class-of type="application/xml" />
</mime-type>
<!-- Tika 4.0.0 rebuilt the CMS/DER magics (pkcs7, pkcs12, timestamped-data) on typeless
<match value="0x..."> elements. Tika defaults an absent type to "string" and
hex-decodes the value; the generator only decodes when type="string" is explicit, so
those upstream rules generate as inert literal-text matchers. That leaves
application/timestamped-data, whose pre-4.0 magic was a working byte match, without
content detection. Carry upstream's rule (same shape and priority, from
tika-mimetypes.xml @ 4.0.0) with the types spelled out: SEQUENCE tag, then the
id-ct-timestampedData OID (1.2.840.113549.1.9.16.1.31) scanned across the DER
length forms. Remove once the generator decodes typeless matches like Tika does. -->
<mime-type type="application/timestamped-data">
<magic priority="60">
<match value="0x30" type="string" offset="0">
<match value="0x060B2A864886F70D010910011F" type="string" offset="2:6" />
</match>
</magic>
</mime-type>
</mime-info>
86 changes: 64 additions & 22 deletions data/tika.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Downloaded from https://raw.githubusercontent.com/apache/tika/c29724782854bb5a319f4a1940d0828a58ace3f9/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml -->
<!-- Downloaded from https://raw.githubusercontent.com/apache/tika/514e1b3d8d29726d02ac6a12479d95f5db263379/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml -->
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
Expand Down Expand Up @@ -353,6 +353,21 @@
<sub-class-of type="application/java-archive"/>
<glob pattern="*.apk"/>
</mime-type>
<mime-type type="application/vnd.android.axml">
<acronym>AXML</acronym>
<_comment>Android Binary XML</_comment>
<tika:link>https://developer.android.com/guide/topics/manifest/manifest-intro</tika:link>
<!-- Compiled AndroidManifest.xml / res/*.xml inside an APK. The .xml extension would
otherwise route it to the XML parser, which fails on the binary header. Signature:
RES_XML_TYPE(0x0003)+headerSize(0x0008)=0x00080003 LE, plus RES_STRING_POOL_TYPE
(0x0001) at offset 8 (the variable per-file size at offset 4 is skipped). Not a
sub-class-of application/xml: must not reach an XML parser. -->
<magic priority="50">
<match value="0x03000800" type="string" offset="0">
<match value="0x0001" type="little16" offset="8"/>
</match>
</magic>
</mime-type>
<mime-type type="application/x-tika-java-enterprise-archive">
<sub-class-of type="application/java-archive"/>
<glob pattern="*.ear"/>
Expand Down Expand Up @@ -841,6 +856,16 @@
<match value="%PDF-2." type="string" offset="1:512"/>
</match>
</magic>
<magic priority="40">
<!-- Print-composition systems prepend a job ticket of %% comments that can run well
past the 512-byte window above (TIKA-4782). Up to 50 %% comment or blank lines of up
to 150 characters, and nothing else, may precede the header. 50x150 is deliberate:
the worst case stays inside the first 8K, which is all MagicDetector feeds a regex,
so every prefix within those bounds really is reachable.
Every quantifier is bounded and the group is atomic, so this cannot backtrack: without
the (?> a hostile CRLF run splits exponentially and hangs detection. Keep it that way. -->
<match value="(?>(?:%%[^\\r\\n]{0,148})?(?:\\r\\n|[\\r\\n])){1,50}%PDF-[12]\\." type="regex" offset="0"/>
</magic>
<magic priority="20">
<!-- Low priority match for %PDF-#.# near the start of the file -->
<!-- Can trigger false positives, so set the priority rather low here -->
Expand Down Expand Up @@ -879,41 +904,44 @@
<mime-type type="application/pkcs7-mime">
<glob pattern="*.p7m"/>
<glob pattern="*.p7c"/>
<!-- CMS content types in the 1.2.840.113549.1.9.16.1.* arc (compressedData .9, authData .2,
authEnvelopedData .23, ...) with the final OID byte masked. timestamped-data (.31) has the
same shape but its own higher-priority magic below. The .7 arc (signed/enveloped/...) is
matched by the pkcs7-signature block. Pkcs7Parser refines all of these to the smime-type. -->
<magic priority="50">
<match value="0x30" offset="0">
<match value="0x060B2A864886F70D0109100100" mask="0xFFFFFFFFFFFFFFFFFFFFFFFF00" type="string" offset="2:6"/>
</match>
</magic>
</mime-type>

<mime-type type="application/pkcs7-signature">
<glob pattern="*.p7s"/>
<magic priority="50">
<!-- PEM encoded -->
<match value="-----BEGIN PKCS7" type="string" offset="0"/>
<!-- DER encoded, sequence+length, object=pkcs7-signedData -->
<match value="0x3080" offset="0">
<!-- DER encoded: SEQUENCE then a CMS ContentInfo whose contentType is in the pkcs7 arc
1.2.840.113549.1.7.x (signedData .2, envelopedData .3, digestedData .5, encryptedData .6,
... last OID byte masked) followed by the [0] content tag (a0). offset 2:6 spans the DER
SEQUENCE length forms (short-form + 80/81/82/83/84), so small short-form containers such
as a bare digestedData are matched too. This is the coarse family label; Pkcs7Parser
refines the exact smime-type. -->
<match value="0x30" offset="0">
<match value="0x06092a864886f70d0107FFa0" type="string"
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="2"/>
</match>
<match value="0x3081" offset="0">
<match value="0x06092a864886f70d0107FFa0" type="string"
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="3"/>
</match>
<match value="0x3082" offset="0">
<match value="0x06092a864886f70d0107FFa0" type="string"
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="4"/>
</match>
<match value="0x3083" offset="0">
<match value="0x06092a864886f70d0107FFa0" type="string"
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="5"/>
</match>
<match value="0x3084" offset="0">
<match value="0x06092a864886f70d0107FFa0" type="string"
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="6"/>
mask="0xFFFFFFFFFFFFFFFFFFFF00FF" offset="2:6"/>
</match>
</magic>
</mime-type>

<mime-type type="application/timestamped-data">
<glob pattern="*.tsd"/>
<magic priority="50">
<match value="0x3080060B2A864886F7" type="string" offset="0"/>
<!-- CMS ContentInfo whose contentType is id-ct-timestampedData (1.2.840.113549.1.9.16.1.31),
matching the full OID across the DER SEQUENCE length forms (short + 1..4-byte long).
Higher priority than the masked pkcs7-mime .9.16.1.* magic so this exact OID wins. -->
<magic priority="60">
<match value="0x30" offset="0">
<match value="0x060B2A864886F70D010910011F" type="string" offset="2:6"/>
</match>
</magic>
</mime-type>

Expand Down Expand Up @@ -4853,6 +4881,20 @@
<mime-type type="application/x-pkcs12">
<glob pattern="*.p12"/>
<glob pattern="*.pfx"/>
<!-- PFX ::= SEQUENCE { version INTEGER (v3), authSafe ContentInfo SEQUENCE, ... }. The
INTEGER 3 then a SEQUENCE (02 01 03 30) tells it apart from CMS (OID next) and keys
(version 0). "02 01 03 30" alone also matches an SNMPv3 message (SEQUENCE { INTEGER 3,
SEQUENCE, ...}), so we additionally require authSafe's ContentInfo type OID: the pkcs7
arc 1.2.840.113549.1.7.x (id-data for password integrity, id-signedData for public-key),
last byte masked. Its offset varies with the two SEQUENCE length forms, hence 7:15. -->
<magic priority="50">
<match value="0x30" offset="0">
<match value="0x02010330" offset="2:6">
<match value="0x06092A864886F70D010701" mask="0xFFFFFFFFFFFFFFFFFFFF00"
type="string" offset="7:15"/>
</match>
</match>
</magic>
</mime-type>
<mime-type type="application/x-pkcs7-certificates">
<glob pattern="*.p7b"/>
Expand Down
8 changes: 6 additions & 2 deletions lib/marcel/tables.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2799,6 +2799,8 @@ module Marcel
['image/svg+xml', [[0, b['<!--'], [[4..1024, b['<svg']]]]]],
['application/cbor', [[0, b["\331\331\367"]]]],
['application/coreldraw', [[0, b['RIFF'], [[8, b['CDR']], [8, b['cdr']], [8, b['DES']], [8, b['des']]]]]],
['application/timestamped-data', [[0, b['0x30'], [[2..6, b["\006\v*\206H\206\367\r\001\t\020\001\037"]]]]]],
['application/timestamped-data', [[0, b['0'], [[2..6, b["\006\v*\206H\206\367\r\001\t\020\001\037"]]]]]],
['application/vnd.etsi.asic-e+zip', [[0, b["PK\003\004"], [[30, b['mimetypeapplication/vnd.etsi.asic-e+zip']]]]]],
['application/vnd.etsi.asic-s+zip', [[0, b["PK\003\004"], [[30, b['mimetypeapplication/vnd.etsi.asic-s+zip']]]]]],
['application/vnd.ms-excel.sheet.2', [[0, b["\t\000\004\000"], [[4, b["\000\000\020\000"]], [4, b["\000\000 \000"]], [4, b["\000\000@\000"]]]]]],
Expand Down Expand Up @@ -2907,13 +2909,14 @@ module Marcel
['application/onenote;format=one', [[0, b["\344R\\{"], [[4, b["\214\330"], [[6, b["\247M"], [[8, b['0xAEB15378D02996D3']]]]]]]]]],
['application/onenote;format=onetoc2', [[0, b["\241/\377C"], [[4, b["\331\357"], [[6, b['vL'], [[8, b['0x9EE210EA5722765F']]]]]]]]]],
['application/pgp-encrypted', [[0, b["\205"], [[3, b["\003"]]]]]],
['application/pkcs7-signature', [[0, b['-----BEGIN PKCS7']], [0, b['0x3080'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]], [0, b['0x3081'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]], [0, b['0x3082'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]], [0, b['0x3083'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]], [0, b['0x3084'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]]]],
['application/pkcs7-mime', [[0, b['0x30'], [[0, b["\006\v*\206H\206\367\r\001\t\020\001"]]]]]],
['application/pkcs7-signature', [[0, b['-----BEGIN PKCS7']], [0, b['0x30'], [[0, b["\006\t*\206H\206\367\r\001\a"], [[11, b["\240"]]]]]]]],
['application/postscript', [[0, b['%!']], [0, b["\004%!"]], [0, b["\305\320\323\306"]], [0, b['%!PS-Adobe-3.0 EPSF-3.0']]]],
['application/rtf', [[0, b["{\\rtf"]]]],
['application/sereal;version=1', [[0, b['=srl']]]],
['application/sereal;version=2', [[0, b['=srl']]]],
['application/sereal;version=3', [[0, b["=\363rl"]]]],
['application/timestamped-data', [[0, b["0\200\006\v*\206H\206\367"]]]],
['application/vnd.android.axml', [[0, b["\003\000\b\000"], [[8, b["\001\000"]]]]]],
['application/vnd.apple.mpegurl', [[0, b['#EXTM3U']]]],
['application/vnd.digilite.prolights', [[0, b["\177\fD+"]]]],
['application/vnd.fdf', [[0, b['%FDF-']]]],
Expand Down Expand Up @@ -3026,6 +3029,7 @@ module Marcel
['application/x-ole-storage', [[0..8, b["\320\317\021\340\241\261\032\341"]]]],
['application/x-parquet', [[0, b['PAR1']]]],
['application/x-pds', [[0, b['PDS_VERSION_ID']]]],
['application/x-pkcs12', [[0, b['0x30'], [[2..6, b['0x02010330'], [[0, b["\006\t*\206H\206\367\r\001\a"]]]]]]]],
['application/x-project', [[0, b['MPX,Microsoft Project for Windows,']]]],
['application/x-prt', [[8, b['0M3C']]]],
['application/x-quattro-pro;version=1+5', [[0, b["\000\000\002\000\001\020"]]]],
Expand Down
4 changes: 2 additions & 2 deletions script/download_tika_data.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
require 'uri'

# To update the MIME database, review a new Tika commit and update both values.
TIKA_COMMIT = "c29724782854bb5a319f4a1940d0828a58ace3f9".freeze
TIKA_DATA_SHA256 = "b213c35123f215e11a9fa12d7907c9bb0a5b79214a89eeaf343984ba69143e6d".freeze
TIKA_COMMIT = "514e1b3d8d29726d02ac6a12479d95f5db263379".freeze
TIKA_DATA_SHA256 = "ae8d2e4c7d8b0eb7669323129c39e81c0dc89614c3aaffc67751172b46413889".freeze
TIKA_URL = "https://raw.githubusercontent.com/apache/tika/#{TIKA_COMMIT}/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml".freeze
TIKA_PROVENANCE = "<!-- Downloaded from #{TIKA_URL} -->\n".freeze
TIKA_PREAMBLE = /\A(?:<\?xml[^\n]*\?>\n)?/.freeze
Expand Down
8 changes: 4 additions & 4 deletions script/generate_tables.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,11 +180,11 @@ def self.namespace_uri(value, source)
end

class UnsupportedRules
# Tika currently contains 59 unsupported XML rules: 58 magic matches plus one root-XML
# rule. Their pretty-printed warnings span 113 physical lines, so pin the canonical rule
# Tika currently contains 60 unsupported XML rules: 59 magic matches plus one root-XML
# rule. Their pretty-printed warnings span 114 physical lines, so pin the canonical rule
# set rather than stderr layout.
EXPECTED_COUNT = 59
EXPECTED_SHA256 = "f42a5b9b0462e1c467e068b114c33706f28394d52da76111ae3271cdc5e606ce"
EXPECTED_COUNT = 60
EXPECTED_SHA256 = "61d54e5e5b543d3d3721e4d056e43e2f928fb08b36e94c8b1da88a416a18bb48"

def initialize
@signatures = []
Expand Down
1 change: 1 addition & 0 deletions test/fixtures/magic/application/timestamped-data/definite.tsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0�A *�H�� ��0�,timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped bytes. timestamped byt
Expand Down
Binary file added test/fixtures/magic/application/timestamped-data/indefinite.tsd
View file
Open in desktop
Binary file not shown.
Binary file added test/fixtures/magic/application/vnd.android.axml/axml.xml
View file
Open in desktop
Binary file not shown.
4 changes: 2 additions & 2 deletions test/generate_tables_test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,8 @@ class Marcel::GenerateTablesTest < Marcel::TestCase

assert status.success?, errors
warning_lines = errors.lines
assert_equal "Skipped 59 unsupported magic rules\n", warning_lines.pop
assert_equal 113, warning_lines.size
assert_equal "Skipped 60 unsupported magic rules\n", warning_lines.pop
assert_equal 114, warning_lines.size
assert File.exist?(tables_path)
end
end
Expand Down
11 changes: 11 additions & 0 deletions test/magic_test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,17 @@ def read(*)
end
end

# Before Tika 4.0.0, the timestamped-data magic matched any 11-byte OID under the
# 1.2.840.113549 arc, so sibling CMS content types (compressedData, authData, ...) in
# 1.2.840.113549.1.9.16.1.* were misdetected as application/timestamped-data. The 4.0.0
# rules (carried in data/custom.xml with explicit match types) require the full
# id-ct-timestampedData OID ending in .31.
test "other CMS content types are not misdetected as timestamped-data" do
compressed_data = "\x30\x80\x06\x0B\x2A\x86\x48\x86\xF7\x0D\x01\x09\x10\x01\x09\xA0\x80".b

assert_equal "application/octet-stream", Marcel::MimeType.for(compressed_data)
end

test "add and remove type" do
Marcel::Magic.add('application/x-my-thing', extensions: 'mtg', parents: 'application/json')
Marcel::Magic.remove('application/x-my-thing')
Expand Down