[Lynnbb88]

Issue

TIKA-4408

Python scripts starting with #!/usr/bin/env python or #!/usr/bin/env python3 are detected as application/x-sh, including when a .py filename hint is supplied. The existing magic rule looks for /bin/env python at offset 1, where these inputs contain !.

Changes

Replace that rule with two literal prefixes anchored at offset 0:

<match value="#!/usr/bin/env python" type="string" offset="0"/>
<match value="#! /usr/bin/env python" type="string" offset="0"/>

This follows the paired no-space/single-space convention of the neighboring direct-path Python rules and preserves prefix matching for python, python3, and version suffixes such as python3.11.

Add 18 regression checks covering content-only detection, .py filename hints, direct Python paths, Shell controls, filename-only detection, and file-start anchoring. Add a changelog entry for the unreleased version. No new dependencies or public API changes.

General whitespace handling, env options before the interpreter (such as -S), and stricter interpreter-token boundaries are outside this change.

Validation

Fresh local checks on macOS with JDK 21, based on Apache main at 9f8b48425fea3630020924f6ae96e98eabb9ce12:

• With the new tests and the original production rule: 18 tests, 7 failures, 0 errors.
• With the fix: all 18 Python regression checks passed.
• Full tika-core unit suite: 1,418 tests reported, 0 failures, 0 errors, 2 skipped.
• Alternate Xerces provider: 12 tests reported, 0 failures, 0 errors, 2 skipped.
• TikaIT: 1 test passed; annotation processor: all 8 tests passed.
• Checkstyle, Spotless, forbidden-API checks, and Apache RAT passed; BUILD SUCCESS.

./mvnw -B -ntp clean install -pl tika-core -am -Ppedantic \
  -Dtika.test.locale=random -Dmaven.repo.local="$PWD/.local_m2_repo"

The negative control used the same command with -Dtest=PythonShebangDetectionTest -Dsurefire.failIfNoSpecifiedTests=false. These checks cover core and its reactor prerequisites; the full repository build has not been run.

AI assistance

OpenAI Codex assisted with investigation, implementation, testing, and PR preparation. The commit includes a Generated-by trailer.

[Copilot]

🟢 Approval recommended

No active review issues are reported for the Python env-shebang MIME detection fix.

No open issues 🎉

What changed in this PR

Fixes TIKA-4408 by updating Tika’s MIME detection rules so Python scripts using /usr/bin/env python shebangs are recognized as text/x-python rather than shell scripts.

Changes:

• Replaces the incorrect offset-based rule with file-start prefixes supporting no-space and single-space shebang forms.
• Adds 18 regression checks covering Python variants, filename hints, shell controls, and file-start anchoring.
• Documents the fix in the unreleased changelog.

File	Description
tika-core/​src/​test/​java/​org/​apache/​tika/​mime/​PythonShebangDetectionTest.java	Adds Python shebang detection regression tests.
tika-core/​src/​main/​resources/​org/​apache/​tika/​mime/​tika-mimetypes.xml	Corrects Python env shebang magic rules.
CHANGES.txt	Records the TIKA-4408 detection fix.

_{Review effort: Balanced}