Skip to content

[TIKA-4408] Fix Python env shebang detection - #4

Merged
Lynnbb88 merged 5 commits into
mainfrom
a5/tika-4408-python-shebang
Nov 1, 2026
Merged

Lynnbb88 merged 5 commits into
mainfrom
a5/tika-4408-python-shebang

Conversation

@Lynnbb88

@Lynnbb88 Lynnbb88 commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Standard Python env shebangs such as #!/usr/bin/env python3 were detected as application/x-sh, even with a .py filename hint. The original magic rule looked for /bin/env python at fixed offset 1.

Replace it with two literal file-start prefixes:

<match value="#!/usr/bin/env python" type="string" offset="0"/>
<match value="#! /usr/bin/env python" type="string" offset="0"/>

The single-space form follows the existing direct-path Python rule style: neighboring /bin/python, /usr/bin/python, and /usr/local/bin/python rules already provide paired no-space and single-space literals.

Prefix matching retains python, python3, and version suffixes such as python3.11. Include 18 checks covering both detection modes, direct Python paths, Shell controls, filename-only detection and file-start anchoring, plus a changelog entry. General whitespace/Tab handling, regex, env -S, and interpreter-token boundary changes are outside scope.

Current branch — 1 November 2026

8152ff5ee merges group main at 7a609bdb58851a8825e1cce3b5fc014fff174979 into this feature branch. The CHANGES.txt conflict was resolved by keeping both the original TIKA-4408 entry and the TIKA-4694 entry already on main. The Python MIME rules and regression-test file are unchanged. This group PR was subsequently merged on 1 November 2026 as f3a9d0ede. Its final diff contained three files, 80 additions and 1 deletion.

Validation — 1 November 2026

Fresh Java 21 verification of the resolved merge tree committed as 8152ff5ee:

  • Final Python suite: all 18 checks passed.
  • Full tika-core: 1,377 tests, 0 failures/errors, 2 skipped.
  • Alternate Xerces provider: 12 tests, 0 failures/errors, 2 skipped.
  • TikaIT and all 8 annotation-processor tests passed.
  • Checkstyle, Spotless, forbidden-API checks and Apache RAT passed; BUILD SUCCESS.
./mvnw -B -ntp -o clean install -pl tika-core -am -Ppedantic \
  -Dtika.test.locale=random -Dmaven.repo.local="$PWD/.local_m2_repo"

Recorded group-branch negative controls from 25 October 2026 on 471afbe222, with the same unchanged 18-check suite (not repeated in the 1 November group-branch run):

  • Original offset-1 rule: 7 failures, 0 errors.
  • Fixed no-space rule only: 1 failure, 0 errors — the single-space content-only case. A .py hint masks that missing rule.

These are local core/reactor results, not whole-repository validation or remote CI.

Review

The Python MIME rules and regression-test file remain byte-identical to the implementation reviewed and approved by Kyubin on cefa669. The original TIKA-4408 changelog text is preserved. That recorded approval covers the unchanged Python change; it does not imply review of the new merge commit or all intervening main changes. The full repository tree is no longer identical to the old approved tree.

Related group issue: #2; upstream ticket: TIKA-4408. This group PR is merged. The change has been submitted to Apache as PR #3473, currently open and awaiting maintainer review.

Course review record

Under the A5 course requirements and corrected group review policy, internal peer review and upstream submission are separate stages. Before an Apache PR, discuss the change with the group, have at least the majority of the group look at it, and have the tutor look at the proposed PR. Project/issue-selection approval is separate from review of a proposed PR. The linked technical reviews establish only their recorded scope and dates; they do not by themselves verify every pre-submission step. This description update does not backdate or certify an unrecorded approval.

Apache submission — 1 November 2026

apache/tika#3473 contains a single commit, 435181f3a, on Apache main 9f8b48425fea3630020924f6ae96e98eabb9ce12. Only the three-file TIKA-4408 patch is included; group coordination files, coursework documents and merge history are excluded.

Fresh validation on the Apache base: original rule 18 tests / 7 failures / 0 errors; fixed Python suite 18 passed; full core 1,418 reported / 0 failures/errors / 2 skipped; Xerces 12 reported / 0 failures/errors / 2 skipped; TikaIT, all 8 annotation-processor tests and pedantic quality checks passed. The command above was used without -o. Full-repository validation and successful upstream CI are not claimed.

Upstream submission and status update prepared with Codex for @Lynnbb88 (Jiajun Li).

@Lynnbb88 Lynnbb88 self-assigned this Oct 17, 2026
@u8026876

Copy link
Copy Markdown
Owner

Codex-assisted technical review of cefa669.

No blocking correctness issue found in this three-file change. The new rules
match the env-Python shebangs at file offset zero, preserve the existing
direct-Python rules, and include shell and leading-newline controls. The
changelog entry and new test file's ASF header are present. The commit already
has a Generated-by: Codex trailer.

Independent local validation on Windows / JDK 21:

  • clean install -pl tika-core -am -Ppedantic -Dtika.test.locale=random passed.
  • Core unit run: 1,377 reported, 0 failures/errors, 11 skipped in this environment.
  • All 18 PythonShebangDetectionTest cases passed.
  • Alternate XML-provider checks: 12 reported, 0 failures/errors, 2 skipped.
  • TikaIT and all 8 annotation-processor tests passed; style and RAT passed.
  • Negative control: reverting only the MIME XML rule, while keeping the new
    tests, produced 7 failures out of 18 cases, with no test errors. The PR's
    production rule was restored afterwards.

The local evidence supports the regression claim. Before an Apache PR, carry
only the intended three-file change onto a recent upstream base, and complete
the majority-group and tutor review required by A5. These are AI-assisted
technical checks; they do not establish additional human reviewers or remote CI.

@u8026876 u8026876 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found in the scoped review of cefa669.

This was a Codex-assisted walkthrough of the MIME-rule correction and regression tests. I reviewed the explained offset mismatch, expected Python/Shell behaviour, why content-only checks matter, and the supplied test results.

Validation performed by Codex:

  • All 18 PythonShebangDetectionTest checks passed. Reverting only the production rule caused 7 of those checks to fail, with no test errors.
  • The earlier core clean install passed: 1,377 reported, 0 failures/errors, 11 skipped; XML-provider checks, TikaIT, annotation-processor tests, style and RAT also passed.
  • Two additional content-only probes used the real Tika engine with the exact original and fixed MIME registries:
Input Original rules Fixed rules
#!/usr/bin/env python3.12 with a Python body application/x-sh text/x-python
Ordinary text x/bin/env python text/x-python text/plain

Both fixed results match the intended behaviour. The second probe also demonstrates removal of an accidental offset-1 match.

The extra probes were isolated checks, not a fresh full-suite run. Test execution and this note were assisted by Codex; remote CI and whole-repository validation are not claimed. Complete the remaining group and tutor review before upstream submission.

Keep a single file-start match for #!/usr/bin/env python3, including
version suffixes such as python3.11. Remove the single-space env rule
and narrow the regression matrix and changelog to the selected scope.

Generated-by: Codex
@Lynnbb88 Lynnbb88 changed the title [TIKA-4408] Fix Python env shebang MIME detection [TIKA-4408] Detect Python 3 env shebangs Oct 25, 2026
Use #!/usr/bin/env python at file offset zero to cover python and its
version suffixes without adding whitespace handling. Restore the bare
env python regression in both detection modes and align the changelog.

Generated-by: Codex
@Lynnbb88 Lynnbb88 changed the title [TIKA-4408] Detect Python 3 env shebangs [TIKA-4408] Fix unspaced Python env shebang detection Oct 25, 2026
Follow the existing direct-path Python rule style with paired no-space
and single-space literals at offset zero. Restore the single-space
regression case in both detection modes and the original changelog text.

Generated-by: Codex
@Lynnbb88 Lynnbb88 changed the title [TIKA-4408] Fix unspaced Python env shebang detection [TIKA-4408] Fix Python env shebang detection Oct 25, 2026
Keep both the TIKA-4408 and TIKA-4694 changelog entries. The Python MIME rules and regression tests are unchanged.

Generated-by: Codex
@Lynnbb88
Lynnbb88 merged commit f3a9d0e into main Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants