Repository navigation
Conversation
Release 0.6.0
`GetStringUTFChars` returns modified UTF-8, which encodes characters outside the BMP as a CESU-8 surrogate pair. Decoding that buffer as plain UTF-8 replaced every such character with U+FFFD, so emoji and CJK extension ideographs were lost from extracted text and metadata.
titusz
added a commit
that referenced
this pull request
Oct 9, 2026
…uncation (#11) - Read Java strings as UTF-16 code units, so characters outside the Basic Multilingual Plane survive extraction (previously each became six U+FFFD) and an unpaired surrogate becomes a single U+FFFD. - Look up metadata values with the original Java key, so keys holding an unpaired surrogate keep their values. - Raise the default extract_string_max_length from 500,000 to 10,000,000; any negative value disables the limit. - Flag text cut off at the limit with Tika's X-TIKA:EXCEPTION:write_limit_reached metadata key (exported as WRITE_LIMIT_REACHED) and drop a dangling high surrogate at the cut. - Add Rust and Python regression tests, run the Rust test suite in CI, run CI on pull requests to develop, and keep test fixtures out of the file hygiene hooks. Supersedes #10. Co-authored-by: CHISANW <keuye06380618@gmail.com>
Member
|
Thank you @CHISANW for finding this and for the fix! Your change was the starting point for #11, which is now merged into #11 builds on your modified UTF-8 approach and goes one step further: Java strings are now read as UTF-16 code units, so an unpaired surrogate (for example from text cut at the length limit) costs a single U+FFFD instead of garbling the whole string. It also adds regression tests for Rust and Python. The fix will ship in the next release. Closing this PR in favor of #11. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix incorrect decoding of Java strings containing Unicode characters outside the BMP (e.g., emoji).
Changes
to_string_lossy()withString::from()to correctly decode JNI modified UTF-8.U+FFFD.Testing