Repository navigation
Fix non-BMP characters in JNI strings (#10) + regression test, CI on develop - #11
Merged
Merged
Conversation
Release 0.6.0
`GetStringUTFChars` returns modified UTF-8, which encodes characters outside the BMP as a CESU-8 surrogate pair. Decoding that buffer as plain UTF-8 replaced every such character with U+FFFD, so emoji and CJK extension ideographs were lost from extracted text and metadata.
Characters outside the BMP were decoded as plain UTF-8 on the way out of JNI, so extracted text and metadata came back as U+FFFD. Cover a UTF-8 text file, an HTML title, and a DOCX body plus dc:title. Co-authored-by: Titusz <tp@py7.de>
Pushes to develop already ran this workflow. Pull requests against develop now use the same jobs as pull requests against main. Co-authored-by: Titusz <tp@py7.de>
cargo test for iscc-tika-core runs in its own job, in parallel with the wheel build. Cargo already puts the native library directory on LD_LIBRARY_PATH. The Graal image is produced by setup_tika_native.py, the same path local builds use, and cached so the next run can reuse it. Co-authored-by: Titusz <tp@py7.de>
Co-authored-by: Titusz <tp@py7.de>
Modified UTF-8 from GetStringUTFChars cannot be decoded at all once a string holds an unpaired surrogate, so the jni crate fell back to lossy decoding of the whole string and garbled every supplementary character in it. Reading UTF-16 units replaces only the unpaired surrogate. Metadata values are looked up with the original Java key instead of a re-encoded copy of the decoded key, which did not round-trip for keys holding an unpaired surrogate.
The 500,000 default cut off the text of most books. Text cut off at the limit is now flagged with Tika's X-TIKA:EXCEPTION:write_limit_reached metadata key (exported as WRITE_LIMIT_REACHED), and a cut that splits a surrogate pair drops the dangling high surrogate. Any negative limit disables truncation; Tika itself only accepts -1.
File hygiene hooks no longer rewrite test_files/, whose BOM and missing final newline are part of the fixtures. The Rust CI job saves the native library cache before running tests, so a failing test keeps the build, and moves to actions/cache v6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings @CHISANW's non-BMP JNI decoding fix onto
developfor the next release, with a regression test and CI fordevelop.Merging this pull request includes their commit (
198e4a1, "Decode Java strings as modified UTF-8"), so GitHub will show #10 as merged.Changes from #10
iscc-tika-core/src/tika/jni_utils.rs: decode Java strings with thejnicrate's modified UTF-8 conversion (String::from(JavaStr)) instead of plainto_string_lossy().CHANGELOG.md: note under Unreleased.Regression test
iscc-tika-core/tests/non_bmp_tests.rstest_files/documents/non-bmp.txt,non-bmp.html,non-bmp.docxdc:title(HTML and DOCX, including metadata-only extraction). Fails onmain/ currentdevelopwithout the JNI fix.CI
.github/workflows/ci.ymlalready ran on pushes todevelop. Pull requests that targetdevelopnow run the same workflow as pull requests that targetmain.Not merged here.