Repository navigation
A strict, exact and fast JSON pull cursor - #1
Merged
Merged
Conversation
The cursor reads keys, strings, numbers, booleans and null one at a time, from a &str or from bytes. `value` reads whatever comes next, `skip` passes over anything with every byte checked, and `validate` checks a whole document. Strings without escapes are borrowed; numbers keep their text, with `as_i64` exact for any whole number and `as_f64` always the nearest double. Speed comes from reading eight bytes at a time, reading each number once while its grammar is checked, one dispatch per value, a skip whose stack is one register-held word, and rare paths kept out of line. benchmarks/ races it against jiter, serde_json, simd-json and sonic-rs on bids, events and GeoJSON, and checks strictness and number exactness; its results and charts back the README. Tested by RFC 8259 cases, all of JSONTestSuite, a differential test against serde_json (two million documents in CI), Rust's own float parser, and 32 mutants, all caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The charts were linked through main, where they do not exist until this branch merges, so the README showed none. They are relative now, as in geo3d. The crate gets its own short docs page, since relative images do not reach docs.rs, and the README's examples still run as doctests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing changes in behaviour or speed. Every json-cursor function compiles to the same machine code as before (4,008 instructions compared), the strictness, exactness and rounding results are byte-identical, the benchmark messages hash the same, and the random tests draw the same cases (the two-million-case run reads 1,112,640 alike and refuses 887,072, as before). - skip.rs moves up beside the modules it uses, so each module is one part of JSON, and lib.rs lists them with a line each. - text::scan and number::scan become read, so "scan" names only the position module, which now holds the eight-byte load both use. - The unit tests, the integration tests and the benchmarks each share one random generator, and the integration tests one case count. - A benchmark shape is a named struct that makes its own messages. - examples/speed.rs goes: benchmarks/ does its job, more carefully. The serde dev-dependency went with it, and so does a .gitignore that repeated the root's. - The README says where to start and what to run before a pull request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The first version of json-cursor: a strict JSON pull cursor. It reads keys, strings, numbers, booleans and
nullone at a time, straight from your bytes, so a caller builds its own types with no tree in between. It is part of FusionVault/streams#397, where it becomes the tokenizer under the streams JSON-to-Arrow decoder.skipkeeps its own stack, with a depth limit of 128 by default, so no nesting can overflow the call stack.Numberkeeps its text as written.as_i64gives the exact whole value however it is written (7.0,1e3), andas_f64always the nearest double.unsafecode,no_stdwithalloc, Rust 1.85 or later.benchmarks/races it against the other parsers and checks strictness and number exactness. Its results (benchmarks/results/*.csv) and charts back the README.Results
Median CPU time per message, in nanoseconds, on one idle core of a Ryzen 9 5950X:
Strictness and exactness:
1e400and a 30-digit integer, and sonic-rs crashes on a million nested arrays.How it is tested
as_f64bit for bit against Rust's own parser, andas_i64against a 128-bit reference, on every edge of the fast paths and 300,000 random numbers.-D warnings, and the docs with-D warnings, all clean.The last commit is a structure pass with no change in behaviour. Every function compiles to the same machine code as before it (4,008 instructions compared). The strictness, exactness and rounding results are byte-identical, and the random tests draw the same cases.
Docs checklist
Docs ship with the code, in the same pull request:
CHANGELOG.mdhas a line for this change (CI fails without one whensrc/changes)Cursor,NumberandErrorthemselves.cargo testpasses (the README examples are doctests)🤖 Generated with Claude Code