Skip to content

A strict, exact and fast JSON pull cursor - #1

Merged
dan-garden merged 4 commits into
mainfrom
feat/397-first-cursor
Nov 8, 2026
Merged

dan-garden merged 4 commits into
mainfrom
feat/397-first-cursor

Conversation

@dan-garden

Copy link
Copy Markdown
Contributor

What changed

The first version of json-cursor: a strict JSON pull cursor. It reads keys, strings, numbers, booleans and null one at a time, straight from your bytes, so a caller builds its own types with no tree in between. It is part of FusionVault/streams#397, where it becomes the tokenizer under the streams JSON-to-Arrow decoder.

  • Strict. Every byte is checked against RFC 8259, in skipped values too. skip keeps its own stack, with a depth limit of 128 by default, so no nesting can overflow the call stack.
  • Exact. A Number keeps its text as written. as_i64 gives the exact whole value however it is written (7.0, 1e3), and as_f64 always the nearest double.
  • Fast. It reads every value faster than serde_json, sonic-rs, simd-json and jiter, on bids, events and GeoJSON. It validates fastest on bids and GeoJSON, and level with jiter on events, where jiter skips the UTF-8 check inside strings.
  • Light. No dependencies, no unsafe code, no_std with alloc, Rust 1.85 or later.

benchmarks/ races it against the other parsers and checks strictness and number exactness. Its results (benchmarks/results/*.csv) and charts back the README.

Results

Median CPU time per message, in nanoseconds, on one idle core of a Ryzen 9 5950X:

Message Job json-cursor best of the others
Bids, 248 bytes read every value 178 sonic-rs 199
Bids validate 111 serde_json 153
Events, 723 bytes read every value 282 jiter 372
Events validate 203 jiter 201
GeoJSON, 3,246 bytes read every value 4,160 sonic-rs 4,384
GeoJSON validate 2,119 serde_json 2,725

Strictness and exactness:

  • json-cursor gives RFC 8259's answer on all 26 cases of the strictness test. jiter and serde_json accept bad UTF-8 in the strings they validate, simd-json refuses 1e400 and a 30-digit integer, and sonic-rs crashes on a million nested arrays.
  • On a million random long decimals, serde_json's default float parse misses the nearest double 319,907 times. json-cursor, jiter, simd-json and sonic-rs never do.

How it is tested

  • RFC 8259 case by case, with each error's kind and position.
  • All 318 JSONTestSuite cases, vendored under its MIT license (see NOTICE).
  • A differential test against serde_json: two million random documents in CI, some of them damaged. Locally: 1,112,640 read alike, 887,072 refused by both, and 288 past a double's range, which serde_json refuses and json-cursor keeps as text.
  • as_f64 bit for bit against Rust's own parser, and as_i64 against a 128-bit reference, on every edge of the fast paths and 300,000 random numbers.
  • The bit tricks against plain byte loops, at every position.
  • Mutation testing: 32 rules broken one at a time, and a test failed for every one.
  • Rust 1.85, clippy with -D warnings, and the docs with -D warnings, all clean.

The last commit is a structure pass with no change in behaviour. Every function compiles to the same machine code as before it (4,008 instructions compared). The strictness, exactness and rounding results are byte-identical, and the random tests draw the same cases.

Docs checklist

Docs ship with the code, in the same pull request:

  • CHANGELOG.md has a line for this change (CI fails without one when src/ changes)
  • README updated where users will see it
  • Every new public item has a doc comment with an example. Every public item has a doc comment, but the examples live in the crate docs and the README, both run as doctests, not yet on Cursor, Number and Error themselves.
  • cargo test passes (the README examples are doctests)

🤖 Generated with Claude Code

dan-garden and others added 4 commits November 8, 2026 13:47
The cursor reads keys, strings, numbers, booleans and null one at a time,
from a &str or from bytes. `value` reads whatever comes next, `skip` passes
over anything with every byte checked, and `validate` checks a whole
document. Strings without escapes are borrowed; numbers keep their text,
with `as_i64` exact for any whole number and `as_f64` always the nearest
double.

Speed comes from reading eight bytes at a time, reading each number once
while its grammar is checked, one dispatch per value, a skip whose stack is
one register-held word, and rare paths kept out of line.

benchmarks/ races it against jiter, serde_json, simd-json and sonic-rs on
bids, events and GeoJSON, and checks strictness and number exactness; its
results and charts back the README.

Tested by RFC 8259 cases, all of JSONTestSuite, a differential test against
serde_json (two million documents in CI), Rust's own float parser, and 32
mutants, all caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The charts were linked through main, where they do not exist until this
branch merges, so the README showed none. They are relative now, as in
geo3d. The crate gets its own short docs page, since relative images do not
reach docs.rs, and the README's examples still run as doctests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing changes in behaviour or speed. Every json-cursor function compiles
to the same machine code as before (4,008 instructions compared), the
strictness, exactness and rounding results are byte-identical, the
benchmark messages hash the same, and the random tests draw the same cases
(the two-million-case run reads 1,112,640 alike and refuses 887,072, as
before).

- skip.rs moves up beside the modules it uses, so each module is one part
  of JSON, and lib.rs lists them with a line each.
- text::scan and number::scan become read, so "scan" names only the
  position module, which now holds the eight-byte load both use.
- The unit tests, the integration tests and the benchmarks each share one
  random generator, and the integration tests one case count.
- A benchmark shape is a named struct that makes its own messages.
- examples/speed.rs goes: benchmarks/ does its job, more carefully. The
  serde dev-dependency went with it, and so does a .gitignore that
  repeated the root's.
- The README says where to start and what to run before a pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dan-garden
dan-garden merged commit b37069f into main Nov 8, 2026
0 of 4 checks passed
@dan-garden
dan-garden deleted the feat/397-first-cursor branch November 8, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant