datavalue-rs

A bump-allocated JSON value type with a built-in zero-copy parser and serde_json::Value-style ergonomics.

---

datavalue-rs provides DataValue<'a>, an arena-allocated JSON DOM designed for high-throughput request pipelines, rule engines, and data-processing workloads where per-node heap allocations dominate execution time. Every composite payload (unescaped strings, arrays, object key-value slices) lives inside a bumpalo::Bump arena that can be reset in $O(1)$ between batches, while OwnedDataValue provides a 1:1 heap-owned counterpart for values that need to outlive a single arena scope.

Quick Start

Install via Cargo:

cargo add datavalue-rs bumpalo

Parse, traverse, and emit JSON with zero per-value heap allocations:

use bumpalo::Bump;
use datavalue_rs::DataValue;

let arena = Bump::new();
let v = DataValue::from_str(r#"{"name":"alice","ages":[30,31]}"#, &arena).unwrap();

// Ergonomic indexing — missing keys or out-of-bounds indices evaluate to Null.
assert_eq!(v["name"].as_str(), Some("alice"));
assert_eq!(v["ages"][1].as_i64(), Some(31));
assert!(v["missing"]["nested"].is_null());

// Render back to JSON (into a String, an arena &str, a Vec<u8>, or via Display).
assert_eq!(v.to_json_string(), r#"{"name":"alice","ages":[30,31]}"#);
println!("{}", v.pretty()); // 2-space indented JSON, matching serde_json::to_string_pretty

Building Values Inline

Use the owned_json! macro for serde_json::json!-style literal construction:

use datavalue_rs::{owned_json, OwnedDataValue};

let count = 2;
let user: OwnedDataValue = owned_json!({
    "name": "alice",
    "ages": [30, 31],
    "next_count": (count + 1),
    "active": true,
    "tags": null,
});

assert_eq!(user["ages"][1].as_i64(), Some(31));
assert_eq!(user["next_count"].as_i64(), Some(3));

Key Features

• Arena-Allocated (Copy DOM Nodes) — DataValue<'a> is a 24-byte Copy enum on 64-bit targets. One Bump holds the entire value tree; reset the arena between batches for amortized zero-allocation parsing.
• Zero-Copy Strings — JSON string literals without escape sequences borrow directly from the input &str without copying bytes into the arena.
• Native Integer Fast Path — NumberValue distinguishes Integer(i64) from Float(f64) (parsed via fast_float2), preserving exact integer representation and arithmetic until float promotion is needed.
• Opt-In Strict Bounded Parsing — DataValue::from_str_with_options and OwnedDataValue::from_json_with_options enforce configurable byte, depth, and node ceilings, duplicate-key rejection, and numeric policies (NativeFinite or SafeIntegers) with zero overhead on the default parser path.
• serde_json::Value-Style Ergonomics — Chained Index (v["a"][0]["b"]), get(), as_*() / is_*(), members(), and entries().
• Heap-Owned Twin (OwnedDataValue) — Variant-for-variant identical to DataValue<'a>, supporting in-place mutation, From conversions for standard Rust types, and fast conversion back and forth (to_owned(), to_arena(), and zero-copy string borrowing via view_in()).
• Optional serde & serde_json Bridges — Direct Serialize / Deserialize support, DataValueSeed (DeserializeSeed) for arena deserialization from any Serde format (MsgPack, CBOR, etc.), and direct AST conversion to/from serde_json::Value.
• Optional datetime & tensor Extensions — First-class DateTime / Duration (chrono-backed) and zero-copy n-dimensional typed Tensor payloads (f32, i64, f16, bf16, etc.).

Performance

Highlights on the 631 KB twitter.json fixture from the serde-rs JSON benchmark suite (Apple Silicon aarch64, release build, single thread, Criterion medians):

Workload	datavalue-rs	serde_json	Speedup
Parse (from_str)	1.68 GiB/s	0.47 GiB/s	3.5×
Access (walk all statuses[])	1.91 µs	6.15 µs	3.2×
Serialize (Display / write_json_into)	1.69 GiB/s	1.43 GiB/s	1.2×
Clone + Mutate (OwnedDataValue)	2.23 GiB/s	1.60 GiB/s	1.4×

Full cross-library comparisons against serde_json, simd-json, sonic-rs, and json-rust across twitter.json, citm_catalog.json, and canada.json are documented in BENCHMARKS.md.

Usage Guide

Arena (DataValue) vs. Owned (OwnedDataValue)

Use DataValue<'a> for request-scoped parsing, evaluation, and serialization. Use OwnedDataValue when values must outlive an arena (caches, channels, shared state) or when mutating trees in place.

use bumpalo::Bump;
use datavalue_rs::{DataValue, OwnedDataValue};

// 1. Parse in the arena, then promote to OwnedDataValue to outlive the arena.
let arena = Bump::new();
let parsed = DataValue::from_str(r#"{"x":42,"label":"ok"}"#, &arena).unwrap();
let owned: OwnedDataValue = parsed.to_owned();
drop(arena); // `owned` is completely independent of `arena`.

assert_eq!(owned["x"].as_i64(), Some(42));

// 2. Or parse directly into OwnedDataValue.
let owned2: OwnedDataValue = r#"{"x":42,"label":"ok"}"#.parse().unwrap();
assert_eq!(owned, owned2);

// 3. Rehydrate into a fresh arena when entering an arena-based hot path:
//    - `to_arena` deep-copies strings into the arena (independent lifetimes).
//    - `view_in` borrows strings/keys from `owned2` and allocates only container slices.
let batch_arena = Bump::new();
let copied: DataValue<'_> = owned2.to_arena(&batch_arena);
let viewed: DataValue<'_> = owned2.view_in(&batch_arena);
assert_eq!(copied, viewed);

Strict Bounded Parsing for Untrusted Input

When accepting untrusted JSON at service boundaries, use ParseOptions::strict to enforce resource limits, reject duplicate object keys before parsing their values, and select an explicit numeric policy:

use bumpalo::Bump;
use datavalue_rs::{
    DataValue, NumberPolicy, ParseLimits, ParseOptions, StrictParseErrorKind,
};

let limits = ParseLimits {
    max_input_bytes: 64 * 1024, // 64 KiB max payload
    max_depth: 32,              // max nesting depth
    max_nodes: 4_096,           // max total JSON values
};

let options = ParseOptions::strict(limits)
    .with_number_policy(NumberPolicy::SafeIntegers) // restrict integers to ±(2^53 - 1)
    .build()
    .unwrap();

let arena = Bump::new();
let ok = DataValue::from_str_with_options(r#"{"id":100,"ok":true}"#, &arena, &options).unwrap();
assert_eq!(ok["id"].as_i64(), Some(100));

// Duplicate keys are rejected deterministically (including escaped spellings).
let dup = DataValue::from_str_with_options(r#"{"k":1,"\u006b":2}"#, &arena, &options);
assert!(matches!(
    dup.unwrap_err().kind(),
    StrictParseErrorKind::DuplicateObjectKey
));

Serde & serde_json Interop

Enable the serde or serde_json Cargo features to integrate with the broader Rust serialization ecosystem:

use bumpalo::Bump;
use datavalue_rs::{DataValue, DataValueSeed, OwnedDataValue};
use serde::de::DeserializeSeed;

let arena = Bump::new();

// Deserialize any serde Deserializer directly into the arena via DataValueSeed:
let raw = r#"{"service":"auth","retries":3}"#;
let mut de = serde_json::Deserializer::from_str(raw);
let arena_val: DataValue<'_> = DataValueSeed::new(&arena).deserialize(&mut de).unwrap();
assert_eq!(arena_val["service"].as_str(), Some("auth"));

// OwnedDataValue implements Serialize and Deserialize directly:
let owned_val: OwnedDataValue = serde_json::from_str(raw).unwrap();
assert_eq!(arena_val.to_owned(), owned_val);

// With feature `serde_json`, convert directly to/from `serde_json::Value` without string round-trips:
let sj_val: serde_json::Value = arena_val.to_serde_value();
let back_in_arena = DataValue::from_serde_value_in(&sj_val, &arena);
assert_eq!(arena_val, back_in_arena);

Tensor Extension (feature = "tensor")

The tensor feature adds a zero-copy n-dimensional typed buffer variant (DataTensor<'a> / OwnedDataTensor) so JSON trees can carry numeric tensors between operators without expanding them into millions of DataValue::Number array nodes:

use bumpalo::Bump;
use datavalue_rs::{DType, DataTensor, DataValue};

let arena = Bump::new();

// Zero-copy construction over an aligned slice:
let weights: &[f32] = arena.alloc_slice_copy(&[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
let shape = arena.alloc_slice_copy(&[2, 3]);
let tensor = DataTensor::from_slice(shape, weights).unwrap();

assert_eq!(tensor.dtype(), DType::F32);
assert_eq!(tensor.shape(), &[2, 3]);
assert_eq!(tensor.as_slice::<f32>().unwrap(), weights);

// Embed in a DataValue tree and emit in the canonical `{"tensor": ...}` wire format:
let val = DataValue::tensor_in(tensor, &arena);
let wire = val.to_json_string();
assert!(wire.starts_with(r#"{"tensor":{"dtype":"f32","shape":[2,3],"data":"#));

// Upgrade the wire object back into a validated DataTensor at operator boundaries:
let parsed_obj = DataValue::from_str(&wire, &arena).unwrap();
let decoded = DataTensor::from_json_value_in(&parsed_obj, &arena).unwrap();
assert_eq!(decoded, tensor);

Cargo Features

All optional features are disabled by default so the core library stays minimal (bumpalo, fast-float2, itoa, ryu).

Feature	Default	Description
serde	off	Implements Serialize for DataValue and OwnedDataValue, Deserialize for OwnedDataValue, and DataValueSeed (DeserializeSeed) for arena-allocated deserialization.
serde_json	off	Implies serde. Adds direct DOM conversions between DataValue / OwnedDataValue and serde_json::Value (from_serde_value_in, to_serde_value, From / Into).
datetime	off	Adds DateTime(DataDateTime) and Duration(DataDuration) variants backed by chrono.
tensor	off	Adds Tensor(&DataTensor) / Tensor(Arc<OwnedDataTensor>) variants, DType, zero-copy typed views (as_slice::<T>()), nested array ↔ tensor conversion, and {"tensor": …} wire format encoding/decoding. Zero external dependencies; little-endian targets only.
tensor-half	off	Implies tensor. Enables native half::f16 and half::bf16 element views and nested-array conversions, re-exporting datavalue_rs::half.

Architecture & Semantics

• Strings: Unescaped JSON strings borrow &'a str slices directly from the input buffer. Strings with escape sequences copy contiguous unescaped runs into the Bump arena and decode escapes in a single pass.
• Numbers: NumberValue stores Integer(i64) or Float(f64). Integer literals stay on the i64 path unless they overflow i64 or include a fraction/exponent (e.g., 2.0 parses as Float(2.0) and formats back as 2.0). Cross-representation equality and ordering (Integer(5) == Float(5.0)) are mathematically exact across the full i64 / f64 range without precision-losing casts.
• Objects & Lookup: Objects are stored as contiguous slices (&'a [(&'a str, DataValue<'a>)] / Vec<(String, OwnedDataValue)>), preserving insertion order and beating hash-table overhead on typical JSON object sizes. Lookup is a linear scan returning the first matching key.
• Order-Insensitive Equality: Object comparison (==) is order-insensitive across distinct keys (matching serde_json::Value) while comparing duplicate keys in occurrence order so == remains a true equivalence relation and agrees with Index lookup. Because f64::NAN != f64::NAN, neither value type implements Eq.
• No Implicit Coercion: Accessors such as as_i64(), as_str(), and as_bool() return None if the underlying variant does not match. Truthiness and cross-type coercion belong in downstream evaluation engines (such as datalogic-rs).
• Explicit Domain Upgrades (DateTime & Tensor): Standard JSON has no native datetime or tensor syntax, so the parser never produces DateTime, Duration, or Tensor variants implicitly. Consumers upgrade strings via DataDateTime::parse / DataDuration::parse and tagged objects via DataTensor::from_json_value_in at operator boundaries.

Development & Contributing

Contributions, bug reports, and benchmark improvements are welcome!

Prerequisites

• Rust 1.98+ (pinned as rust-version in Cargo.toml).

Running Checks Locally

Before opening a pull request, run the test and lint matrix enforced by CI:

# Formatting
cargo fmt --all -- --check

# Clippy across feature combinations
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo clippy --all-targets --features tensor -- -D warnings
cargo clippy --all-targets --features tensor-half -- -D warnings
cargo clippy --all-targets --features serde -- -D warnings
cargo clippy --all-targets --features datetime -- -D warnings

# Unit, integration, and documentation tests
cargo test
cargo test --all-features
cargo test --features tensor
cargo test --features tensor-half
cargo test --features serde
cargo test --features datetime

# Documentation build (verifies intra-doc links)
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

# Benchmarks (smoke run)
cargo bench --bench compare --features serde_json -- --quick

Related Projects

• datalogic-rs — High-performance JSONLogic evaluation engine built on datavalue-rs.

About Plasmatic

Created by Plasmatic, building open-source tools for financial infrastructure and high-throughput data processing.

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.