infernix-perplexity measures the causal perplexity produced by a v3 .ninfer artifact.
It uses the artifact's tokenizer, Text model, selected Main KV representation, final normalization,
and main output head. It is an offline evaluator, not a serving endpoint or a logits-export API.
Only Text weights and resources are loaded; Vision and speculative components are not required.
The repository includes ninfer-ppl-1m-v1, a fixed set of 16 independent UTF-8 streams covering
English reference text, English long-form text, Chinese reference text, and Infernix C++/CUDA code.
full selects all streams; --quick selects one stream from each domain.
./build/apps/infernix-perplexity models/qwen3_8_27b_nvfp4.ninfer \
--corpus eval/corpora/perplexity-1m/manifest.json \
--quick \
--kv-dtype fp8The default evaluation uses a 4,096-token context and a 2,048-token stride. Use --context and
--stride to change that protocol, or score one UTF-8 file with --text FILE. Each window is
prefilled in passes of --prefill-chunk tokens (a multiple of 128, default 16384 for
Qwen3.8-Flash-Next and 1024 otherwise), recorded in
report.json as prefill_chunk_tokens; INT8, NVFP4, FP8 and K8V4 KV round it down to whole
prompt-attention waves (896 tokens on Qwen3.8-27B). The pass size changes only the order of
rounding, but on artifacts with 4-bit activations that order matters more than it seems: on the
official Qwen3.8-27B NVFP4 artifact, chunks of 256-7168 tokens moved perplexity by -0.6 % to
+0.8 % against one pass per window, with no trend in the chunk size, and every chunked run sat at
a KL divergence of 0.025-0.033 from the one-pass run (about 4 % of positions change their top
token), as large as switching the KV cache between BF16 and INT8. The Q6 artifact, with 16-bit
activations, moved 15 times less (KL 0.0021). So treat a perplexity difference under about 1 %
between two NVFP4 builds or settings as noise unless it holds at several chunk sizes.
--reference runs the design reference, the quality gate's base arm: the lossless route with the
configuration's declared lossy steps applied exactly (the activation quantization the artifact permits,
FP4-Local's 4-bit experts with --expert-a4-activations, and the --kv-dtype codec: bf16, V stored as
FP16, while the other codecs are built); it implies --lossless, and report.json records it as
execution.reference (definition).
--lossless (Qwen3.5/3.6/3.8 and Qwen3.8-Flash-Next) runs the lossless route, the quality
reference's: every operation of the model's Hugging Face forward pass correctly rounded, so a score
depends on the window's tokens alone; one pass per window (give --prefill-chunk the context), BF16
KV, no fast path, FP32 logits (definition). Saved with
--save-top-tokens, it is the reference that --kl-reference measures other runs against, and
report.json records it as lossless. Model-level numerical quality
has the method and the recorded results. Qwen3.8-Flash-Next reads its n-gram volume from <model>.ngram unless
--ngram-volume FILE names it, as in infernix (--expert-stripe FILE as there too). Its windows
prefill through the generation path, the SSD expert tier included, keeping each position's final
residual; the LM head then runs over them in tiles of 64 positions with FP32 logits. The available Main
KV representations are bf16, int8, fp8, nvfp4, k8v4, vq2, and k4v2; unlike infernix and
infernix-serve, which default to bf16, infernix-perplexity defaults to fp8. int8 scores with the fast
prompt-attention kernel, and nvfp4 with its fast kernel over more than 768 visible keys, as
infernix-serve prefills them by default; --use-original-int8-prefill-kernel and
--use-original-nvfp4-prefill-kernel score them with the original kernels and require the matching
--kv-dtype. As in infernix-serve, the fast paths qualified for the artifact's configuration (its kernels and KV format) run
unless --reproducible selects the reproducible route; --lossless never runs a fast path. report.json records the kernels as original_int8_prefill_kernel and
original_nvfp4_prefill_kernel, and the fast paths that ran as fast_paths.
./build/apps/infernix-perplexity models/qwen3_8_27b.ninfer \
--text notes.txt \
--context 16384 --stride 8192 \
--kv-dtype int8Run ./build/apps/infernix-perplexity --help for the complete command surface. The evaluator loads
the model once, reads and tokenizes every selected stream before scoring, and writes readable
startup, corpus, scoring, and per-stream summaries to stderr. Interactive weight loading and
scoring use one transient progress line; redirected scoring emits persistent progress every ten
seconds. --log-level debug exposes internal startup and stream-begin detail. The final
domain/overall table remains product output on stdout; the independent full-precision machine
report is report.json under profiles/perplexity/ unless --output supplies an empty directory.
For KV-format comparisons, the recommended long-context profile is the full corpus with
--context 65536 --stride 32768 and without --quick.
Perplexity only sees each target token's probability, so a numerical change can move it in either
direction, and a few streams can dominate the result. A KL divergence comparison measures how far
each next-token distribution moved. First record a reference, normally with --kv-dtype bf16:
./build/apps/infernix-perplexity models/qwen3_8_27b.ninfer \
--corpus eval/corpora/perplexity-1m/manifest.json --context 65536 --stride 32768 \
--kv-dtype bf16 --save-top-tokens ref-bf16-64k.bin--save-top-tokens FILE writes each scored position's 64 most probable tokens (the most scoring
returns) and their log-probabilities (about 540 MB for the full corpus). Then score the setting
under test with the same corpus, mode, context and stride:
./build/apps/infernix-perplexity models/qwen3_8_27b.ninfer \
--corpus eval/corpora/perplexity-1m/manifest.json --context 65536 --stride 32768 \
--kv-dtype nvfp4 --kl-reference ref-bf16-64k.binThe run scores the reference's tokens at every position and reports KL(reference ‖ this run)
over those tokens plus one bucket for all other tokens. Merging the other tokens can only lower the
divergence, so the result is a lower bound of the full KL divergence; mean_reference_top_mass
shows how much probability they cover (95 % for 32 tokens on Qwen3.8-27B; references written
before 2026-10-12 hold 32). It also reports top-1 agreement (the same most probable token) and the
mean target-token NLL change. report.json gains a kl_divergence
object: overall mean, standard deviation and quantiles, and the same statistics by context length,
the number of tokens before the predicted one, in buckets [0,1K), [1K,2K), [2K,4K) and so on.
Each stream and each window carries its own kl_divergence (a window's has the overall statistics
without quantiles or buckets), so stream- and window-level paired comparisons are possible. The
evaluator refuses a reference recorded for a different corpus, mode, context or stride, or whose
windows hold different tokens.
--save-tokens FILE writes each scored window's input tokens, targets and fingerprint as JSON, so an
external evaluator can write a --kl-reference file for the same windows
(tools/flash_next/fp64_kl_reference.py does so with the FP64 reference forward of
Qwen3.8-Flash-Next).
--fast-path NAME (repeatable) also runs that fast path whether or not it is qualified for the
configuration; it exists to measure a candidate. Qwen3.8-Flash-Next artifacts score through the
same route as generation; --ngram-volume FILE names their n-gram volume when it is not
<artifact>.ngram.
--expert-a4-activations scores nvidia/Qwen3.8-Flash-Next-FP4-Local with 4-bit expert activations
(guide); report.json records it as
expert_a4_activations. tools/quality/fast_path_gate.py pairs an
exact-route run and a fast-path run window by window to decide the
numerical acceptance policy.
--rope-yarn-factor F accepts finite values in [1,4] (default 1, native RoPE unchanged).
This startup-fixed Engine override does not alter the artifact or converter. It only extends the
allowed ceiling: request longer scoring windows explicitly with --context; the default remains
4,096 tokens and the stride remains 2,048. Memory limits still apply. Long-context extrapolation
is not a guarantee of quality. Keep the factor fixed when comparing other numerical settings;
report.json records it as rope_yarn_factor in the execution configuration.
For a stream x[0..N), every token after x[0] is scored exactly once. A window [b,e) with target
suffix [s,e) contributes:
log p(x[i] | x[b], ..., x[i-1]) for i in [s,e)
Each window starts from empty State and Main KV, so history before b is deliberately excluded.
The reported metric is therefore fixed-window, truncated-context causal perplexity:
mean_nll = -sum(logprob) / scored_tokens
perplexity = exp(mean_nll)
The first window scores [1,min(context,N)). Each later window advances by stride targets while
retaining up to context-stride preceding tokens as local context. Streams never share history.
For a numerical comparison, keep the corpus, context, stride, and execution settings fixed except the variable being measured. Compare KV formats with the same artifact and weight formats with the same KV format.
The corpus name is a workload scale, not an exact token count. Exact input and scored-token counts are runtime results from the current artifact tokenizer and are recorded in each report. Reports contain unrounded NLL/PPL values for every window, stream, domain, and the token-weighted overall aggregate.
The schema-v3 report identifies the artifact's architecture, public name, actual weight formats and prefill signature alongside the workload and numerical results.