build/apps/infernix runs one request against one v3 .infernix (or .ninfer) artifact. Build Infernix and
download an artifact using the project README before following this guide.
The examples use Qwen3.8-27B NVFP4 with FP8 KV storage.
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Summarize the difference between prefill and decode." \
--max-context 32768 \
--max-new 8192 \
--kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--lm-head-draftExactly one of --prompt and --messages is required. The CLI normally omits --kv-capacity, so
the shared Main Text KV pool follows the example's 32,768-token --max-context.
Answer content is streamed to stdout. Human-readable startup milestones and runtime errors are
written to stderr without service timestamps. Reasoning and the CLI result report (timings,
throughput, GPU memory, token IDs when requested, and speculative-decoding statistics) also use
stderr as unprefixed product output, so stdout can be redirected independently. On a terminal,
weight materialization is one transient progress line followed by a compact Engine-ready summary.
Redirected stderr contains persistent readable progress for long loads and no carriage returns or
ANSI escapes. --log-level debug exposes every startup phase. FFmpeg's media-decoding messages are
records prefixed media |: FFmpeg errors are warnings and everything milder is debug. Option and
local prompt/message input failures remain direct command diagnostics:
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Return one sentence." \
--max-context 4096 \
--max-new 64 \
--kv-dtype fp8 \
> answer.txt 2> run.log--chat-template FILE overrides the artifact's built-in template with a local Jinja file.
Changes to the file take effect after restarting Infernix:
./build/apps/infernix models/qwen3_8_27b.ninfer \
--chat-template tools/chat_templates/qwen3_8.jinja --prompt "Hello"--grammar-file FILE constrains the answer with a GBNF grammar whose entry rule is root:
printf 'root ::= "yes" | "no"\n' > answer.gbnf
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Is 17 prime?" --no-thinking --grammar-file answer.gbnf --max-new 64--choice TEXT selects a literal candidate; repeat the flag to supply the candidate set.
--regex PATTERN constrains the complete answer to a regular expression:
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Classify this review: the service was excellent." --no-thinking \
--choice positive --choice neutral --choice negative
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Return ticket TASK-0042." --no-thinking --regex '(BUG|TASK)-[0-9]{4}'Candidates preserve exact text; --choice '' explicitly permits empty content. --regex ''
permits only empty content. See the language contract
for supported regex syntax.
--json-object constrains the answer to a JSON object. --json-schema-file FILE applies a JSON
Schema. All output constraint options are mutually exclusive:
printf '%s\n' '{"type":"object","properties":{"answer":{"type":"integer"}},"required":["answer"],"additionalProperties":false}' > answer.schema.json
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Return the answer to 6 times 7 as JSON." --no-thinking \
--json-schema-file answer.schema.json --max-new 64GBNF supports recursive rules, Unicode character classes and repetition. All constraint modes work with
ordinary decoding, MTP, DFlash and DFlash2, and on Qwen3.8-Flash-Next with its MTP and n-gram
copy rounds. Thinking may precede the constrained answer; an output
limit or cancellation can leave it incomplete. JSON modes can accompany tools supplied in messages:
the answer is either JSON or a tool-call sequence. GBNF, choice and regex require no active tools.
Constraints reject custom stops and --raw-output. JSON uses compact separators and declared property order. Describe
the desired content in the prompt; the schema is not added to it automatically. See the
supported schema subset.
Omitted thinking and effort options use the selected template's defaults. --no-thinking or
--reasoning-effort none requests disabled thinking; other effort values cannot be combined with
--no-thinking. The template interprets the selected effort; a value the template rejects renders
as the nearest one it accepts (see serving). --greedy selects exact argmax
decoding independently.
--thinking-budget N places a positive upper bound on accepted model-origin tokens while the
new-turn Qwen thinking block remains open. If the model has not emitted </think> at that exact
boundary, Engine appends Qwen's canonical early-close guidance
and </think> to the same resident sequence without sampling, publishes the guidance through the
reasoning stream, then resumes ordinary generation from the updated context. A natural thinking
close, stop condition, cancellation, or total output/context limit at the boundary takes priority
and suppresses this insertion. The option cannot be combined with --no-thinking, but it can be
combined with --reasoning-effort.
--reasoning-loop off|stop|conclude (default off) ends the reply (stop) or closes the thinking
with the same control span and lets the model answer (conclude) when the open thinking keeps
repeating whole passages: every 512 thinking tokens, 25 % of the last 2,000 words inside a 12-word
passage seen three times. The check and its limits are described in
serving, under --reasoning-loop.
--max-new counts every committed generated token, including internally inserted control tokens.
When the effective output capacity extends beyond the thinking budget, it must have room for the
complete tokenizer-derived control suffix plus one post-close model token; an undersized request is
rejected rather than truncating the suffix. Normal output sends the inserted guidance to stderr as
reasoning. --print-token-ids includes the inserted IDs, while --raw-output preserves the raw
control representation.
For example, this allows at most 512 model-origin thinking tokens while retaining enough total output capacity for the inserted suffix and the answer:
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--prompt "Explain speculative decoding, then give a concise conclusion." \
--max-context 4096 \
--max-new 1024 \
--thinking-budget 512 \
--kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--lm-head-draftGPU residency is frozen when the Engine starts:
- no
--specomits MTP/DFlash/DFlash2 weights and state and the optimized proposal head; --spec mtp,--spec dflash(35B-A3B), and--spec dflash2(Qwen3.8-27B; Qwen3.8-Flash-Next with its companion artifact,--draft-model) load only the selected speculative backend;- a speculative backend with the full proposal head omits the optimized proposal head;
- Vision is disabled by default, omitting its weights and Vision-specific unified-workspace extent;
--visionloads the weights, expands the one Program workspace for Vision encode/handoff, and enables image/video input.- the one-request CLI disables the prefix cache, so it reserves no Device snapshot StateImage and no Host tier.
The complete .ninfer inventory is still validated. These choices are not lazy loading: an Engine
started without Vision rejects media and cannot enable Vision later. DFlash/DFlash2 and Vision may
be enabled together; these backends apply to generated-text decode after multimodal prefill and does not
accelerate Vision encode. The default speculative and Vision settings produce the smallest resident
profile.
--messages accepts either a non-empty JSON message array or an object containing messages
and an optional tools array.
[
{
"role": "system",
"content": "Answer concisely."
},
{
"role": "user",
"content": [
{
"type": "image",
"image": "examples/cli/media/visual_chart.png"
},
{
"type": "text",
"text": "Describe the chart."
}
]
}
]Run message files from the repository root when they contain repository-relative media paths:
./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer \
--messages examples/cli/messages/image_chart.json \
--max-context 8192 \
--max-new 128 \
--kv-dtype fp8 \
--vision \
--spec mtp --draft-tokens 3 \
--lm-head-draftSupported roles are system, developer, user, assistant, and tool.
The selected template formats these roles. The maintained Qwen templates keep system/developer
messages at their input positions.
Message content may be a string or an ordered array containing:
| Content type | Source field | Accepted source |
|---|---|---|
| text | text |
string |
| image / image_url | image or image_url |
local path, HTTP(S) URL, or base64 data URI |
| video / video_url | video or video_url |
local path, HTTP(S) URL, or base64 data URI |
image_url and video_url may be strings or objects containing a string url. Assistant
history may include reasoning_content and tool_calls; a tool result uses role tool and
tool_call_id.
See examples/cli/ for committed text, image, video, mixed-media, thinking,
long-decode, and long-context inputs.
Speculative decoding is disabled by default. Select MTP with one to five draft positions, or the
35B-A3B DFlash or Qwen3.8-27B DFlash2 backend with one to fifteen. Both masked-draft backends
may be combined with --vision.
--lm-head-draft selects the optimized proposal head and requires a selected backend:
./build/apps/infernix models/qwen3_6_35b_a3b.ninfer \
--prompt "Write a short explanation of speculative decoding." \
--max-context 16384 \
--max-new 512 \
--kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--lm-head-draftFor DFlash:
./build/apps/infernix models/qwen3_6_35b_a3b.ninfer \
--prompt "Write a short explanation of speculative decoding." \
--max-context 16384 --max-new 512 \
--kv-dtype fp8 \
--spec dflash --draft-tokens 7 --lm-head-draftFor Qwen3.8-27B artifacts containing the DFlash2 companion weights, select
--spec dflash2 --draft-tokens 7, optionally with --lm-head-draft and --vision.
DFlash2 accepts every draft count from 1 through 15; seven is the checkpoint recommendation.
Qwen3.8-Flash-Next artifacts take their DFlash2 drafter from a separate companion artifact
(--spec dflash2 --draft-model Qwen3.8-Flash-Next-DFlash2-Infernix.infernix --draft-tokens 7,
optionally --lm-head-draft), with 1 to 7 drafts (auto: each round's length from its acceptance,
up to 7); see the Flash-Next guide.
Both groupwise-int and nvfp4 artifacts use the same Engine route, including CUDA Graph,
concurrent requests, sampling penalties, and prefix reuse. An artifact without the companion
weights reports a missing DFlash2 component when selected. Vision, MTP and DFlash follow the same
rule: their weights are required only when that component is enabled at startup.
Only one speculative backend can be enabled per Engine. The published performance results use MTP with three draft tokens and DFlash with seven draft tokens (block length eight), both with the optimized proposal head. DFlash accepts one to fifteen draft tokens; seven forms the measured block length eight, while fifteen uses the maximum supported block length sixteen.
The table lists executable defaults. The examples above select FP8 KV and MTP3.
| Option | Meaning | Default |
|---|---|---|
--config FILE |
read options, and the model path, from a TOML file (Configuration file); command-line options override it | unset |
--rope-yarn-factor F |
startup-fixed runtime YaRN factor, finite [1,4]; extends allowed ceiling only |
1 |
--max-context N |
per-sequence logical context ceiling | 2048 |
--kv-capacity N|auto |
explicit shared Main Text KV capacity, or maximize it from remaining GPU memory; omitted means --max-context |
2048 |
--vram-headroom-mib N|auto |
VRAM in MiB that startup sizing leaves free. Qwen3.5: after --kv-capacity auto sizes the KV pool (requires auto; auto = 1024). Qwen3.8-Flash-Next: after the expert cache takes the rest, for the display and other programs (auto = 1024 with a display on the GPU, 256 without; details) |
auto |
--vram-past-budget |
Qwen3.8-Flash-Next on Windows: also size the expert cache into the VRAM the Windows per-process budget withholds (~0.6 GiB, ~240 more expert frames on an RTX 5090), measured at startup; that memory may be moved to system memory if another program needs VRAM (details) | off |
--expert-tail-skip R |
Qwen3.8-Flash-Next, approximate: trades quality for decode speed. In decode and verification rounds, a routed expert missing from VRAM that every token of the round routes at top-10 rank R or lower (1..9, 0-based; 9 skips least) is neither copied nor computed: it adds zero and the other experts keep their router weights. The output changes, is slightly worse, and depends on which experts the cache holds; prefill is unchanged; the exact route (--disable-fast-paths) refuses it. Strata does the same by default at rank 7 (STRATA_ROUTE_TAIL_SKIP=7) (details) |
off |
--prefill-chunk N |
positive text-prefill chunk, in multiples of 128; rounded down to whole prompt-attention waves (896 tokens for the 24-head model on RTX 5090) with INT8, NVFP4, FP8, K8V4, VQ2 or K4V2 KV unless an original prompt kernel is selected | 16384 for Qwen3.8-Flash-Next, 1024 otherwise |
--max-new N |
requested output-token limit | 128 |
--device N |
CUDA device index | 0 |
--ram-headroom-mib N |
physical Host RAM in MiB that pinned weights and caches leave free for the system; startup refuses a lock that would take it (Qwen3.8-Flash-Next plans every allocation against it in its RAM ledger) | 2048 |
--expert-ram-mib N|auto |
Qwen3.8-Flash-Next: Host RAM in MiB for the routed experts; less than all of them need (~64.5 GiB) keeps that many in RAM and reads the rest from the artifact (SSD expert tier); more than is free is an error | auto (what the ledger leaves) |
--ngram-residency auto|ram|volume |
Qwen3.8-Flash-Next: where the PLE n-gram table lives. ram reads the whole table into pinned Host RAM at startup (27.1 GiB for a GGUF import's IQ4_NL table, 48.8 GiB for the FP8 table) and never reads the volume again; volume reads rows on demand through a 2^20-row cache; auto takes RAM only when the RAM ledger still keeps every expert in RAM beside it (details) |
auto |
--expert-pin-mib N|auto |
Qwen3.8-Flash-Next: at most N MiB of the experts' RAM page-locked for the GPU; the rest stays in RAM unpinned and the CPU serves it (Page-locking limit) | auto (what Windows lets the GPU page-lock) |
--kv-dtype bf16|int8|fp8|nvfp4|k8v4|vq2|k4v2 |
KV-cache storage | bf16 |
--use-original-int8-prefill-kernel |
prefill INT8 KV with the original prompt-attention kernel; requires --kv-dtype int8 |
fast kernel |
--use-original-nvfp4-prefill-kernel |
prefill NVFP4 KV with the tiled prompt-attention kernel; requires --kv-dtype nvfp4 |
fast kernel over more than 768 visible keys |
--prefill-split-workspace-mib N |
workspace in MiB, 0..16384, for splitting one prompt-attention launch's keys across SMs (INT8, NVFP4, FP8, K8V4, VQ2 and K4V2 KV); less frees KV cache but slows 1-2K-token chunks over long context (64: 17-25 % slower attention over 32K-128K keys; 128: 2-5 %; 192 and up: under 1 %); 0 turns splitting off. See serving |
256 |
--disable-fast-paths |
run the exact route. By default every fast path qualified for the loaded artifact's configuration (the kernels it runs and its KV format; artifacts with the same kernels share one) and measured faster there runs: faster kernels (such as 8-bit P×V in prompt attention) whose output is not bit-exact with the exact route but whose measured quality cost stays within Infernix's numerical acceptance policy. Startup logs the paths selected (fast paths | ...) and whether a request's output is independent of the other requests (request-invariant). The exact route is bit-exact from release to release and, on artifacts whose formats support it (NVFP4 and FP8 dense text decoders), request-invariant: concurrency, other requests, the prefix cache's contents and KV pressure do not change a request's output. It then runs decode at 16-bit activations at every batch size, verifies one fixed-width chain per speculative round (no DFlash2 trees), resumes from the prefix cache only on the prefill chunk grid, and admits a request only when its whole KV extent (prompt + max_tokens) fits, so it is never preempted; the fast-path registry lists each path, the configurations it is qualified for, and its measured cost and speedup |
fast paths on |
--fast-path NAME |
also run this fast path (a name from the registry), qualified for the configuration or not; repeatable. For qualification runs: a path that cannot apply to the loaded model fails startup | none |
--spec mtp|dflash|dflash2 |
speculative backend | off |
--draft-tokens N|auto |
MTP 1..7 (Qwen3.5/3.6 1..5); DFlash/DFlash2 1..15 (Qwen3.8-Flash-Next DFlash2 1..7). N drafts and verifies N tokens every round (fewer only near the output limit). auto on Qwen3.8-Flash-Next DFlash2: each round drafts 7 and verifies the number its drafts' calibrated probabilities make fastest (guide). auto (MTP): each round drafts the length the learned acceptance makes fastest, 0..7 on Qwen3.8-Flash-Next (with the --spec-min-p cut, design §11.3); on Qwen3.5/3.6 it is the adaptive-mtp fast path (1..5) where that is qualified (the NVFP4 Qwen3.8-27B and Qwen3.6-35B-A3B); elsewhere, and with --disable-fast-paths, every round drafts the artifact's fastest fixed length (5 on the NVFP4 27B, 3 otherwise), which startup reports |
unset |
--lm-head-draft |
optimized proposal head | off |
--draft-model FILE |
Qwen3.8-Flash-Next with --spec dflash2: the DFlash2 drafter's companion artifact (tools.convert --components dflash2); Qwen3.5/3.6 artifacts bundle their drafters and refuse it |
unset |
--spec-min-p P |
Qwen3.8-Flash-Next with --spec mtp or --spec dflash2 only: a one-request round verifies its drafts up to the first whose drafter probability (MTP's draft head, DFlash2's candidate selector) is below P, 0..1; 0 verifies every draft. Greedy output still equals plain decoding; a seeded sampled run may realize differently with the same distribution. See serving for the measured default |
0.5 with --draft-tokens auto, else 0 |
--draft-tree-nodes auto|N |
DFlash2 tree verification: each round verifies a small tree of alternative drafts, built on the device from the drafter's candidate lattice, instead of a single draft, so more drafted tokens are accepted with the same output distribution; any --kv-dtype. auto (recommended) chooses the single draft or a tree of draft tokens + 5 or + 9 tokens every round from measured speed and acceptance at the current context length, and keeps single drafts (with a warning) on an artifact without tree support; N fixes a tree of draft tokens + 2..32 tokens and keeps seeded runs reproducible, which auto does not; see tree verification |
off |
--draft-tree-paths N |
most branches (root-to-leaf paths) per tree, 2..8 |
8 |
--ngram-draft-tokens N |
verified n-gram copy proposals per round beside the --spec drafter (Qwen3.8-Flash-Next: also alone, 1..15), 1..63; 0 disables; see ngram copy proposals |
0 |
--ngram-min-match N |
minimum matched tokens for an n-gram proposal, 4..64 |
12 |
--vision |
enable image/video input and load Vision GPU allocations | off |
--vision-offload auto|on|off |
keep the vision tower in pinned system RAM and borrow Device memory only while encoding; takes effect only with --vision. auto is on for Qwen3.8-Flash-Next (the expert cache lends the encode memory) and off for Qwen3.5 |
auto |
--vision-max-merged N |
merged vision tokens per image or video, 64..32768; larger media is downscaled during preprocessing |
32768 |
--no-cuda-graph |
disable CUDA Graph decode | graphs on |
--expert-state FILE|off |
Qwen3.8-Flash-Next: the expert cache's saved state. At startup the engine fills the VRAM expert cache with the experts the file ranks highest (about 1 s for a full cache) and starts their use counts from it; it writes the file when it stops and every 10 minutes between requests (through FILE.tmp and a rename). A file from another artifact or of another size is ignored and replaced. Without a usable file the engine seeds the cache from a built-in calibrated expert order instead. off starts with an empty cache, no seed, and saves nothing; cold benchmarks need it |
<artifact>.expert-state |
--expert-stripe FILE |
Qwen3.8-Flash-Next with the SSD expert tier: an expert stripe (tools/flash_next/expert_stripe.py) holding the tail of every expert record on a second drive; each record not in RAM is then read in two parts, its head from the artifact and its tail from the stripe, at the same time. The stripe must belong to the artifact. Unused when every expert fits in RAM (guide) |
none |
--chat-template FILE |
use a local Jinja template | artifact template |
--no-thinking |
disable thinking | template default |
--thinking-budget N |
positive model-origin thinking-token cap; omitted means unlimited | unset |
--reasoning-loop off|stop|conclude |
end the reply or close the thinking when it keeps repeating whole passages | off |
--reasoning-effort none|minimal|low|medium|high|xhigh|max |
pass an effort value to the selected template | template default |
--greedy |
exact argmax decoding | off |
--temperature F |
sampling temperature override | registered model/mode default |
--top-p F |
nucleus-threshold override | registered model/mode default |
--top-k N |
top-k-threshold override (0..20; zero selects the top-20 cap) |
registered model/mode default |
--min-p F |
min-p-threshold override | registered model/mode default |
--presence-penalty F |
presence-penalty override | registered model/mode default; 0 with an output constraint |
--frequency-penalty F |
frequency-penalty override | registered model/mode default (0) |
--seed N |
sampling seed | 0 |
--log-colours on|off |
colour the statistics output on stderr | on when stderr is a terminal |
When a sampling flag is omitted, Engine selects the general-task preset for the loaded architecture and rendered prompt mode. The current official models use:
| Model | Prompt mode | Temperature | Top-p | Top-k | Min-p | Presence penalty |
|---|---|---|---|---|---|---|
| Qwen3.6-27B | thinking | 1.0 |
0.95 |
20 |
0 |
0 |
| Qwen3.6-27B | non-thinking | 0.7 |
0.80 |
20 |
0 |
1.5 |
| Qwen3.8-27B | thinking | 1.0 |
0.95 |
20 |
0 |
0 |
| Qwen3.8-27B | non-thinking | 0.7 |
0.80 |
20 |
0 |
1.5 |
| Qwen3.6-35B-A3B | thinking | 1.0 |
0.95 |
20 |
0 |
1.5 |
| Qwen3.6-35B-A3B | non-thinking | 0.7 |
0.80 |
20 |
0 |
1.5 |
| Qwen3.8-Flash-Next | thinking | 1.0 |
0.95 |
20 |
0 |
0 |
| Qwen3.8-Flash-Next | non-thinking | 0.7 |
0.80 |
20 |
0 |
1.5 |
Frequency penalty is 0 in every registered preset. Task-specific profiles such as Qwen's
precise-coding profile use explicit sampling overrides.
Repeat --stop-token-id, --stop, or --reasoning-stop to add stop conditions. Use
--raw-output to expose the frontend's raw output stream and --print-token-ids to include
generated token IDs in diagnostics.
Run ./build/apps/infernix --help for the exact option contract.
--config FILE reads options from a TOML file keyed by the long option names without the dashes,
in the format infernix-serve reads (serving has the full rules).
Top-level keys and those of a [cli] table apply to infernix, the [cli] table's overriding
the top level's; a [serve] table is infernix-serve's and is skipped. model names the model
when the command line does not, and repeatable options take arrays:
model = "models/qwen3_8_27b_nvfp4.ninfer"
kv-dtype = "int8"
spec = "dflash2"
draft-tokens = 7
[cli]
max-context = 32768
max-new = 1024
prompt = "Summarize sparse attention."
stop = ["END", "STOP"]./build/apps/infernix --config infernix.toml --temperature 0.6Options on the command line override the file; the values of a repeatable option such as --stop
add to the file's. The startup log lists the options the file gave.
INFERNIX_CUDA_SYNC selects the CUDA device synchronization schedule at startup for both the CLI
and HTTP server. When unset, it defaults to spin, prioritizing low synchronization latency at
the cost of CPU usage while waiting for the GPU. Use blocking to let the waiting thread sleep;
the decode performance cost depends on the host. yield yields the CPU while waiting, and auto
uses CUDA's scheduling heuristic, not an automatic performance benchmark.
For example, infernix-serve on an RTX 5090 under Windows, replaying an agentic workload at two
concurrent requests, used 165 % of a CPU core on average with spin, 103 % with yield and 2 %
with blocking. Against spin, output tok/s was 1.8 % lower with yield and 1.2 % lower with
blocking, and mean time to first token moved by at most 0.5 %.
INFERNIX_CUDA_SYNC=blocking ./build/apps/infernix models/qwen3_8_27b_nvfp4.ninfer --prompt "Hello"The Engine-ready log reports the selected mode. Empty or unrecognized values, or failure to apply the schedule, fail startup. This controls device scheduling (including stream synchronization); it does not override individual CUDA event creation flags.
The official artifacts have a native context limit of 262,144 tokens. The practical allocation
on one RTX 5090 depends on the selected artifact, media workload, output budget, and KV-cache type.
The artifact describes its model configuration and weight representations;
--rope-yarn-factor F opts into startup-fixed runtime YaRN with a finite factor in [1,4]
(default 1, native RoPE unchanged). It does not modify the artifact or converter. The factor
extends the allowed context ceiling, but does not enlarge the default --max-context or KV pool;
request a larger context explicitly. Memory and selected speculative-backend limits still apply.
This is long-context extrapolation, not a guarantee of answer quality; evaluate your workload.
--kv-dtype independently selects runtime KV storage. The prepared prompt must fit
--max-context; generation stops at the remaining context capacity when necessary.
Per token and KV head, K+V take 1024 bytes as bf16, 528 as int8, 516 as fp8, 402 as k8v4,
288 as nvfp4, 196 as k4v2 and 132 as vq2. vq2 stores 2-bit vector-quantized K and V (one
16-bit code per 8 rotated values) and k4v2 4-bit K with the same 2-bit V; both keep the attention
sinks and the most recent 768 keys exact through a per-sequence INT8 window, about 40 MB per state
slot for the 27B models, which every running request and every Device snapshot slot holds.
--kv-capacity N controls the shared physical Main Text KV pool independently and is rounded up to
the 64-token page size. --kv-capacity auto loads the selected weights, measures the remaining GPU
memory, and directly chooses the largest legal page capacity for the complete enabled runtime
layout. This includes the selected speculative backend, fixed sequence state, unified workspace,
and CUDA Graph allowance, while leaving the default 1 GiB automatic headroom
unallocated. It does not probe allocations or resize the pool at request time. The single-request
CLI normally leaves the option omitted so it follows
--max-context; the distinction matters primarily to a concurrent Engine or server.
At Engine startup Infernix reserves model weights, persistent sequence state, one phase-reused
Program workspace, and a separate CUDA Graph driver allowance. With Vision enabled, that one
workspace contains a general execution prefix and a fixed item-output handoff region. Vision encode
may reuse the full backing before producing the output; Text/MTP/decode work remains inside the
general prefix while the handoff is live. The capacity is therefore the maximum legal simultaneous
extent, not the sum of Text, Vision scratch, and Vision output allocations. Text prefill uses
min(--prefill-chunk,--max-context); Vision keeps the existing 32,768-token aggregate prompt budget
but plans Device execution for the registered 16,384-token maximum single item. Requests perform no
project-owned device allocation or growth. Context-cache capacity controls are intentionally absent
from this one-request interface; the persistent Engine and server routes own cross-request reuse and
optional Host backing.
All weight, sequence, workspace, and graph allocations are released when the Engine is destroyed.