Repository navigation
Improve workspace defaults and add workload benchmark comparisons - #499
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workspace opens generation settings by default, prefers compatible DFlash2 with INT8 packed drafter projections, and raises the Chat output allowance to 16,384 tokens in local/RunPod templates. Explicit user choices are retained, with MTP/Off fallback when DFlash2 is unavailable. The proposal count continues to follow the checkpoint block size; server startup preparation is unchanged.
Benchmark now compares three fixed workloads (Python code, English story, Chinese-to-English translation) using six independently selectable profiles:
All profiles use text-only greedy decoding with thinking disabled, identical prompts within each workload and a configurable common output cap (default 512) and 1–5 repeats (default 3). Each workload/profile pair receives one discarded warmup; profiles are grouped to avoid repeated reloads between workloads. Native dtype, target/drafter precision and draft counts remain distinct.
The results table retains previous comparisons until page reload and reports medians, first-token latency, acceptance and completion status. JSON export includes exact prompts/settings, hardware, context/packing/compile metadata, separate warmups and raw measured runs. Early stops, incorrect actual speculation modes, cancellation and errors cannot produce a complete fixed-length median. CUDA OOM stops the suite, retaining available results. Model switching/preparation is excluded from decode timing. The final engine remains loaded until another request or unload.
Validation:
Screenshot is a short validation run, not the default 512-token, three-repeat performance comparison.