Skip to content

Improve workspace defaults and add workload benchmark comparisons - #499

Merged
vince62s merged 3 commits into
eole-nlp:mainfrom
vince62s:feat/workspace-dflash2-default
Nov 2, 2026
Merged

vince62s merged 3 commits into
eole-nlp:mainfrom
vince62s:feat/workspace-dflash2-default

Conversation

@vince62s

@vince62s vince62s commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

The workspace opens generation settings by default, prefers compatible DFlash2 with INT8 packed drafter projections, and raises the Chat output allowance to 16,384 tokens in local/RunPod templates. Explicit user choices are retained, with MTP/Off fallback when DFlash2 is unavailable. The proposal count continues to follow the checkpoint block size; server startup preparation is unchanged.

Benchmark now compares three fixed workloads (Python code, English story, Chinese-to-English translation) using six independently selectable profiles:

  • Base with native vocabulary heads.
  • MTP with native or INT4 vocabulary heads.
  • DFlash2 with BF16 or INT8 internal drafter projections and native vocabulary heads.
  • DFlash2 with INT8 internal projections and INT4 vocabulary heads.

All profiles use text-only greedy decoding with thinking disabled, identical prompts within each workload and a configurable common output cap (default 512) and 1–5 repeats (default 3). Each workload/profile pair receives one discarded warmup; profiles are grouped to avoid repeated reloads between workloads. Native dtype, target/drafter precision and draft counts remain distinct.

The results table retains previous comparisons until page reload and reports medians, first-token latency, acceptance and completion status. JSON export includes exact prompts/settings, hardware, context/packing/compile metadata, separate warmups and raw measured runs. Early stops, incorrect actual speculation modes, cancellation and errors cannot produce a complete fixed-length median. CUDA OOM stops the suite, retaining available results. Model switching/preparation is excluded from decode timing. The final engine remains loaded until another request or unload.

Validation:

  • Full CPU suite: 811 tests, OK (43 skipped).
  • Browser helper/runner tests: warmup exclusion, fixed requests, retained comparisons, unavailable profiles, early EOS, actual-mode checks, OOM and cancellation.
  • Real RTX 5090 browser smoke tests: six complete MTP/INT4 and DFlash2/INT8+INT4 rows across all three workloads at a 128-token cap and one measured repeat; repeated successfully. Report download and its contents verified. No browser console errors.
  • Black, flake8, Prettier, JavaScript syntax and whitespace checks passed.
  • Model unloaded after testing; preview server remains available on localhost:5018.

Benchmark workspace

Screenshot is a short validation run, not the default 512-token, three-repeat performance comparison.

@vince62s vince62s changed the title Show generation settings by default and prefer available DFlash2 Improve workspace defaults and add workload benchmark comparisons Nov 1, 2026
@vince62s
vince62s merged commit 884a99f into eole-nlp:main Nov 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant