Skip to content

Restart rebooted harts on a coroutine stack - #788

Merged
jserv merged 1 commit into
masterfrom
rework-step
Oct 3, 2026
Merged

jserv merged 1 commit into
masterfrom
rework-step

Conversation

@jserv

@jserv jserv commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

A guest reboot used to longjmp from the SBI reset handler back to a setjmp in rv_step(), which set the jump point up on every step and jumped across whatever interpreter, JIT or T2C frames lay in between. Native system builds now run the hart on a coroutine (src/coro.c: a short context switch on x86-64 and Arm64, ucontext elsewhere). A reboot abandons the coroutine stack, and the next resume starts the hart's run loop from the top. The coroutine returns to rv_run() only when the hart halts or restarts; under gdbstub it yields after every instruction. The Emscripten build keeps a direct-step path. The coroutine stack is 8 MiB of lazily committed memory with a 64 KiB guard region, and an overflow into it prints a diagnostic instead of corrupting the heap. All of this is gated by a new RV32_HAS_HART_CORO (SYSTEM_MMIO and not Emscripten); user mode, the ELF loader and wasm are unaffected.

Validation, on x86-64 (Threadripper 2990WX) unless noted:

  • Builds with gcc and clang, all passing: defconfig, jit_defconfig (T2C), system_defconfig, system_interpreter_defconfig, system_jit_defconfig (T2C), system with ELF_LOADER, and system and user mode with GDBSTUB.
  • .ci/boot-linux-test.sh, built as CI does (INITRD_SIZE=32 ENABLE_MOP_FUSION=0, then make artifact), on system_defconfig (gcc), system_interpreter_defconfig (clang) and system_jit_defconfig (clang): 26/26 each, covering boot, cold and warm reboot, virtio-blk, and virtio-net over TAP and SLIRP.
  • make check on defconfig, jit_defconfig and system with ELF_LOADER: all pass.
  • Arm64 (Ampere eMAG, clang, ENABLE_SDL=0): system_defconfig and system_interpreter_defconfig build; cold and warm reboot pass on both.
  • A standalone test linking src/coro.c keeps integer and floating-point values live across a million switches in both directions, and checks restart, the overflow diagnostic, and that other faults are passed on. It passes at -O0, -O2 and -O3 with gcc and clang, on x86-64 (asm and ucontext) and Arm64.
  • Boot time to the login prompt is unchanged: x86-64 system_defconfig takes 3.07 to 3.10 s on this branch against 3.08 s on master, and Arm64 takes 8.50 s against 8.58 s. On x86-64, identical source built in two trees differed by up to 6%, so only differences beyond that were treated as real.

Not covered: the wasm build (no emcc here), and gdbstub beyond the first few dozen stepi. Master already hangs in system-mode gdbstub at the same point (pc 0x100), and this change does not alter that.


Summary by cubic

Reboots used to longjmp from the SBI reset handler to a setjmp point set up on every step. Native system builds now run each hart on a coroutine, so a guest reboot abandons the whole host call stack and restarts the hart's run loop from the top on the next resume.

  • Adds src/coro.c: assembly context switches on x86-64 and Arm64, ucontext elsewhere; 8 MiB lazily committed stack with a 64 KiB guard region that prints a diagnostic on overflow.
  • Gated by RV32_HAS_HART_CORO (system mode, no Emscripten); user mode, the ELF loader, and wasm are unaffected.
  • The coroutine returns to rv_run() only on halt or restart; under gdbstub it yields after every instruction.
  • The Emscripten build keeps the direct-step path and resets dispatcher state after a reboot instead.
  • rv_debug() now releases the breakpoint map and gdbstub connection when a session ends.
  • Boot time to the login prompt is unchanged on x86-64 and Arm64.

Written for commit 85d0f38. Summary will update on new commits.

Review in cubic

@jserv jserv added this to the release-2026.2 milestone Oct 2, 2026
cubic-dev-ai[bot]

This comment was marked as resolved.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Benchmark comparison against the base commit

Current: 85d0f38. Workflow result: success (details).
Measured on one shared runner; treat verdicts as advisory.

Benchmark comparison (T1C)

Benchmark Previous: master e111bc0 (median) Current: 85d0f38 (median) Change 95% CI Pairs Verdict
Dhrystone 15353.000 DMIPS 15355.000 DMIPS -0.32% [-0.98%, +0.35%] 6 no significant change
CoreMark 6748.023 iterations/sec 6703.952 iterations/sec -0.12% [-1.38%, +1.17%] 6 no significant change

Both builds ran concurrently on AMD EPYC 7763 64-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change.

Current: 85d0f38 merged into master. Previous: master at e111bc0.

Benchmark comparison (T2C)

Benchmark Previous: master e111bc0 (median) Current: 85d0f38 (median) Change 95% CI Pairs Verdict
Dhrystone 36747.000 DMIPS 36700.000 DMIPS -0.66% [-1.69%, +0.39%] 6 no significant change
CoreMark 13434.664 iterations/sec 13457.097 iterations/sec -0.16% [-0.71%, +0.40%] 6 no significant change

Both builds ran concurrently on AMD EPYC 9V74 80-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change.

Current: 85d0f38 merged into master. Previous: master at e111bc0.

Benchmark comparison (interpreter)

Benchmark Previous: master e111bc0 (median) Current: 85d0f38 (median) Change 95% CI Pairs Verdict
Dhrystone 2049.000 DMIPS 2105.000 DMIPS -0.83% [-5.23%, +3.78%] 5 no significant change
CoreMark 1248.989 iterations/sec 1250.872 iterations/sec +0.22% [+0.05%, +0.40%] 6 no significant change

Both builds ran concurrently on AMD EPYC 7763 64-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change.

Current: 85d0f38 merged into master. Previous: master at e111bc0.

A guest reboot used to longjmp from the SBI reset handler to a setjmp in
rv_step(). The jump point was set up again on every step, the jump
crossed whatever interpreter, JIT or T2C frames lay in between, and the
jump buffer had to live in riscv_t for the handler to reach it.

Native system builds now run the hart on a coroutine: src/coro.c
switches contexts in a few instructions on x86-64 and Arm64 and falls
back to ucontext elsewhere. A reboot abandons the coroutine stack, and
the next resume starts the hart's run loop from the top. That is safe
because nothing between guest execution and the reset ecall holds a
lock or owns heap memory. The coroutine returns to rv_run() only when
the hart halts or restarts, so the virtio-net refresh runs inside it;
under gdbstub it yields after every instruction. The Emscripten build
steps the hart from the browser main loop, so it finishes the slice and
resets the dispatcher state instead.

The coroutine stack is 8 MiB of lazily committed memory with a 64 KiB
guard region below it. A fault in the guard region prints a diagnostic
from a handler on an alternate signal stack; every other fault goes on
to the demand-paging handler. rv_debug() now releases the breakpoint
map and the gdbstub connection whenever a session ends.

Boot time to the login prompt is unchanged on x86-64 and Arm64.

@Charlie-Tsai1123 Charlie-Tsai1123 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I also tested this change with the virtio-snd work from #761 on my branch.

After rebasing virtio-snd on top of this PR and moving rv_refresh_vsnd() into rv_hart_coroutine() alongside rv_refresh_vnet(), virtio-net and virtio-snd work successfully while repeatedly reboot.

@jserv
jserv merged commit 12ab0b6 into master Oct 3, 2026
74 of 75 checks passed
@jserv
jserv deleted the rework-step branch October 3, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants