Repository navigation
Restart rebooted harts on a coroutine stack - #788
Conversation
Benchmark comparison against the base commitCurrent: 85d0f38. Workflow result: success (details). Benchmark comparison (T1C)
Both builds ran concurrently on AMD EPYC 7763 64-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change. Current: 85d0f38 merged into master. Previous: master at e111bc0. Benchmark comparison (T2C)
Both builds ran concurrently on AMD EPYC 9V74 80-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change. Current: 85d0f38 merged into master. Previous: master at e111bc0. Benchmark comparison (interpreter)
Both builds ran concurrently on AMD EPYC 7763 64-Core Processor. Changes smaller than 2% or whose interval spans zero are reported as no significant change. Current: 85d0f38 merged into master. Previous: master at e111bc0. |
A guest reboot used to longjmp from the SBI reset handler to a setjmp in rv_step(). The jump point was set up again on every step, the jump crossed whatever interpreter, JIT or T2C frames lay in between, and the jump buffer had to live in riscv_t for the handler to reach it. Native system builds now run the hart on a coroutine: src/coro.c switches contexts in a few instructions on x86-64 and Arm64 and falls back to ucontext elsewhere. A reboot abandons the coroutine stack, and the next resume starts the hart's run loop from the top. That is safe because nothing between guest execution and the reset ecall holds a lock or owns heap memory. The coroutine returns to rv_run() only when the hart halts or restarts, so the virtio-net refresh runs inside it; under gdbstub it yields after every instruction. The Emscripten build steps the hart from the browser main loop, so it finishes the slice and resets the dispatcher state instead. The coroutine stack is 8 MiB of lazily committed memory with a 64 KiB guard region below it. A fault in the guard region prints a diagnostic from a handler on an alternate signal stack; every other fault goes on to the demand-paging handler. rv_debug() now releases the breakpoint map and the gdbstub connection whenever a session ends. Boot time to the login prompt is unchanged on x86-64 and Arm64.
A guest reboot used to
longjmpfrom the SBI reset handler back to asetjmpinrv_step(), which set the jump point up on every step and jumped across whatever interpreter, JIT or T2C frames lay in between. Native system builds now run the hart on a coroutine (src/coro.c: a short context switch on x86-64 and Arm64, ucontext elsewhere). A reboot abandons the coroutine stack, and the next resume starts the hart's run loop from the top. The coroutine returns torv_run()only when the hart halts or restarts; under gdbstub it yields after every instruction. The Emscripten build keeps a direct-step path. The coroutine stack is 8 MiB of lazily committed memory with a 64 KiB guard region, and an overflow into it prints a diagnostic instead of corrupting the heap. All of this is gated by a newRV32_HAS_HART_CORO(SYSTEM_MMIOand not Emscripten); user mode, the ELF loader and wasm are unaffected.Validation, on x86-64 (Threadripper 2990WX) unless noted:
defconfig,jit_defconfig(T2C),system_defconfig,system_interpreter_defconfig,system_jit_defconfig(T2C), system withELF_LOADER, and system and user mode withGDBSTUB..ci/boot-linux-test.sh, built as CI does (INITRD_SIZE=32 ENABLE_MOP_FUSION=0, thenmake artifact), onsystem_defconfig(gcc),system_interpreter_defconfig(clang) andsystem_jit_defconfig(clang): 26/26 each, covering boot, cold and warm reboot, virtio-blk, and virtio-net over TAP and SLIRP.make checkondefconfig,jit_defconfigand system withELF_LOADER: all pass.ENABLE_SDL=0):system_defconfigandsystem_interpreter_defconfigbuild; cold and warm reboot pass on both.src/coro.ckeeps integer and floating-point values live across a million switches in both directions, and checks restart, the overflow diagnostic, and that other faults are passed on. It passes at -O0, -O2 and -O3 with gcc and clang, on x86-64 (asm and ucontext) and Arm64.system_defconfigtakes 3.07 to 3.10 s on this branch against 3.08 s on master, and Arm64 takes 8.50 s against 8.58 s. On x86-64, identical source built in two trees differed by up to 6%, so only differences beyond that were treated as real.Not covered: the wasm build (no emcc here), and gdbstub beyond the first few dozen
stepi. Master already hangs in system-mode gdbstub at the same point (pc 0x100), and this change does not alter that.Summary by cubic
Reboots used to
longjmpfrom the SBI reset handler to asetjmppoint set up on every step. Native system builds now run each hart on a coroutine, so a guest reboot abandons the whole host call stack and restarts the hart's run loop from the top on the next resume.src/coro.c: assembly context switches on x86-64 and Arm64, ucontext elsewhere; 8 MiB lazily committed stack with a 64 KiB guard region that prints a diagnostic on overflow.RV32_HAS_HART_CORO(system mode, no Emscripten); user mode, the ELF loader, and wasm are unaffected.rv_run()only on halt or restart; under gdbstub it yields after every instruction.rv_debug()now releases the breakpoint map and gdbstub connection when a session ends.Written for commit 85d0f38. Summary will update on new commits.