Add hardware A/B and native Thumb test framework - #11561
Open
Eric Anderson (humanapp) wants to merge 13 commits into
Open
Add hardware A/B and native Thumb test framework#11561Eric Anderson (humanapp) wants to merge 13 commits into
Eric Anderson (humanapp) wants to merge 13 commits into
Conversation
tests/thumb-test compiles programs to native ARM Thumb in-process against a captured micro:bit (mbcodal) CompileOptions fixture and asserts on the generated assembly listing; runs offline via gulp testthumb. lang-test0 gains three semantic files covering condition truthiness and lowering and interface dispatch (including both sides of the call-site-count thresholds that gate dispatch specialization), plus a deliberately disabled repro of the default-parameters-on-dynamic-dispatch defect and a coverage map (README-codegen.md) tying failure modes to tests and layers.
Cross-platform node scripts (macOS/Linux/Windows) to build candidate and reference hexes, flash a micro:bit, capture serial, and diff the traces, with a gcStats-based soak mode for leak detection. Driven by one wrapper: npm run hwab -- ab truthiness.
Findings from running the capture against a real micro:bit: - DAPLink buffers serial while no host is reading, so a fresh capture can open onto the previously flashed program's output, including its PASS banner. Verdicts are now read only from the flashed program's own HWAB START banner onward, and --expect <case> (passed by hwab) requires the banner and verdict to name the flashed case. Timeout diagnostics report ignored foreign output and point at FAIL.TXT when no banner appears. - Panic does not flush the serial FIFO, which truncated the assert id on the wire. The generated device assert now drains for 250 ms between logging the id and panicking.
- the diff takes its expected case from --case or the log's directory instead of log content; DAPLink's buffer can replay a previous program's entire run, START banner included, which fooled content derivation - a stale fragment glued to the front of the expected START banner is cut off - verified against the capture logs that produced the false divergence, plus synthetic glued and genuinely divergent pairs
… program - a soak whose program never prints (a failed flash) gives up after a 90s grace instead of consuming the whole soak window, which hands it to the existing re-flash retry - the soak program's object literals capture a call parameter instead of the let loop variable, whose closure capture sees the loop's final value (issue filed: microsoft#11563); its dispatch sanity assert now holds - verified: the corrected dispatch sum on the simulator; soak telemetry flowing through the changed capture path on hardware
- discard the first seconds of serial after open: DAPLink replays buffered history, which can include a previous run of the same case
- native-compile 57defaultparamdispatch.ts with the other language tests, so the repro survives the native emitter and the eventual fix is validated there from the start
Thomas Ball (thomasjball)
enabled auto-merge
September 3, 2026 16:14
Thomas Ball (thomasjball)
requested review from
Thomas Ball (thomasjball)
and removed request for
Thomas Ball (thomasjball)
September 3, 2026 16:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a framework for testing compiled programs, exercised offline and on actual microbit hardware. The motivation: validating the codegen optimizations in #11336 and #11339, so the initial test suite covers exactly what they change: condition lowering and interface dispatch. The framework itself though is general. Once this gets merged, I will rebase those PRs onto it and they'll gain real test coverage.
Testing on microbit
tests/hw-ab/builds candidate and reference hexes, flashes a microbit, captures serial, and diffs the traces -- plus a soak mode that watchescontrol.gcStats()for leaks. One command per case:Writing the tests caught a pre-existing compiler bug
Default parameter values are not applied on dynamic dispatch --
m(a: number, b = 5)called asx.m(1)through an interface- orany-typed reference yieldsNaN(the arity wrapper fillsundefined; the initializer never runs). A ready-made red test is checked in disabled: uncomment the REPRO block in57defaultparamdispatch.tsandgulp testlangfails with the assertion id. Example of code that breaks:Repros in both JS and on hardware.
(Just to make sure, I verified this bug existed prior to any of my compiler optimizations.)
The test framework's structure
The three layers
gulp testlanggulp testthumbnpm run hwabCoverage map (failure mode -> test -> layer): see
tests/compile-test/lang-test0/README-codegen.md.microbit test fixtures in pxt repo? why?
The thumb tests have to compile against some hardware target; microbit is the most natural one.
tests/thumb-test/fixtures/microbit-mbcodal.json.gzsnapshots a real mbcodal build: compile config, runtime hex + function table, core TS sources. It is inert input data (not pre-compiled; the current compiler always runs against it), it fails loudly if a test references a missing API, and regeneration is one documented command. Details intests/thumb-test/README.md.cc: Thomas Ball (@thomasjball)