Skip to content

Add hardware A/B and native Thumb test framework - #11561

Open
Eric Anderson (humanapp) wants to merge 13 commits into
microsoft:masterfrom
humanapp:eanders/codegen-tests
Open

Add hardware A/B and native Thumb test framework#11561
Eric Anderson (humanapp) wants to merge 13 commits into
microsoft:masterfrom
humanapp:eanders/codegen-tests

Conversation

@humanapp

Copy link
Copy Markdown
Contributor

Adds a framework for testing compiled programs, exercised offline and on actual microbit hardware. The motivation: validating the codegen optimizations in #11336 and #11339, so the initial test suite covers exactly what they change: condition lowering and interface dispatch. The framework itself though is general. Once this gets merged, I will rebase those PRs onto it and they'll gain real test coverage.

Testing on microbit

tests/hw-ab/ builds candidate and reference hexes, flashes a microbit, captures serial, and diffs the traces -- plus a soak mode that watches control.gcStats() for leaks. One command per case:

npm run hwab -- ab truthiness

Writing the tests caught a pre-existing compiler bug

Default parameter values are not applied on dynamic dispatch -- m(a: number, b = 5) called as x.m(1) through an interface- or any-typed reference yields NaN (the arity wrapper fills undefined; the initializer never runs). A ready-made red test is checked in disabled: uncomment the REPRO block in 57defaultparamdispatch.ts and gulp testlang fails with the assertion id. Example of code that breaks:

interface Shape {
    scaled(x: number, factor?: number): number;
}

class Thing implements Shape {
    scaled(x: number, factor = 5) {
        return x * factor;
    }
}

const t = new Thing();
const s: Shape = t;

console.log(t.scaled(2));   // 10  -- static call, default applied
console.log(s.scaled(2));   // NaN -- same object via interface: default not applied

Repros in both JS and on hardware.

(Just to make sure, I verified this bug existed prior to any of my compiler optimizations.)


The test framework's structure

The three layers

layer run catches
JS-executed semantics gulp testlang wrong answers -- 3 new lang-test0 files covering condition truthiness/lowering and interface dispatch, run on the simulator backend
native assembly shape gulp testthumb which sequences codegen chose, unencodable instructions, size swings -- compiles to ARM Thumb
on-device A/B + soak npm run hwab runtime failures on device: GC, panics, leaks over time

Coverage map (failure mode -> test -> layer): see tests/compile-test/lang-test0/README-codegen.md.

microbit test fixtures in pxt repo? why?

The thumb tests have to compile against some hardware target; microbit is the most natural one. tests/thumb-test/fixtures/microbit-mbcodal.json.gz snapshots a real mbcodal build: compile config, runtime hex + function table, core TS sources. It is inert input data (not pre-compiled; the current compiler always runs against it), it fails loudly if a test references a missing API, and regeneration is one documented command. Details in tests/thumb-test/README.md.

cc: Thomas Ball (@thomasjball)

tests/thumb-test compiles programs to native ARM Thumb in-process against a
captured micro:bit (mbcodal) CompileOptions fixture and asserts on the
generated assembly listing; runs offline via gulp testthumb.

lang-test0 gains three semantic files covering condition truthiness and
lowering and interface dispatch (including both sides of the call-site-count
thresholds that gate dispatch specialization), plus a deliberately disabled
repro of the default-parameters-on-dynamic-dispatch defect and a coverage map
(README-codegen.md) tying failure modes to tests and layers.
Cross-platform node scripts (macOS/Linux/Windows) to build candidate and
reference hexes, flash a micro:bit, capture serial, and diff the traces, with
a gcStats-based soak mode for leak detection. Driven by one wrapper:
npm run hwab -- ab truthiness.
Findings from running the capture against a real micro:bit:

- DAPLink buffers serial while no host is reading, so a fresh capture can
  open onto the previously flashed program's output, including its PASS
  banner. Verdicts are now read only from the flashed program's own
  HWAB START banner onward, and --expect <case> (passed by hwab) requires
  the banner and verdict to name the flashed case. Timeout diagnostics
  report ignored foreign output and point at FAIL.TXT when no banner
  appears.
- Panic does not flush the serial FIFO, which truncated the assert id on
  the wire. The generated device assert now drains for 250 ms between
  logging the id and panicking.
- the diff takes its expected case from --case or the log's directory instead of log content; DAPLink's buffer can replay a previous program's entire run, START banner included, which fooled content derivation
- a stale fragment glued to the front of the expected START banner is cut off
- verified against the capture logs that produced the false divergence, plus synthetic glued and genuinely divergent pairs
… program

- a soak whose program never prints (a failed flash) gives up after a 90s grace instead of consuming the whole soak window, which hands it to the existing re-flash retry
- the soak program's object literals capture a call parameter instead of the let loop variable, whose closure capture sees the loop's final value (issue filed: microsoft#11563); its dispatch sanity assert now holds
- verified: the corrected dispatch sum on the simulator; soak telemetry flowing through the changed capture path on hardware
- discard the first seconds of serial after open: DAPLink replays buffered history, which can include a previous run of the same case
- native-compile 57defaultparamdispatch.ts with the other language tests, so the repro survives the native emitter and the eventual fix is validated there from the start
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants