Skip to content

enable faulthandler in every generated script, not just OOM mode - #261

Merged
devdanzin merged 1 commit into
mainfrom
faulthandler-always
Aug 19, 2026
Merged

enable faulthandler in every generated script, not just OOM mode#261
devdanzin merged 1 commit into
mainfrom
faulthandler-always

Conversation

@devdanzin

Copy link
Copy Markdown
Owner

Asked for directly while triaging a PyPy fleet: can fusil record a backtrace on SEGV, so we
capture the problem when it happens?

The gap

faulthandler.enable() was emitted only under --oom-fuzz. In every other mode a
SIGSEGV/SIGABRT left the session's stdout simply stopping at the last call line, with no
indication of where the target died:

[c18m11] Queue._init()
Starting thread: [c18m12] Queue.__eq__()
        <- process gone, that's all we get

This is worst for exactly the crashes that need it most. Two SEGVs in the PyPy stdlib fleet
(multiprocessing.dummy, asyncio.queues) are rare, heavily threaded and load-dependent;
10+ standalone replays never reproduced either, so there will never be a second chance to
observe them. The crash-time dump is the only evidence available.

The change

faulthandler.enable() is now emitted first in the prelude, before any other import, so
a crash while the prelude itself is still running is caught too. Wrapped in try/except
not every target interpreter ships a working faulthandler. The two OOM blocks no longer
enable it themselves; the prelude covers every mode.

--no-faulthandler opts out, for fuzzing signal handling itself or if it ever perturbs
--tsan race signatures (that campaign's dedup format is a cross-repo contract, so it gets
an escape hatch).

Verified live

PyPy 3.11.15, real generated script, SEGV raised inside a worker thread:

Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File ".../_ctypes/function.py", line 394 in _call_funcptr
  File ".../_ctypes/function.py", line 299 in __call__
  File ".../session-1/source.py", line 9770 in <lambda>      <- the exact fuzzed call
  File ".../threading.py", line 971 in run

It names the crashing source.py line from inside the thread. Bonus: fusil already scores
Fatal Python error at 1.0, so these sessions stop depending on the signal probe alone.

One trap worth flagging

A bare MagicMock() options stub auto-vivifies a truthy attribute, which reads as
--no-faulthandler and silently suppresses the block — the same hazard
test_tsan_generation already documents in a comment. test_oom_fuzz and
test_target_introspect now pin it explicitly, as they already do for no_numpy.

Five tests cover default-on, ordering-before-other-imports, the try/except guard, that the
emitted prelude still parses (it is spliced into a dedent()ed f-string, where a wrong
indent would break every generated script), and the opt-out. Verified each fails without the
change. Full suite green (1260); ruff clean; golden regenerated.

🤖 Generated with Claude Code

A target that dies on a fatal signal used to leave stdout ending at the last call line,
with no indication of WHERE it died -- faulthandler was enabled only under --oom-fuzz.
That is worst for exactly the crashes that need it most: the rare, threaded,
load-dependent ones that do not reproduce on a later replay, where the crash-time dump is
the only evidence there will ever be. Two such SEGVs in a PyPy stdlib fleet
(multiprocessing.dummy, asyncio.queues) are unattributable for this reason -- 10+ standalone
replays never reproduced either.

Now emitted first in the prelude, before any other import, so a crash while the prelude is
still running is caught too. Guarded in try/except: not every target interpreter ships a
working faulthandler. --no-faulthandler opts out, for fuzzing signal handling itself or if
it ever perturbs --tsan race signatures.

The two OOM blocks no longer enable it themselves; the prelude covers every mode.

Verified on PyPy 3.11.15 with a real generated script: a SEGV raised inside a worker thread
now dumps the full stack, naming the exact source.py line of the crashing call. fusil
already scores "Fatal Python error" at 1.0, so such sessions also stop depending on the
signal probe alone.

Note for MagicMock-based option stubs: a bare MagicMock auto-vivifies a TRUTHY attribute,
which reads as --no-faulthandler and silently suppresses the block. test_oom_fuzz and
test_target_introspect now pin it explicitly, as they already do for no_numpy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhcpLoyjUWLbETGZnA9boj
@devdanzin
devdanzin merged commit cfed36e into main Aug 19, 2026
1 of 2 checks passed
@devdanzin
devdanzin deleted the faulthandler-always branch August 19, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant