enable faulthandler in every generated script, not just OOM mode - #261
Merged
Conversation
A target that dies on a fatal signal used to leave stdout ending at the last call line, with no indication of WHERE it died -- faulthandler was enabled only under --oom-fuzz. That is worst for exactly the crashes that need it most: the rare, threaded, load-dependent ones that do not reproduce on a later replay, where the crash-time dump is the only evidence there will ever be. Two such SEGVs in a PyPy stdlib fleet (multiprocessing.dummy, asyncio.queues) are unattributable for this reason -- 10+ standalone replays never reproduced either. Now emitted first in the prelude, before any other import, so a crash while the prelude is still running is caught too. Guarded in try/except: not every target interpreter ships a working faulthandler. --no-faulthandler opts out, for fuzzing signal handling itself or if it ever perturbs --tsan race signatures. The two OOM blocks no longer enable it themselves; the prelude covers every mode. Verified on PyPy 3.11.15 with a real generated script: a SEGV raised inside a worker thread now dumps the full stack, naming the exact source.py line of the crashing call. fusil already scores "Fatal Python error" at 1.0, so such sessions also stop depending on the signal probe alone. Note for MagicMock-based option stubs: a bare MagicMock auto-vivifies a TRUTHY attribute, which reads as --no-faulthandler and silently suppresses the block. test_oom_fuzz and test_target_introspect now pin it explicitly, as they already do for no_numpy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WhcpLoyjUWLbETGZnA9boj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Asked for directly while triaging a PyPy fleet: can fusil record a backtrace on SEGV, so we
capture the problem when it happens?
The gap
faulthandler.enable()was emitted only under--oom-fuzz. In every other mode aSIGSEGV/SIGABRT left the session's stdout simply stopping at the last call line, with no
indication of where the target died:
This is worst for exactly the crashes that need it most. Two SEGVs in the PyPy stdlib fleet
(
multiprocessing.dummy,asyncio.queues) are rare, heavily threaded and load-dependent;10+ standalone replays never reproduced either, so there will never be a second chance to
observe them. The crash-time dump is the only evidence available.
The change
faulthandler.enable()is now emitted first in the prelude, before any other import, soa crash while the prelude itself is still running is caught too. Wrapped in
try/except—not every target interpreter ships a working faulthandler. The two OOM blocks no longer
enable it themselves; the prelude covers every mode.
--no-faulthandleropts out, for fuzzing signal handling itself or if it ever perturbs--tsanrace signatures (that campaign's dedup format is a cross-repo contract, so it getsan escape hatch).
Verified live
PyPy 3.11.15, real generated script, SEGV raised inside a worker thread:
It names the crashing
source.pyline from inside the thread. Bonus: fusil already scoresFatal Python errorat 1.0, so these sessions stop depending on the signal probe alone.One trap worth flagging
A bare
MagicMock()options stub auto-vivifies a truthy attribute, which reads as--no-faulthandlerand silently suppresses the block — the same hazardtest_tsan_generationalready documents in a comment.test_oom_fuzzandtest_target_introspectnow pin it explicitly, as they already do forno_numpy.Five tests cover default-on, ordering-before-other-imports, the try/except guard, that the
emitted prelude still parses (it is spliced into a
dedent()ed f-string, where a wrongindent would break every generated script), and the opt-out. Verified each fails without the
change. Full suite green (1260); ruff clean; golden regenerated.
🤖 Generated with Claude Code