[TEST] attribute-based extraction — CI matrix run (do not merge)#1
Closed
oliver-kriska wants to merge 2 commits into
Closed
[TEST] attribute-based extraction — CI matrix run (do not merge)#1oliver-kriska wants to merge 2 commits into
oliver-kriska wants to merge 2 commits into
Conversation
mix gettext.extract can only discover messages by force-recompiling the whole project, because extraction happens during macro expansion and the results only exist in the extractor agent for the duration of that compilation. On large projects this dominates extraction time. Implement the design proposed by @maennchen in elixir-gettext#373: during normal compilation, when the current Mix environment is included in the new :extraction_environments gettext configuration (default [:dev]), the gettext macros persist each message into an accumulated, persisted @__gettext_messages__ attribute of the calling module, and backends persist a @__gettext_backend_module__ marker. A new experimental --from-attributes flag on mix gettext.extract then runs a normal incremental compile, reads the persisted entries back from the compiled BEAM files with :beam_lib (without loading modules), feeds them into the existing extractor agent, and reuses the unchanged Gettext.Extractor.pot_files/2 merge logic. Staleness needs no bookkeeping: changed files are recompiled by the incremental compile and get fresh attributes, and deleted files' beams are pruned by the compiler. --check-up-to-date composes with the new flag and gets the same speedup. Environments not listed in :extraction_environments (such as :prod) persist nothing, so release artifacts are unaffected. If the scan finds no persisted messages or backends at all, the task raises with instructions instead of silently producing empty output. The force-recompile path is untouched and remains the default. Old-path and new-path POT output is byte-identical, covered by tests including macro-generated messages (gettext calls injected into modules that do not literally mention them) and plural/context/comment/noop variants.
|
Ready to review this PR? Stage has broken it down into 4 individual chapters for you: 
Chapters generated by Stage for commit d538cb4 on Jun 10, 2026 1:49pm UTC. |
Address review findings on the --from-attributes prototype:
* Validate the shape of persisted message entries when scanning BEAM
files, skipping malformed ones instead of crashing the scan with a
CaseClauseError (an unrelated module could carry an attribute with
the same name, and future format changes need a safe path).
* Reenable "compile" and friends in the incremental compile, so that
extraction still picks up source changes when the compile tasks
already ran in the current VM (for example, through a task alias).
* Guard the backend marker registration with Module.has_attribute?/2,
matching persist_message/6.
Test improvements: assert fixed content on the rebuilt POT (not only
equality with the recompilation path), reset the extractor agent's full
initial state between tests, make reference line assertions robust to
fixture edits, pin the stale-POT filename in the --check-up-to-date
assertion, and restore Code.compiler_options/1 after the suite.
New coverage: _with_backend macro variants, reference merging when two
modules share a msgid, --from-attributes combined with --merge, custom
:priv backends, and extraction across umbrella apps (each app gets its
own POT through the task's recursion).
oliver-kriska
force-pushed
the
attribute-based-extraction
branch
from
2e8808e to
d538cb4
Compare
Owner
Author
|
Superseded by the upstream PR elixir-gettext#437, whose CI runs the full matrix including the Elixir 1.16/OTP 24 floor. No longer needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fork-internal draft PR. Not for merging into fork main (fork main tracks upstream). Two purposes:
attribute-based-extractionbranch — green on both legs (1.16/OTP 24.2 and 1.18/OTP 27.2).Status: in active production-app trial
This branch is currently pinned (immutable ref) as the gettext dependency of our larger commercial Phoenix app, where it is being used day-to-day for all extraction work — local
gettext.extract --merge, the LLM translation pipeline, and the per-PR CI up-to-date check — ahead of the upstream submission.Real-world benchmarks (the trial app)
Testbed: 1,875 source files; 4,713 msgids across 11 gettext domains × 16 locales (176 PO files), incl. macro-generated gettext calls and HEEx templates. Elixir 1.20.1 / OTP 29.0.2, Apple Silicon. Branch @ 2e8808e.
--from-attributesmix gettext.extract --mergemix gettext.extract --check-up-to-date(local)Correctness: equivalence under mutation
Beyond steady-state runs (which stay byte-clean and idempotent), six mutation classes were each applied to source and run through both paths (new path first, so its incremental-compile detection is what finds the change), with the resulting PO/POT diffs compared byte-for-byte:
gettext()call (.ex)The batched mutations produced 1,322 changed lines across the 176-PO catalog — identical from both paths (new path: 9.1s incl. incremental compile of the changed files; old path: 65.3s).
Also caught only by real-app testing:
Extractor.extracting?()returnsnil(notfalse) when the agent isn't running, which crashed a dependency compile under strict boolean ops in the first iteration — fixed on this branch.Note: the sibling branch
parallel-locale-mergewas merged upstream as elixir-gettext#436. This branch predates that merge; it will be rebased onto upstream main before the upstream PR is opened.