Add script to predict and mark required phrases in license rules - #5262
Draft
Kaushik-Kumar-CEG wants to merge 10 commits into
Draft
Add script to predict and mark required phrases in license rules#5262Kaushik-Kumar-CEG wants to merge 10 commits into
Kaushik-Kumar-CEG wants to merge 10 commits into
Conversation
loads the rules once instead of calling get_updatable_rules_by_expression per expression, that reparses every rule file each time
reuses validate_and_reindex for the checks after a write, and still prints the reindex reminder when it is not asked to do it
the tagger and tokenizer are faked so this needs no weights and no network
a checkpoint with the same number of labels in another order would map every prediction to the wrong tag
printing the reminder is enough, reindexing is the caller's call
aux_ce_weight registers class_weights during training, a strict load into the inference model choked on it
Kaushik-Kumar-CEG
force-pushed
the
gsoc/inject-ml-phrases
branch
from
August 9, 2026 13:32
7d37348 to
82cca13
Compare
Kaushik-Kumar-CEG
marked this pull request as draft
August 9, 2026 16:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of #5243. Adds the tool that runs the trained tagger from #5222 over license
rules and marks the required phrases it predicts with
{{ }}74% of license expressions have no required phrase coverage, this fills them in
this PR:
add_ml_phrases.py- picks the rules that have no phrases, predicts the spansand marks them.
--model(local dir or hf id),--license-expression,--dry-run,--limit,--verbose. fp32, cpu or gputest_add_ml_phrases.py- unit tests, the tagger and tokenizer are faked sothey need no weights and no network
rule loading, validation and marking all go through the existing helpers in
required_phrases.py, and the words come fromrequired_phrase_splitterso theyline up with what the model was trained on. nothing in
src/changesneeds #5222 for
PhraseTaggerand the label scheme. it never reindexes, it justprints a reminder once it has written something
notes
rules are loaded once instead of calling
get_updatable_rules_by_expressionperexpression, that reparses all 36k rule files on each call and returns nothing when
passed
None. i can fix both there in a separate PR if you preferrules that already have
{{ }}are skipped, 9875 of them. the braces are tokenstoo so they shift every word after them and the tags stop lining up
checks
--dry-runover apache-2.0, mit, gpl-2.0 and bsd-new leaves every rule fileunchanged. a real write reloads fine, keeps the ignorables, and the rule is
skipped on a second run so an interrupted run resumes. feeding the gold labels
back in as a perfect prediction, every phrase is still found by
find_phrase_spans_in_textand 97% passis_goodthe checkpoint is not published yet, so the download path in
load_modelhas notbeen run against real weights
disclosure : used Claude to help review and clean up bugs in script