Skip to content

Add script to predict and mark required phrases in license rules - #5262

Draft
Kaushik-Kumar-CEG wants to merge 10 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/inject-ml-phrases
Draft

Add script to predict and mark required phrases in license rules#5262
Kaushik-Kumar-CEG wants to merge 10 commits into
aboutcode-org:developfrom
Kaushik-Kumar-CEG:gsoc/inject-ml-phrases

Conversation

@Kaushik-Kumar-CEG

@Kaushik-Kumar-CEG Kaushik-Kumar-CEG commented Aug 9, 2026

Copy link
Copy Markdown

part of #5243. Adds the tool that runs the trained tagger from #5222 over license
rules and marks the required phrases it predicts with {{ }}

74% of license expressions have no required phrase coverage, this fills them in

this PR:

  • add_ml_phrases.py - picks the rules that have no phrases, predicts the spans
    and marks them. --model (local dir or hf id), --license-expression,
    --dry-run, --limit, --verbose. fp32, cpu or gpu
  • test_add_ml_phrases.py - unit tests, the tagger and tokenizer are faked so
    they need no weights and no network

rule loading, validation and marking all go through the existing helpers in
required_phrases.py, and the words come from required_phrase_splitter so they
line up with what the model was trained on. nothing in src/ changes

needs #5222 for PhraseTagger and the label scheme. it never reindexes, it just
prints a reminder once it has written something

notes

rules are loaded once instead of calling get_updatable_rules_by_expression per
expression, that reparses all 36k rule files on each call and returns nothing when
passed None. i can fix both there in a separate PR if you prefer

rules that already have {{ }} are skipped, 9875 of them. the braces are tokens
too so they shift every word after them and the tags stop lining up

checks

--dry-run over apache-2.0, mit, gpl-2.0 and bsd-new leaves every rule file
unchanged. a real write reloads fine, keeps the ignorables, and the rule is
skipped on a second run so an interrupted run resumes. feeding the gold labels
back in as a perfect prediction, every phrase is still found by
find_phrase_spans_in_text and 97% pass is_good

the checkpoint is not published yet, so the download path in load_model has not
been run against real weights

disclosure : used Claude to help review and clean up bugs in script

loads the rules once instead of calling get_updatable_rules_by_expression per
expression, that reparses every rule file each time
reuses validate_and_reindex for the checks after a write, and still prints the
reindex reminder when it is not asked to do it
the tagger and tokenizer are faked so this needs no weights and no network
a checkpoint with the same number of labels in another order would map every
prediction to the wrong tag
printing the reminder is enough, reindexing is the caller's call
aux_ce_weight registers class_weights during training, a strict load into the
inference model choked on it
@Kaushik-Kumar-CEG Kaushik-Kumar-CEG changed the title Add tool to mark required phrases using the trained tagger Add script to predict and mark required phrases in license rules Aug 9, 2026
@Kaushik-Kumar-CEG
Kaushik-Kumar-CEG marked this pull request as draft August 9, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant