Repository navigation
Conversation
A missing newline merged meta-llama/Llama-3.2-1B and BAAI/Aquila-7B into one entry, so neither model was being checked.
Replace the GitHub launcher, which drove a Studio through lightning_sdk with LIGHTNING_USER_ID/LIGHTNING_API_KEY, by a native .lightning workflow with one L4 job per script. hf_llm.py now uses an ungated model so no HF token is needed.
The coverage script builds models on the meta device and only runs the prologue with the pytorch executor, so it does not need a GPU. Shard the model list across GitHub runners instead of launching a Studio with Lightning keys, and merge the shard results for the Slack report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
CI quickstartandCI jit coveragehave failed on every run since early May: they drive a Lightning Studio from a GitHub runner withLIGHTNING_USER_ID/LIGHTNING_API_KEY, and the very firstupload_foldercall now returns 404. This moves both off personal Lightning keys..lightning/workflows/quickstart.yaml): one L4 job per example script, on the same thunder image as the other GPU workflows.hf_llm.pyswitches to the ungatedDeepSeek-R1-Distill-Qwen-1.5B, so no HF token is needed.metadevice and only runs the prologue with the pytorch executor, so the GPU was never used. The model list is sharded 4 ways, and a final job merges the shard results for the Slack report.examples/coverage/all.txtthat merged two model names into one entry.Trade-offs and notes
main(5 short L4 jobs) instead of nightly plus path-filtered PRs. It also has no Slack summary any more; results show as GitHub checks.HF_TOKENis not available, so the ~20 gated Llama/Gemma configs show as[SKIPPED]there;mainand nightly runs still use the secret.ci-benchmark.ymlstill uses the Lightning keys but has been disabled since Oct 2025; it is left untouched.