Skip to content

load: one backbone loader, so a vision-language model stops being unr… - #102

Merged
WestWaters merged 1 commit into
mainfrom
fix/vl-probe-and-ladder
Sep 16, 2026
Merged

WestWaters merged 1 commit into
mainfrom
fix/vl-probe-and-ladder

Conversation

@WestWaters

Copy link
Copy Markdown
Owner

…eachable

Eleven tools called AutoModelForCausalLM.from_pretrained directly. That auto class REFUSES a vision-language config outright -- "Unrecognized configuration class Qwen2VLConfig for this kind of AutoModel" -- so every one of them died on a VL model, even though nothing they do afterwards cares. Probing, smoothing, abliterating, KL and low-bit all work on the text stack, which a VL model keeps under model.language_model.

It reads as an unsupported model rather than an unasked question, and it put a whole quant ladder in the ditch: pollard-probe could not measure sensitivity on Qwen2-VL, so the build produced one rung and stopped.

pollard_load.load_backbone tries the causal LM, falls back to AutoModelForImageTextToText and AutoModelForVision2Seq, and reports ALL the failures rather than only the last -- a fallback chain that hides the first error is its own debugging problem. text_layers() finds the decoder stack for either family, so tools that count or iterate layers get the same answer on both.

Every tool now uses it: probe, abliterate, gptq, hf_smooth, kl, lowbit, palette, doctor, refcheck. pollard_flybrain delegates to it too, so there is ONE implementation rather than two that drift.

Verified against the model that broke the ladder:

VL model loaded: Qwen2VLForConditionalGeneration
text layers found: 28

A test walks every pollard_*.py and fails if any of them loads a backbone directly again, because a second copy is exactly how this comes back.

56/56 pass.

…eachable

Eleven tools called AutoModelForCausalLM.from_pretrained directly. That auto class REFUSES a
vision-language config outright -- "Unrecognized configuration class Qwen2VLConfig for this kind of
AutoModel" -- so every one of them died on a VL model, even though nothing they do afterwards cares.
Probing, smoothing, abliterating, KL and low-bit all work on the text stack, which a VL model keeps
under model.language_model.

It reads as an unsupported model rather than an unasked question, and it put a whole quant ladder in
the ditch: pollard-probe could not measure sensitivity on Qwen2-VL, so the build produced one rung
and stopped.

pollard_load.load_backbone tries the causal LM, falls back to AutoModelForImageTextToText and
AutoModelForVision2Seq, and reports ALL the failures rather than only the last -- a fallback chain
that hides the first error is its own debugging problem. text_layers() finds the decoder stack for
either family, so tools that count or iterate layers get the same answer on both.

Every tool now uses it: probe, abliterate, gptq, hf_smooth, kl, lowbit, palette, doctor, refcheck.
pollard_flybrain delegates to it too, so there is ONE implementation rather than two that drift.

Verified against the model that broke the ladder:

  VL model loaded: Qwen2VLForConditionalGeneration
  text layers found: 28

A test walks every pollard_*.py and fails if any of them loads a backbone directly again, because a
second copy is exactly how this comes back.

56/56 pass.
@WestWaters
WestWaters merged commit a8e84fc into main Sep 16, 2026
1 check passed
@WestWaters
WestWaters deleted the fix/vl-probe-and-ladder branch September 16, 2026 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant