load: one backbone loader, so a vision-language model stops being unr… - #102
Merged
Merged
Conversation
…eachable Eleven tools called AutoModelForCausalLM.from_pretrained directly. That auto class REFUSES a vision-language config outright -- "Unrecognized configuration class Qwen2VLConfig for this kind of AutoModel" -- so every one of them died on a VL model, even though nothing they do afterwards cares. Probing, smoothing, abliterating, KL and low-bit all work on the text stack, which a VL model keeps under model.language_model. It reads as an unsupported model rather than an unasked question, and it put a whole quant ladder in the ditch: pollard-probe could not measure sensitivity on Qwen2-VL, so the build produced one rung and stopped. pollard_load.load_backbone tries the causal LM, falls back to AutoModelForImageTextToText and AutoModelForVision2Seq, and reports ALL the failures rather than only the last -- a fallback chain that hides the first error is its own debugging problem. text_layers() finds the decoder stack for either family, so tools that count or iterate layers get the same answer on both. Every tool now uses it: probe, abliterate, gptq, hf_smooth, kl, lowbit, palette, doctor, refcheck. pollard_flybrain delegates to it too, so there is ONE implementation rather than two that drift. Verified against the model that broke the ladder: VL model loaded: Qwen2VLForConditionalGeneration text layers found: 28 A test walks every pollard_*.py and fails if any of them loads a backbone directly again, because a second copy is exactly how this comes back. 56/56 pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…eachable
Eleven tools called AutoModelForCausalLM.from_pretrained directly. That auto class REFUSES a vision-language config outright -- "Unrecognized configuration class Qwen2VLConfig for this kind of AutoModel" -- so every one of them died on a VL model, even though nothing they do afterwards cares. Probing, smoothing, abliterating, KL and low-bit all work on the text stack, which a VL model keeps under model.language_model.
It reads as an unsupported model rather than an unasked question, and it put a whole quant ladder in the ditch: pollard-probe could not measure sensitivity on Qwen2-VL, so the build produced one rung and stopped.
pollard_load.load_backbone tries the causal LM, falls back to AutoModelForImageTextToText and AutoModelForVision2Seq, and reports ALL the failures rather than only the last -- a fallback chain that hides the first error is its own debugging problem. text_layers() finds the decoder stack for either family, so tools that count or iterate layers get the same answer on both.
Every tool now uses it: probe, abliterate, gptq, hf_smooth, kl, lowbit, palette, doctor, refcheck. pollard_flybrain delegates to it too, so there is ONE implementation rather than two that drift.
Verified against the model that broke the ladder:
VL model loaded: Qwen2VLForConditionalGeneration
text layers found: 28
A test walks every pollard_*.py and fails if any of them loads a backbone directly again, because a second copy is exactly how this comes back.
56/56 pass.