fix: place a model too big for the box, and stop instead of silently … - #107
Merged
Merged
Conversation
…going uniform pollard-probe pinned the whole model to one device, so the first model bigger than the accelerator OOM'd (Gemma4 12B: 22.3GB onto a 16GB Mac). plan_placement measures the weights first: fits the accelerator -> use it; fits RAM -> CPU; fits neither -> shard and offload the tail to disk. --device now defaults to auto (cuda > mps > cpu) rather than mps. Both callers then swallowed the failure. _ensure_sensitivity fell back to a uniform allocation, which pollard-fit itself warns has no quality win, and _ensure_imatrix never checked llama-imatrix's exit code -- so a missing imatrix stayed invisible until llama-quantize could not open it. Both now stop. Uniform is available on purpose via --no-measure. --ngl gains 'auto': offload the share that fits and stream the rest, instead of a static 99 that is fatal on a model bigger than the accelerator. Apple Silicon is unified memory, so the GPU and CPU budgets must not be summed. Windows has neither sysconf nor /proc, so detect_available_ram_gb returned None there and every budget decision on the build box was made blind; both paths now read RAM on all three platforms. Tests: 4 regressions (63/63).
pollard_probe imports torch at module scope, so importing it at the top of a test hard-fails on CI. The cross-platform memory assertions are a source scan and need no torch at all -- CI is precisely the machine that would otherwise let a POSIX-only probe through -- so those now run unconditionally, and only the live calls sit behind the suite's existing torch skip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…going uniform
pollard-probe pinned the whole model to one device, so the first model bigger than the accelerator OOM'd (Gemma4 12B: 22.3GB onto a 16GB Mac). plan_placement measures the weights first: fits the accelerator -> use it; fits RAM -> CPU; fits neither -> shard and offload the tail to disk. --device now defaults to auto (cuda > mps > cpu) rather than mps.
Both callers then swallowed the failure. _ensure_sensitivity fell back to a uniform allocation, which pollard-fit itself warns has no quality win, and _ensure_imatrix never checked llama-imatrix's exit code -- so a missing imatrix stayed invisible until llama-quantize could not open it. Both now stop. Uniform is available on purpose via --no-measure.
--ngl gains 'auto': offload the share that fits and stream the rest, instead of a static 99 that is fatal on a model bigger than the accelerator.
Apple Silicon is unified memory, so the GPU and CPU budgets must not be summed. Windows has neither sysconf nor /proc, so detect_available_ram_gb returned None there and every budget decision on the build box was made blind; both paths now read RAM on all three platforms.
Tests: 4 regressions (63/63).