Dynamic VRAM: a ModelSamplingDiscrete schedule leaks into later runs of the same checkpoint (buffers restored by attribute path), giving wrong or black images reported as success

[wwoosshh]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (reproduced with --disable-all-custom-nodes)

Expected Behavior

A run samples with the schedule of the nodes in that run. The same prompt (same checkpoint, seed and settings) without a ModelSamplingDiscrete node should give the same image whether or not a workflow with that node ran earlier in the session.

Actual Behavior

With dynamic VRAM on (the default here), after running a workflow that uses ModelSamplingDiscrete on a checkpoint, later runs of that checkpoint without the node keep sampling with the node's schedule. Every run is reported as a success, and it lasts until a restart.

Plain txt2img on an SDXL eps checkpoint, run after a ModelSamplingDiscrete(v_prediction, zsnr=true) workflow (queued three times in the measured session), compared with the same prompt as the first run of a fresh session (mean absolute pixel difference, 0-255):

seed	dynamic VRAM (default)	same, --disable-all-custom-nodes	--disable-dynamic-vram
11 (first plain run after the node run)	55.84 (another image)	55.84	identical
22	163.70 (black)	163.70 (black)	1.40
33	171.40 (black)	171.40 (black)	0.42

The black images are NaN (RuntimeWarning: invalid value encountered in cast at SaveImage). Inspecting the model after the node run showed that the checkpoint's own model_sampling object had sigma_max 4518.8 (the zsnr schedule) instead of 14.6.

It also happens the other way round. If a node is used after a plain run of the same checkpoint, it is silently ignored from its second run on. I measured this on NoobAI-XL-Vpred 1.0 (v-prediction, zsnr), comparing plain runs with runs through ModelSamplingDiscrete(v_prediction, zsnr=false):

• Plain first, then the node: the node's second and third images were pixel-identical to the plain images of the same seeds. The first node run was a different image (58.73/255 from the node's fresh-session image).
• Node first, then plain: the mirror image of the above. The plain runs took the node's schedule (64.39 for the first, then pixel-identical to the node's images).

Steps to Reproduce

1. Start ComfyUI with its defaults (dynamic VRAM on), optionally with --disable-all-custom-nodes.
2. Workflow A: CheckpointLoaderSimple (any SDXL eps checkpoint; I used waiIllustriousSDXL v1.70), then ModelSamplingDiscrete (sampling v_prediction, zsnr on), then KSampler (euler, normal, 25 steps, cfg 5, 1024x1024, seed 11), then VAEDecode and SaveImage. Queue it. I queued it three times (seeds 11, 22, 33); from the mechanism below, once should be enough, but I only measured after three. Its image is flat grey. That is expected, because the node does not fit an eps model; it is not the bug.
3. Workflow B: the same graph without ModelSamplingDiscrete. Queue it with seeds 11, 22 and 33.
4. Restart and queue B first, with the same seeds. Compare: in step 3, seed 11 is a different image, and 22 and 33 are black.

API-format prompts are below. B is A with node 9 removed and KSampler.model set to ["1", 0].

Workflow A (API format)

{
 "1": {
  "class_type": "CheckpointLoaderSimple",
  "inputs": {
   "ckpt_name": "waiIllustriousSDXL_v170.safetensors"
  }
 },
 "2": {
  "class_type": "CLIPTextEncode",
  "inputs": {
   "text": "masterpiece, best quality, 1girl, solo, smile, school uniform, cherry blossoms, outdoors, sunlight",
   "clip": [
    "1",
    1
   ]
  }
 },
 "3": {
  "class_type": "CLIPTextEncode",
  "inputs": {
   "text": "lowres, bad anatomy, worst quality, low quality",
   "clip": [
    "1",
    1
   ]
  }
 },
 "4": {
  "class_type": "EmptyLatentImage",
  "inputs": {
   "width": 1024,
   "height": 1024,
   "batch_size": 1
  }
 },
 "5": {
  "class_type": "KSampler",
  "inputs": {
   "model": [
    "9",
    0
   ],
   "positive": [
    "2",
    0
   ],
   "negative": [
    "3",
    0
   ],
   "latent_image": [
    "4",
    0
   ],
   "seed": 11,
   "steps": 25,
   "cfg": 5.0,
   "sampler_name": "euler",
   "scheduler": "normal",
   "denoise": 1.0
  }
 },
 "6": {
  "class_type": "VAEDecode",
  "inputs": {
   "samples": [
    "5",
    0
   ],
   "vae": [
    "1",
    2
   ]
  }
 },
 "7": {
  "class_type": "SaveImage",
  "inputs": {
   "images": [
    "6",
    0
   ],
   "filename_prefix": "ComfyUI_leak_A"
  }
 },
 "9": {
  "class_type": "ModelSamplingDiscrete",
  "inputs": {
   "model": [
    "1",
    0
   ],
   "sampling": "v_prediction",
   "zsnr": true
  }
 }
}

Debug Logs

This is the full log of steps 1-3 with --disable-all-custom-nodes. The first three prompts are A and the last three are B, each with seeds 11, 22 and 33. Progress-bar redraws are trimmed and local paths are shortened to <ComfyUI>.

log

[INFO] Found comfy_kitchen backend cuda: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Found comfy_kitchen backend triton: {'available': False, 'disabled': True, 'unavailable_reason': "ImportError: No module named 'triton'", 'capabilities': []}
[INFO] Found comfy_kitchen backend hip: {'available': False, 'disabled': False, 'unavailable_reason': 'PyTorch ROCm/HIP runtime not available', 'capabilities': []}
[INFO] Found comfy_kitchen backend eager: {'available': True, 'disabled': False, 'unavailable_reason': None, 'capabilities': ['adaln', 'apply_rope', 'apply_rope1', 'apply_rope1_', 'apply_rope_', 'apply_rope_split_half', 'apply_rope_split_half1', 'apply_rope_split_half1_', 'apply_rope_split_half_', 'convrot_w4a4_linear', 'dequantize_convrot_w4a4_weight', 'dequantize_int8_convrot_weight', 'dequantize_int8_convrot_weight_dtype', 'dequantize_int8_embedding', 'dequantize_int8_simple', 'dequantize_int8_simple_dtype', 'dequantize_mxfp8', 'dequantize_nvfp4', 'dequantize_per_tensor_fp8', 'dequantize_w4a8_int8_weight', 'gemv_awq_w4a16', 'int8_linear', 'na3d', 'prepare_int4_weight_for_int8_linear', 'quantize_and_rotate_rowwise', 'quantize_convrot_w4a4_weight', 'quantize_int8_convrot_weight', 'quantize_int8_rowwise', 'quantize_int8_tensorwise', 'quantize_mxfp8', 'quantize_nvfp4', 'quantize_per_tensor_fp8', 'quantize_svdquant_w4a4', 'quantize_w4a8_int8_weight', 'rms_adaln', 'rms_rope', 'rms_rope1', 'rms_rope1_', 'rms_rope_', 'rms_rope_split_half', 'rms_rope_split_half1', 'rms_rope_split_half1_', 'rms_rope_split_half_', 'rotate_int8_convrot_weight', 'scaled_mm_mxfp8', 'scaled_mm_nvfp4', 'scaled_mm_svdquant_w4a4', 'stochastic_rounding_fp8', 'w4a8_int8_linear']}
[INFO] Checkpoint files will always be loaded safely.
[INFO] Total VRAM 12282 MB, total RAM 32532 MB
[INFO] pytorch version: 2.13.0+cu130
[INFO] Set vram state to: NORMAL_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 4070 Ti : cudaMallocAsync
[INFO] Using async weight offloading with 2 streams
[INFO] Enabled pinned memory 13012.0
[INFO] Using pytorch attention
[INFO] aimdo: src-win/cuda-detour.c:38:INFO:aimdo_setup_hooks: installing 6 hooks
[INFO] aimdo: src/control.c:262:INFO:comfy-aimdo NVML pressure enabled
[INFO] aimdo: src-win/shmem-detect.c:80:INFO:comfy-aimdo WDDM adapter match: NVIDIA GeForce RTX 4070 Ti runtime_luid=00000000:00011b16 dxgi_luid=00000000:00011b16
[INFO] aimdo: src/control.c:277:INFO:comfy-aimdo inited for GPU: NVIDIA GeForce RTX 4070 Ti (VRAM: 12281 MB)
[INFO] DynamicVRAM support detected and enabled
[INFO] Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr  8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
[INFO] ComfyUI version: 0.34.1
[INFO] comfy-aimdo version: 0.4.15
[INFO] comfy-kitchen version: 0.2.31
[INFO] comfyui-frontend-package version: 1.49.6
[INFO] comfyui-workflow-templates version: 0.11.48
[INFO] comfyui-embedded-docs version: 0.5.10
[INFO] comfy-kitchen version: 0.2.31
[INFO] comfy-aimdo version: 0.4.15
[INFO] [Prompt Server] web root: <ComfyUI>\.venv\Lib\site-packages\comfyui_frontend_package\static
[INFO] Asset seeder disabled
[INFO] No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
[INFO] Skipping loading of custom nodes
[INFO] Context impl SQLiteImpl.
[INFO] Will assume non-transactional DDL.
[INFO] Using RAM pressure cache.
[INFO] Starting server
[INFO] To see the GUI go to: http://127.0.0.1:8189
[INFO] got prompt
[INFO] model weight dtype torch.float16, manual cast: None
[INFO] model_type EPS
[INFO] Using pytorch attention in VAE
[INFO] Using pytorch attention in VAE
[INFO] VAE load device: cuda:0, offload device: cpu, dtype: torch.float16
[INFO] CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
[INFO] Requested to load SDXLClipModel
[INFO] Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached. Force pre-loaded 180 weights: 400 KB.
[INFO] Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached. Force pre-loaded 180 weights: 400 KB.
[INFO] Requested to load SDXL
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.54it/s]
[INFO] Requested to load AutoencoderKL
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
[INFO] Prompt executed in 11.76 seconds
[INFO] got prompt
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.45it/s]
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
[INFO] Prompt executed in 6.17 seconds
[INFO] got prompt
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.44it/s]
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
[INFO] Prompt executed in 6.20 seconds
[INFO] got prompt
[INFO] Requested to load SDXL
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.47it/s]
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
[INFO] Prompt executed in 6.18 seconds
[INFO] got prompt
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.45it/s]
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
<ComfyUI>\nodes.py:1698: RuntimeWarning: invalid value encountered in cast
  img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))
[INFO] Prompt executed in 6.12 seconds
[INFO] got prompt
[INFO] Model SDXL prepared for dynamic VRAM loading. 4896MB Staged. 0 patches attached. Force pre-loaded 512 weights: 1197 KB.
100%|██████████| 25/25 [00:05<00:00,  4.44it/s]
[INFO] 0 models unloaded.
[INFO] Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached. Force pre-loaded 112 weights: 91 KB.
[INFO] Prompt executed in 6.14 seconds

Other

Cause, as far as I can tell from comfy/model_patcher.py:

ModelPatcherDynamic.load() backs up every buffer by attribute path:

for key, buf in self.model.named_buffers(recurse=True):
    if key not in self.backup_buffers:
        self.backup_buffers[key] = buf

restore_loaded_backups(), called at the start of the next load(), writes them back by path:

for key in list(self.backup_buffers.keys()):
    comfy.utils.set_attr_buffer(self.model, key, self.backup_buffers.pop(key))

• backup_buffers is shared by all clones.
• ModelSamplingDiscrete puts its own object at model_sampling through an object patch. So do ModelSamplingSD3, ModelSamplingAuraFlow, ModelSamplingFlux, and others.
• partially_load() runs unpatch_model(), then patch_model(), then load(). By the time restore_loaded_backups() runs, a different object sits at model_sampling, and it receives the previous object's sigmas and log_sigmas.
• From then on, the object keeps the other schedule. This covers the checkpoint's own object in one direction and the node's object in the other.
• Before sampling, the sampler computes its sigmas from get_model_object("model_sampling"). Inside apply_model, timestep() then uses the overwritten log_sigmas. That mismatch explains why the first run after the switch is a different image rather than simply the other schedule.

These lines are unchanged in v0.37.0 and on master: restore_loaded_backups and the backup loop. I have only run 0.34.1, so I cannot confirm the behaviour on master.

A workaround that works on 0.34.1 (an external patch, not a PR):

• After load(), remember which module each backed-up buffer came from.
• In restore_loaded_backups(), if a different module now sits at that path, hand the backup back to its own module instead of writing it into the new one.

With that change, both directions matched a fresh session pixel for pixel in the tests above. Keying the backup by module inside ModelPatcherDynamic would amount to the same thing. I found this while testing entail, which applies that patch.

Possibly related: #15452 (black output when a warm model is reused under dynamic VRAM). The trigger there is different.

Environment:

• OS and GPU: Windows 11, RTX 4070 Ti 12 GB, driver 616.56
• Python 3.12.10, torch 2.13.0+cu130
• ComfyUI 0.34.1, comfy-aimdo 0.4.15, comfy-kitchen 0.2.31, comfyui-frontend-package 1.49.6

[LukeG89]

Ping @rattus128

I think it's a similar issue I reported a while ago about ModelSamplingFlux:

• [Bug] ModelSamplingFlux affects globally the workflow, bypassing it won't disable its effect (Krea2 testing) #14754