feat(flux2): add regional prompting for FLUX.2 Klein#9255
Conversation
FLUX.2 Klein's diffusers transformer already accepts an attention mask through joint_attention_kwargs, and its unified attention sequence [txt, img] matches FLUX 1, so regional prompting works without a forward override — we just build a restricted mask and pass it down. Backend: - Flux2RegionalPromptingExtension builds a restricted [txt, img] mask from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic, no CLIP pooled). Unlike FLUX 1 the same mask is applied to every block (no per-block alternation, since we use the diffusers forward). - flux2 denoise() threads pos_joint_attention_kwargs to all positive transformer forwards; negative passes stay unmasked. - flux2_denoise invocation accepts list[FluxConditioningField] and skips masking when reference images extend the image stream. Frontend: - buildFLUXGraph wires posCondCollect + addRegions for the FLUX.2 path. - addRegions clones flux2_klein_text_encoder edges (qwen3_encoder, max_seq_len, mask) for each regional prompt. - validators surface unsupported features (regional negative, auto-neg, reference images) so FLUX.2 regions are filtered consistently with FLUX 1. - schema.ts regenerated to reflect the list-or-single union.
- Update flux2 denoise tests for the new conditioning API: mock conditioning fields need mask=None, and the dummy transformer must accept joint_attention_kwargs - Bump flux2_denoise invocation version to 1.6.0 for the positive_text_conditioning type change (regenerated openapi.json) - Log a warning when regional masks are dropped because reference images extend the image token stream - Add defensive asserts in addRegions for the FLUX.2 negative-prompt and auto-negative branches (unreachable today via validators) - Add unit tests for Flux2RegionalPromptingExtension mask construction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for this @Pfannkuchensack — the implementation is solid and I was able to verify the core claims end-to-end. Summary of the review, plus a note on the fixes I pushed directly to the branch (e4ccfc4).
Verified good
- Mask construction is a faithful port of FLUX 1's
_prepare_restricted_attn_mask(same background-region semantics, outer-product region self-attention,> 0.5bool conversion). With no region masks it short-circuits toNone, and I confirmed the no-region path reproduces the old inlinetxt/txt_idsconstruction exactly — global FLUX.2 generations are unchanged. - Diffusers plumbing: the installed
Flux2Transformer2DModel.forwardacceptsjoint_attention_kwargs, threadsattention_mask→attn_maskinto SDPA, and concatenates[txt, img](encoder states first) in both double- and single-stream blocks, matching the mask layout. The additive(1, 1, S, S)mask in inference dtype is what SDPA expects. - No NaN risk: every attention row has at least one allowed key (txt diagonal blocks; every img token is either in a region or in the background, both of which have self-attention).
- CFG: mask applied only to positive forwards at both call sites; negative passes run unmasked with their own ids.
- Ref-image guard: correctly skips masking when kontext tokens extend the image stream (the mask isn't shaped for them).
- Frontend:
addRegionsfilters regions with warnings before building, so thenegCondCollect: nullpath is unreachable; old workflows with a direct single-conditioning connection still validate against the new union type;tsc/eslint/prettier/ruffall clean.
Fixes pushed to the branch (e4ccfc4)
- Test updates — three existing tests in
test_denoise_noise_inputs.pybroke against the new API surface (all test-harness artifacts, not runtime bugs): the two flux2 denoise tests mockedpositive_text_conditioningwith a bareMagicMock, whose auto-created truthy.masksent the new code down the mask-preprocessing path; and the LCM scheduler test's dummy transformer didn't acceptjoint_attention_kwargs. Fixed by passingmask=Noneand adding the kwarg. - Version bump —
flux2_denoise→1.6.0for thepositive_text_conditioningtype change (openapi.jsonregenerated;schema.tsunchanged). - Warning on silent mask drop — the invocation now logs a warning when regional masks are discarded because reference images are present, instead of silently ignoring the regions.
- Defensive asserts —
addRegionsnow fails loudly if a FLUX.2 region with a negative prompt or auto-negative ever slips past the validator filter (those branches would otherwise silently build acompelnode into a FLUX.2 graph). - New unit tests —
tests/backend/flux2/test_regional_prompting_extension.pycovers mask preprocessing, the no-mask short-circuit, restricted-mask semantics (region↔txt isolation, cross-region blocking, background/global behavior, no fully-blocked rows), and the additive-mask conversion.
Full suite passes with these: 2553 passed, 0 failed.
Known UX issue: misleading "Reference Image" button on regional layers
Functional testing surfaced this: with a FLUX.2 Klein model selected, the "+ Reference Image" button on a Regional Guidance layer opens an IP-adapter model picker (which can never be satisfied — no flux2-base IP adapters exist) rather than using Klein's built-in image-edit capability. The result is a dead-end picker, and once an image is attached to the layer the generation requirements are not met, so the Invoke button remains inactive.
Two clarifications so this doesn't read as a blocker:
- It's pre-existing behavior, not a regression from this PR —
getDefaultRegionalGuidanceRefImageConfigunconditionally returns an IP-adapter config (z-image and anima hit the same dead end). The global ref-image default already special-cases flux2; the regional one doesn't. - Regional kontext isn't something this PR's backend could honor anyway: kontext tokens extend the image sequence is warned andand the regional mask doesn't cover them, which is exactly why masking is skipped when ref images are present. Supporting it would mean building the mask over
txt + img + reftokens and assigning ref tokens to regions — a real follow-on feature.
Suggested follow-up (fine as a separate PR): hide or disable the regional "+ Reference Image" button and the "regional reference image" add-layer menu entries when the selected base doesn't support regional reference images (flux2, and arguably z-image/anima, which share the dead end).
Remaining design note (no action needed)
Applying the restricted mask to every block differs from FLUX 1, which deliberately alternates masked/unmasked blocks for global coherence. I don't see a problem with seams, but if users start to complain of hard seams, this decision should be reevaluated.
lstein
left a comment
There was a problem hiding this comment.
Looks good! See comment above for a couple of issues for possible followon PRs.
Summary
FLUX.2 Klein's diffusers transformer already accepts an attention mask through
joint_attention_kwargs, and its unified attention sequence[txt, img]matches FLUX 1, so regional prompting works without a forward override — we just build a restricted mask and pass it down.Backend:
Flux2RegionalPromptingExtensionbuilds a restricted[txt, img]mask from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic, no CLIP pooled). Unlike FLUX 1 the same mask is applied to every block (no per-block alternation, since we use the diffusers forward).flux2denoise()threadspos_joint_attention_kwargsto all positive transformer forwards; negative passes stay unmasked.flux2_denoiseinvocation acceptslist[FluxConditioningField]and skips masking when reference images extend the image stream.Frontend:
buildFLUXGraphwiresposCondCollect+addRegionsfor the FLUX.2 path.addRegionsclonesflux2_klein_text_encoderedges (qwen3_encoder,max_seq_len,mask) for each regional prompt.validatorssurface unsupported features (regional negative, auto-neg, reference images) so FLUX.2 regions are filtered consistently with FLUX 1.schema.tsregenerated to reflect the list-or-single union.Related Issues / Discussions
n/a
QA Instructions
None).Merge Plan
Standard merge. No DB/schema migration.
schema.tsis regenerated and committed.Checklist
What's Newcopy (if doing a release after this PR)