Skip to content

feat(flux2): add regional prompting for FLUX.2 Klein#9255

Merged
lstein merged 15 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/flux2-klein-regional-guidance
Jul 17, 2026
Merged

feat(flux2): add regional prompting for FLUX.2 Klein#9255
lstein merged 15 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/flux2-klein-regional-guidance

Conversation

@Pfannkuchensack

Copy link
Copy Markdown
Collaborator

Summary

FLUX.2 Klein's diffusers transformer already accepts an attention mask through joint_attention_kwargs, and its unified attention sequence [txt, img] matches FLUX 1, so regional prompting works without a forward override — we just build a restricted mask and pass it down.

Backend:

  • Flux2RegionalPromptingExtension builds a restricted [txt, img] mask from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic, no CLIP pooled). Unlike FLUX 1 the same mask is applied to every block (no per-block alternation, since we use the diffusers forward).
  • flux2 denoise() threads pos_joint_attention_kwargs to all positive transformer forwards; negative passes stay unmasked.
  • flux2_denoise invocation accepts list[FluxConditioningField] and skips masking when reference images extend the image stream.

Frontend:

  • buildFLUXGraph wires posCondCollect + addRegions for the FLUX.2 path.
  • addRegions clones flux2_klein_text_encoder edges (qwen3_encoder, max_seq_len, mask) for each regional prompt.
  • validators surface unsupported features (regional negative, auto-neg, reference images) so FLUX.2 regions are filtered consistently with FLUX 1.
  • schema.ts regenerated to reflect the list-or-single union.

Related Issues / Discussions

n/a

QA Instructions

  1. Load any FLUX.2 Klein model in the canvas.
  2. Create two or more Regional Guidance layers with distinct prompts and non-overlapping masks (e.g. left half = "a red car", right half = "a blue truck").
  3. Invoke txt2img — generated image should respect the regional split.
  4. Verify the unsupported-feature warnings show up on a FLUX.2 region that has:
    • a negative prompt
    • auto-negative enabled
    • a reference image attached
  5. Confirm a region with reference images is filtered out (no regional masking applied) — invocation should still run as a global prompt.
  6. Sanity-check that a global FLUX.2 Klein generation (no regions) is byte-identical to before this PR (no mask is built when no region masks are present, short-circuit returns None).
  7. Regression-check FLUX 1 and Z-Image regional prompting still works.

Merge Plan

Standard merge. No DB/schema migration. schema.ts is regenerated and committed.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

FLUX.2 Klein's diffusers transformer already accepts an attention mask
through joint_attention_kwargs, and its unified attention sequence
[txt, img] matches FLUX 1, so regional prompting works without a forward
override — we just build a restricted mask and pass it down.

Backend:
- Flux2RegionalPromptingExtension builds a restricted [txt, img] mask
  from per-conditioning Qwen3 embeddings + image masks (FLUX 1 logic,
  no CLIP pooled). Unlike FLUX 1 the same mask is applied to every
  block (no per-block alternation, since we use the diffusers forward).
- flux2 denoise() threads pos_joint_attention_kwargs to all positive
  transformer forwards; negative passes stay unmasked.
- flux2_denoise invocation accepts list[FluxConditioningField] and
  skips masking when reference images extend the image stream.

Frontend:
- buildFLUXGraph wires posCondCollect + addRegions for the FLUX.2 path.
- addRegions clones flux2_klein_text_encoder edges (qwen3_encoder,
  max_seq_len, mask) for each regional prompt.
- validators surface unsupported features (regional negative, auto-neg,
  reference images) so FLUX.2 regions are filtered consistently with
  FLUX 1.
- schema.ts regenerated to reflect the list-or-single union.
@github-actions github-actions Bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files labels May 30, 2026
@lstein lstein self-assigned this Jun 3, 2026
@lstein lstein added the 6.13.5 Library Updates label Jun 3, 2026
@lstein lstein moved this to 6.13.5 LIBRARY UPDATES in Invoke - Community Roadmap Jun 3, 2026
@lstein lstein added 6.14.x and removed 6.13.5 Library Updates labels Jun 22, 2026
@lstein lstein moved this from 6.13.5 LIBRARY UPDATES to 6.14.x Theme: USER EXPERIENCE in Invoke - Community Roadmap Jun 22, 2026
@Pfannkuchensack
Pfannkuchensack marked this pull request as ready for review June 29, 2026 22:25
- Update flux2 denoise tests for the new conditioning API: mock
  conditioning fields need mask=None, and the dummy transformer must
  accept joint_attention_kwargs
- Bump flux2_denoise invocation version to 1.6.0 for the
  positive_text_conditioning type change (regenerated openapi.json)
- Log a warning when regional masks are dropped because reference
  images extend the image token stream
- Add defensive asserts in addRegions for the FLUX.2 negative-prompt
  and auto-negative branches (unreachable today via validators)
- Add unit tests for Flux2RegionalPromptingExtension mask construction

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the python-tests PRs that change python tests label Jul 17, 2026

@lstein lstein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @Pfannkuchensack — the implementation is solid and I was able to verify the core claims end-to-end. Summary of the review, plus a note on the fixes I pushed directly to the branch (e4ccfc4).

Verified good

  • Mask construction is a faithful port of FLUX 1's _prepare_restricted_attn_mask (same background-region semantics, outer-product region self-attention, > 0.5 bool conversion). With no region masks it short-circuits to None, and I confirmed the no-region path reproduces the old inline txt/txt_ids construction exactly — global FLUX.2 generations are unchanged.
  • Diffusers plumbing: the installed Flux2Transformer2DModel.forward accepts joint_attention_kwargs, threads attention_maskattn_mask into SDPA, and concatenates [txt, img] (encoder states first) in both double- and single-stream blocks, matching the mask layout. The additive (1, 1, S, S) mask in inference dtype is what SDPA expects.
  • No NaN risk: every attention row has at least one allowed key (txt diagonal blocks; every img token is either in a region or in the background, both of which have self-attention).
  • CFG: mask applied only to positive forwards at both call sites; negative passes run unmasked with their own ids.
  • Ref-image guard: correctly skips masking when kontext tokens extend the image stream (the mask isn't shaped for them).
  • Frontend: addRegions filters regions with warnings before building, so the negCondCollect: null path is unreachable; old workflows with a direct single-conditioning connection still validate against the new union type; tsc/eslint/prettier/ruff all clean.

Fixes pushed to the branch (e4ccfc4)

  1. Test updates — three existing tests in test_denoise_noise_inputs.py broke against the new API surface (all test-harness artifacts, not runtime bugs): the two flux2 denoise tests mocked positive_text_conditioning with a bare MagicMock, whose auto-created truthy .mask sent the new code down the mask-preprocessing path; and the LCM scheduler test's dummy transformer didn't accept joint_attention_kwargs. Fixed by passing mask=None and adding the kwarg.
  2. Version bumpflux2_denoise1.6.0 for the positive_text_conditioning type change (openapi.json regenerated; schema.ts unchanged).
  3. Warning on silent mask drop — the invocation now logs a warning when regional masks are discarded because reference images are present, instead of silently ignoring the regions.
  4. Defensive assertsaddRegions now fails loudly if a FLUX.2 region with a negative prompt or auto-negative ever slips past the validator filter (those branches would otherwise silently build a compel node into a FLUX.2 graph).
  5. New unit teststests/backend/flux2/test_regional_prompting_extension.py covers mask preprocessing, the no-mask short-circuit, restricted-mask semantics (region↔txt isolation, cross-region blocking, background/global behavior, no fully-blocked rows), and the additive-mask conversion.

Full suite passes with these: 2553 passed, 0 failed.

Known UX issue: misleading "Reference Image" button on regional layers

Functional testing surfaced this: with a FLUX.2 Klein model selected, the "+ Reference Image" button on a Regional Guidance layer opens an IP-adapter model picker (which can never be satisfied — no flux2-base IP adapters exist) rather than using Klein's built-in image-edit capability. The result is a dead-end picker, and once an image is attached to the layer the generation requirements are not met, so the Invoke button remains inactive.

Two clarifications so this doesn't read as a blocker:

  • It's pre-existing behavior, not a regression from this PR — getDefaultRegionalGuidanceRefImageConfig unconditionally returns an IP-adapter config (z-image and anima hit the same dead end). The global ref-image default already special-cases flux2; the regional one doesn't.
  • Regional kontext isn't something this PR's backend could honor anyway: kontext tokens extend the image sequence is warned andand the regional mask doesn't cover them, which is exactly why masking is skipped when ref images are present. Supporting it would mean building the mask over txt + img + ref tokens and assigning ref tokens to regions — a real follow-on feature.

Suggested follow-up (fine as a separate PR): hide or disable the regional "+ Reference Image" button and the "regional reference image" add-layer menu entries when the selected base doesn't support regional reference images (flux2, and arguably z-image/anima, which share the dead end).

Remaining design note (no action needed)

Applying the restricted mask to every block differs from FLUX 1, which deliberately alternates masked/unmasked blocks for global coherence. I don't see a problem with seams, but if users start to complain of hard seams, this decision should be reevaluated.

@lstein lstein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! See comment above for a couple of issues for possible followon PRs.

@lstein
lstein merged commit 338a7e9 into invoke-ai:main Jul 17, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.14.x backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests

Projects

Status: 6.14.x Theme: USER EXPERIENCE

Development

Successfully merging this pull request may close these issues.

2 participants