[Feature Request]: Flux

[StableLlama]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Today https://blackforestlabs.ai/ (from the same people that created Stable Diffusion) introduced their first new modes: FLUX

Weights: https://huggingface.co/black-forest-labs/FLUX.1-dev
Code to run: https://github.com/black-forest-labs/flux

It would be great to use it with the stable-diffusion-webui as well

Proposed workflow

1. Open WebUI
2. Select FLUX
3. Be happy

Additional information

No response

[mr-lab]

much needed

[Desm0nt]

Completely agree. Technicaly it is the real next SD model (instead of SD 3) from SD authors, and it's deserve to be run on really comfortable UI (instead of Comfy)

[micodev]

thumbs up

[BrechtCorbeel]

Lets hope this one doesn't get banned, but what we really need is more video models and better opensource video generators!!!

The people producing these are beating image models to death with unnecessary amounts of them, we need video and 3D!!

[cccyberwolke]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

[levicki]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

Here's what the authors say:

• Upgrade diffusers to latest
• Use FluxPipeline from diffusers

Example usage:

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    output_type="pil",
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save("flux-dev.png")

[Desm0nt]

What is needed to get this done? Am i wrong in assuming it is not too big of a task seeing how quickly comfy supported it?

It's not a big task for comfy or sd.next because they use diffusers (that already adapted for flux).

But SD webui use own custom implementation for SD, so it is need to be particularly rewritten and adapted for a new model architecture.

[colotoon]

it works in comfy in 8bit on 3090, its pretty fast in distilled mode, but loading times... oh man. img2img does not work that well, up to 0.8 denoise it just upscales original, past 0,81 it generates totally different pic. But it has best anatomy coherence from all local models, can generate people upside down like pony, no other models can do this.

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

[wizz13150]

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

My node was stuck too.
The flux checkpoint actually go in the 'unet' folder, not in the 'checkpoint' folder.
Don't ask me why.... it's comfy lol

Cheers 🥂

[colotoon]

please show ur node list, i dont understand how to make this works, especially node Load Diffusion model

My node was stuck too. The flux checkpoint actually go in the 'unet' folder, not in the 'checkpoint' folder. Don't ask me why.... it's comfy lol

Cheers 🥂

aaand, how select flux model?
its not shows in the load checkpoint node

[rkfg]

its not shows in the load checkpoint node

You need to use a different node:

I suppose FLUX needs to be reassembled into just one .safetensors file first. It consists of UNET, VAE and two CLIP encoders, all separate. A1111 expects one file per model.

[colotoon]

its not shows in the load checkpoint node

You need to use a different node:

I suppose FLUX needs to be reassembled into just one .safetensors file first. It consists of UNET, VAE and two CLIP encoders, all separate. A1111 expects one file per model.

https://prnt.sc/MzIljv3j-tc9

it ask clip but in load diff model node i cant see clip connectors..

can u share ur node list plz
IDK how it shoud be ((

[wizz13150]

can u share ur node list plz IDK how it shoud be ((

To end this discussion, obviously not about A1111, just use the default workflows ? :
https://comfyanonymous.github.io/ComfyUI_examples/flux/

[AcademiaSD]

This if FLUX1-dev checkpoint with included VAE, maybe it can help
https://huggingface.co/Comfy-Org/flux1-dev/tree/main