FLUX.1-Kontext-dev Support for GGUF Quantized Model

[sahandkh1419]

Model/Pipeline/Scheduler description

The original model weights and pipeline are available at:
https://huggingface.co/black-forest-labs/FLUX.1-Kontext-dev/tree/main
Quantized (GGUF) versions of the model can be found here:
https://huggingface.co/QuantStack/FLUX.1-Kontext-dev-GGUF/tree/main

Using the code below, I am able to run the original model for image generation and editing:

from diffusers import FluxKontextPipeline
import torch
from PIL import Image
import numpy as np
import random


np.random.seed(42)
MAX_SEED = np.iinfo(np.int32).max

pipe = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", torch_dtype=torch.bfloat16
).to("cuda")

guidance_scale = 2.5
steps = 30

def generate_image(
    prompt: str,
    start_image: Image = None,
    guidance_scale: float = guidance_scale,
    steps: int = steps,
) -> Image:
    seed = random.randint(0, MAX_SEED)
    print(f"Seed: {seed}")
    if start_image:
        start_image = start_image.convert("RGB")
        return pipe(
            image=start_image,
            prompt=prompt,
            guidance_scale=guidance_scale,
            width=start_image.size[0],
            height=start_image.size[1],
            num_inference_steps=steps,
            generator=torch.Generator().manual_seed(seed),
        ).images[0]
    return pipe(
        prompt=prompt,
        guidance_scale=guidance_scale,
        num_inference_steps=steps,
        generator=torch.Generator().manual_seed(seed),
    ).images[0]

It would be very helpful to have support for running the GGUF-quantized version of this model using the FluxKontextPipeline, allowing inference on consumer-grade GPUs with limited VRAM.

THANKS!

Open source status

• The model implementation is available.
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[DN6]

@sahandkh1419 You should be able to load the GGUF version with single file loading.

from diffusers import FluxKontextPipeline, FluxTransformer2DModel, GGUFQuantizationConfig
import torch
from PIL import Image
import numpy as np
import random

np.random.seed(42)
MAX_SEED = np.iinfo(np.int32).max
ckpt_path = "https://huggingface.co/QuantStack/FLUX.1-Kontext-dev-GGUF/blob/main/flux1-kontext-dev-Q3_K_M.gguf"

 transformer = FluxTransformer2DModel.from_single_file(
    ckpt_path,
    quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
    config="black-forest-labs/FLUX.1-Kontext-dev",
    subfolder="transformer",
)

pipe = FluxKontextPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-Kontext-dev", transformer=transformer, torch_dtype=torch.bfloat16
).to("cuda")