HunyuanImage21

[yiyixuxu]

fix #12321

HuyuanImage2-1

from diffusers import HunyuanImagePipeline
import torch

device = "cuda:0"
dtype = torch.bfloat16
repo = "YiYiXu/HunyuanImage-2.1-Diffusers"

pipe = HunyuanImagePipeline.from_pretrained(repo, torch_dtype=dtype)
pipe = pipe.to(device)

prompt = "A cute, cartoon-style anthropomorphic penguin plush toy with fluffy fur, standing in a painting studio, wearing a red knitted scarf and a red beret with the word “Tencent” on it, holding a paintbrush with a focused expression as it paints an oil painting of the Mona Lisa, rendered in a photorealistic photographic style."

generator = torch.Generator(device=device).manual_seed(649151)
out = pipe(
    prompt, 
    num_inference_steps=50, 
    true_cfg_scale =3.5,
    negative_prompt = "",
    height=2048, 
    width=2048, 
    generator=generator,
).images[0]

out.save("test_hyimage_output.png")

HuyuanImage2.1-Distilled

from diffusers import HunyuanImagePipeline
import torch

device = "cuda:0"
dtype = torch.bfloat16

repo = "YiYiXu/HunyuanImage-2.1-Distilled-Diffusers"

pipe = HunyuanImagePipeline.from_pretrained(repo, torch_dtype=dtype)
pipe = pipe.to(device)

prompt = "A cute, cartoon-style anthropomorphic penguin plush toy with fluffy fur, standing in a painting studio, wearing a red knitted scarf and a red beret with the word “Tencent” on it, holding a paintbrush with a focused expression as it paints an oil painting of the Mona Lisa, rendered in a photorealistic photographic style."
generator = torch.Generator(device=device).manual_seed(649151)
out = pipe(
    prompt, 
    num_inference_steps=8, 
    guidance_scale =3.5,
    height=2048, 
    width=2048,
    generator=generator,
).images[0]

out.save("yiyi_test_hyimage-distilled_output.png")

HunyuanImage-2.1-Refiner

from diffusers import HunyuanImageRefinerPipeline
import torch
from diffusers.utils import load_image

device = "cuda:1"
dtype = torch.bfloat16


repo = "YiYiXu/HunyuanImage-2.1-Refiner-Diffusers"

pipe = HunyuanImageRefinerPipeline.from_pretrained(repo, torch_dtype=dtype)
pipe = pipe.to(device)

prompt = "A cute, cartoon-style anthropomorphic penguin plush toy with fluffy fur, standing in a painting studio, wearing a red knitted scarf and a red beret with the word “Tencent” on it, holding a paintbrush with a focused expression as it paints an oil painting of the Mona Lisa, rendered in a photorealistic photographic style."

image = load_image("generated_image.png")

generator = torch.Generator(device=device).manual_seed(649151)
out = pipe(
    prompt, 
    image=image,
    num_inference_steps=4, 
    guidance_scale =3.5,
    height=2048, 
    width=2048, 
    generator=generator,
).images[0]

out.save("test_hyimage_refiner_output.png")

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

LGTM 👍🏽

[sayakpaul]

Looks quite ready! My comments are mostly minor apart from some suggestions on potentially reducing some code (definitely not merge-blocking).

Let's also add tests and a doc page entry 👀

[sayakpaul]

In order for FromOriginalModelMixin to work properly do we not have to add a mapping function in the single_utils.py? Cc: @DN6

[DN6]

Yeah can remove if single file support for this isn't needed. Or we add in a follow up if it is

[Vargol]

Considering how big the model is I would imagine GGUF support would be a reason to support single file.

[sayakpaul]

You can do your own GGUFs out of diffusers checkpoints:
https://huggingface.co/docs/diffusers/main/en/quantization/gguf#convert-to-gguf

[sayakpaul]

I haven't compared too deeply but is there a chance we can fold the other VAE class implementation and this one into a single combined class? Or are the changes too many for that? Regardless, it's definitely not something merge-blocking.

[yiyixuxu]

one is 2d, one is 3d (I think maybe fine-tuned from hunyuan video , similar to qwen image & wan situation)

[sayakpaul]

If we subclass from AttentionMixin, I think utilities like attn_processors will become available automatically and we won't have to implement them here.:

diffusers/src/diffusers/models/transformers/transformer_flux.py

Line 522 in a72bc0c

AttentionMixin,

Cc: @DN6

[sayakpaul]

Not a merge blocker but we could consider doing HunyuanImageAttention like:

diffusers/src/diffusers/models/transformers/transformer_flux.py

Line 265 in a72bc0c

class FluxAttention(torch.nn.Module, AttentionModuleMixin):

Happy to open a PR myself as a followup.

[sayakpaul]

To be updated? 👀

[kk3dmax]

hunyuan21, this branch seems was not merged sucessfully, is there any action for next step?

[vladmandic]

@sayakpaul @yiyixuxu gentle ping as this pr is close-to-merge for past 3 weeks?

[yiyixuxu]

@kk3dmax @vladmandic
I will get this merged ASAP
sorry about the delay, i had to step away for a couple weeks due to a family emergency