[New Model]: ByteDance-Seed/BAGEL-7B-MoT

[XueSongTap]

The model to consider.

https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT

The closest model vllm already supports.

https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen2_5_vl.py

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Dexterai]

hi, I'm interested in contributing but am new to the project, can I work on this ?

[princepride]

@DarkLight1337 it seems that the current vLLM framework does not support this kind of multi-modal input and multi-modal output model, right?

[DarkLight1337]

Sorry @Dexterai I missed your reply.

We only have minimal support for this kind of model. For now, the pooling task can be used to support multimodal outputs, check out the example of Prithvi-EO model: https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/prithvi_geospatial_mae.py

[princepride]

So it only support offline inference?

[DarkLight1337]

Yes, only offline inference

[Dexterai]

@DarkLight1337 That's ok. Can vllm now support images or video generation? This model support both image understanding and generating.

[DarkLight1337]

Sorry @Dexterai I missed your reply.

We only have minimal support for this kind of model. For now, the pooling task can be used to support multimodal outputs, check out the example of Prithvi-EO model: https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/prithvi_geospatial_mae.py

See this

[princepride]

@Dexterai are you working on this model, I am also interested in this model

[Dexterai]

@princepride I give it up, vllm now don't support multimodal outputs. and bagel supports image and video output.

[princepride]

@DarkLight1337 As you said, we can only implement the offline inference version of this model, but I am very interested in the implementation of this model. Can you assign this model to me?

[princepride]

@Isotr0py Sorry to bother you, if the model's weight were split to two different model and the original project code use hard code to load the model's weight, what should I do, to specific, I want know how vLLM auto load the model's weight file from huggingface repo.
For exmaple (from source code):

...
...
from safetensors.torch import load_file as load_sft
def load_ae(local_path: str) -> AutoEncoder:
    ae_params = AutoEncoderParams(
            resolution=256,
            in_channels=3,
            downsample=8,
            ch=128,
            out_ch=3,
            ch_mult=[1, 2, 4, 4],
            num_res_blocks=2,
            z_channels=16,
            scale_factor=0.3611,
            shift_factor=0.1159,
    )

    # Loading the autoencoder
    ae = AutoEncoder(ae_params)

    if local_path is not None:
        sd = load_sft(local_path)
        missing, unexpected = ae.load_state_dict(sd, strict=False, assign=True)
        print_load_warning(missing, unexpected)
    return ae, ae_params

# load the vae model
vae_model, vae_config = load_ae(local_path=os.path.join(args.model_path, "ae.safetensors"))

# load the ema model
model = load_checkpoint_and_dispatch(
        model,
        checkpoint=os.path.join(args.model_path, "ema.safetensors"),
        device_map=device_map,
        offload_buffers=True,
        offload_folder="offload",
        dtype=torch.bfloat16,
        force_hooks=True,
    ).eval()
...