How can I fine-tune CogVLM on my custom dataset?

[Iven2132]

I'm a big fan of CogVLM and I'm really excited to fine-tune it on my custom JSON dataset on Modal, I saw the fine-tuning guide but I can't find a way to fine-tune it on my custom dataset. so is there any guide or notebook to do that? It would be great if anyone could share a notebook to fine-tune it with me!

Thank you for your work! I appreciate it.

[Iven2132]

@zRzRzRzRzRzRzR. Can you please help?

[zRzRzRzRzRzRzR]

you can, what kind of dataset ?

For finetune usually use QA pair and you need to turn it to our dataset.py formate

[Iven2132]

@zRzRzRzRzRzRzR It's a JSON dataset like:

[
  {
    "id": "997bb945-628d-4724-b370-b84de974a19f",
    "image": "part-000001/997bb945-628d-4724-b370-b84de974a19f.jpg",
    "conversations": [
      {
        "from": "human",
        "value": "<image>\nWrite a prompt for Stable Diffusion to generate this image."
      },
      {
        "from": "gpt",
        "value": "a beautiful painting of chernobyl by nekro, pascal blanche, john harris, greg rutkowski, sin jong hun, moebius, simon stalenhag. in style of cg art. ray tracing. cel shading. hyper detailed. realistic. ue 5. maya. octane render. "
      },
    ]
  },
  ...
]

[Iven2132]

@zRzRzRzRzRzRzR can you please help me out? are you on Discord, I'd love to chat with you!