What is performance gain over Llama 3V ?

[schwarzwalder93]

Thanks for this fantastic work.

The PerceptionLM paper mentions that synthetic data engine (in Appendix K) is designed entirely to use open-source models (instead of using proprietary models) and scale up synthetic data generation to around 64.7M samples of images and videos. This is can be considered as using distillation from large-sized open-source models (in this case Llama 3V) to label training data. However, I could not see Llama 3V included as a baseline in the evaluation.

Can the team please share the performance gain of PerceptionLM over Llama 3V ? If there is gain, any observations/findings to justify this gain ?

[dbolya]

Thanks for your interest! The paper Appendix F.1 has been updated with a Llama 3V comparison.

PerceptionLM is independent from Llama 3V, not a distilled model. The only relationship is using Llama 3V 90B as a image captioner in our video captioning data engine. However, this image captioner is just one piece of the data engine--it also includes the PerceptionLM model in the loop, the original video metadata, and a cycle of human refinement.

Thus, it can't be said that PerceptionLM was distilled from Llama 3V, only that Llama 3V was one one of the components of the data engine.