Clarification of Decoding Settings for Video Benchmarks of PLM-8B

[cheongalc]

Dear authors, thank you very much for releasing this work. I am trying to reproduce the results of PLM-8B on Video-MME and PLM-VideoBench, and would like to clarify the following points:

• During evaluation, can I check that 32, not 8 frames are uniformly sampled from each test video? I see that in page 16, Table 12 of the PE paper, it is stated that "For our PLM-8B we use PElangG as the vision encoder with 36 tiles for images and 32 frames for video". The default number of frames in params.json on Huggingface is also 32 indeed. Were 32 frames uniformly sampled from each video in each benchmark, regardless of the video length?

• May I know what was the temperature, and whether there was any top_p and top_k decoding when doing the evaluations on page 16, Table 12 of the PE paper?

• May I know what was max_tokens set as during your evaluations on page 16, Table 12 of the PE paper? The current value of 11264 is causing VRAM usage to be high.

The reason for the 2nd and 3rd question is that I am experiencing a strange issue, where PLM-8B tends to loop its outputs and fail to emit the end of sequence token properly.

This problem manifests even when I clone this repository, setup a fresh environment as per the instructions in README, and use the default generation config which is deterministic, has temperature=0, and does not use top_p or top_k sampling.

An example video is this 30 second one, with a prompt of Describe what is going on in this video in great detail. The generation I'm getting is this:

A black cat is lying on a rug and a ball rolls towards it. The cat rolls over and plays with the ball. The ball rolls away and the cat follows it. The cat then lies on its back and plays with the ball. The ball rolls away and the cat follows it. The cat then sits up and plays with the ball. The ball rolls away and the cat follows it. The cat then lies on its back and plays with the ball. The ball rolls away and the cat follows it. The cat then sits up and plays with the ball. The ball rolls away and the cat follows it. The cat then lies on its back and plays with the ball. The ball rolls away and the cat follows it. The cat then sits up and plays with the ball. The ball rolls away and the cat follows it. The cat then lies on its back and plays with the ball. The ball rolls away and the cat follows it. The cat then sits up and plays with the ball. The ball rolls away and the cat follows it. The cat then lies on its back and plays with the ball. The ball rolls away and the cat follows it. The cat then sits up and plays with the ball. The ball rolls away and the cat follows it. The

Thank you very much!

[cheongalc]

I was searching lmms-eval for the videomme benchmark, and found that in the config for videomme, the generation settings are listed as

generation_kwargs:
  max_new_tokens: 16
  temperature: 0
  top_p: 1.0
  num_beams: 1
  do_sample: false

Thus, temperature = 0, but since do_sample = false, then top_p and top_k are ignored and greedy decoding should be used.

However, in the actual implementation for PLM, this line seems to suggest that temperature stays at 0, but top_k = 100. However, I do not see any other code modifying the do_sample : false. Thus, I am not sure whether the do_sample : false is still causing the model to do greedy decoding despite the identified line suggesting that top_k is 100.

Thus, may I clarify for Video-MME and PLM-VideoBench, the temperature setting comes from the individual yaml config of the benchmark, but the top_k is set to 100 due to the identified line?

Thank you very much for your time!

[shuminghu]

• 32 frames are uniformly sampled from each video in each benchmark, if video is longer than 32s. If a video is shorter than 32s, it's sampled uniformly at 1 FPS.
• do_sample : false overrides any sampling parameters. Essentially only greedy decoding is used in all our evaluations.
• max_tokens is always set to 11264 during our evaluations.
  For reduced memory usage, one can try this.

[shuminghu]

For that cat video, we probably need to debug to see what went wrong here :(

[cheongalc]

Thank you for the clarification!

May I also ask: I note that in the PE Lang README, there is a line there that says "This checkpoint was further aligned using tiling. We will release the tiling aligned checkpoint soon."

On my end, I downloaded PLM-8B, then extracted the weights of the vision tower and 2 layer vision projector. May I know how different this encoder is from the unreleased checkpoint of PE-Lang which was aligned with tiling?

Also, I have more corner cases for videos.
With a prompt of "Describe what is going on in this video in great detail.", this video has the following response:

A video of a group of capybaras in a zoo. The video is a montage of different clips of the capybaras. The video starts with a clip of a group of capybaras in a pool of water. The video then cuts to a clip of a capybara with a lemon on its head. The video then cuts to a clip of a group of capybaras in a zoo. The video then cuts to a clip of a capybara eating a piece of lettuce. The video then cuts to a clip of a group of capybaras in a zoo. The video then cuts to a clip of a capybara being petted by a person. The video then cuts to a clip of a group of capybaras in a zoo. The video then cuts to a clip of a capybara being fed by a person. The video then cuts to a clip of a group of capybaras in a zoo. The video then cuts to a clip of a capybara being petted by a person. The video then cuts to a clip of a group of capybaras in a zoo. The video then cuts to a clip of a capybara being fed by a person. The

I can inform you of more edge cases as I continue to test the model. Thank you!

[shuminghu]

PE-lang is released, e.g., https://huggingface.co/facebook/PE-Lang-L14-448.
See comment below on context of PE-lang.

Thanks for the examples. I can take a look.

[dbolya]

Some additional context for PE Lang:

On my end, I downloaded PLM-8B, then extracted the weights of the vision tower and 2 layer vision projector. May I know how different this encoder is from the unreleased checkpoint of PE-Lang which was aligned with tiling?

Extracting the final PLM-8B encoder is very similar to the unrelease checkpoint (which we can release in a few weeks, some people are out right now). The PE lang checkpoints are extracted from Stage 2 (midtraining) of the PLM training pipeline, but the one we released is from an unreleased version of PLM.

The currently released PE lang used a version of PLM created for benchmarking in the PLM paper: PE-G w/ Llama 3.2 3B, which is the setting we found to transfer the best to other models. This "benchmarking" PLM model is not one that we released. Instead, the PLM-8B that uses PE-G is midtrained with tiling, and that's the checkpoint that we refer to in this comment:

This checkpoint was further aligned using tiling. We will release the tiling aligned checkpoint soon.

Thus, extracting PE-G from the release PLM-8B is the checkpoint this comment refers to except that it has additionally been "SFT'd" (Stage 3). The model we would release would not have this SFT in order to be more general, but extracting the PE-G from PLM-8B would be a good compromise in the mean time. Shuming pointed out that it's possible this extraction would perform much worse than a Stage 2 extracted model, as the SFT dataset we use is relatively small and narrowly focused. So, use at your own peril until we release the Stage 2 model in a few weeks.

[cheongalc]

Thank you so much for the details. I would like to clarify some points. Sorry, I got a bit confused.

When you said that

The PE lang checkpoints are extracted from Stage 2 (midtraining) of the PLM training pipeline, but the one we released is from an unreleased version of PLM.

and

The currently released PE lang used a version of PLM created for benchmarking in the PLM paper: PE-G w/ Llama 3.2 3B, which is the setting we found to transfer the best to other models.

are you saying that this PE-Lang checkpoint released on HF came from the Stage 2 of an unreleased version of PLM, which was configured as PE-G+Llama 3.2 3B?

Then, is it correct to say that the PE-Lang checkpoint mentioned by the comment

This checkpoint was further aligned using tiling. We will release the tiling aligned checkpoint soon.

will come from the Stage 2 of the PLM-8B currently released on HF?

Lastly, in page 16, table 12 of the PE paper,

does the PLM-8B at the last row refer to the same PLM-8B currently released on HF?

Thank you very much!

[cheongalc]

Moreover, @shuminghu, I think this particular prompt Describe what is going on in this video in great detail. is triggering the token looping behavior of the PLM-8B on HF.

If I send in a prompt like Describe what is going on in this video. the model is very terse. But once we say "in great detail", it starts looping.

Here are 3 more YouTube IDs of videos from the videomme dataset that cause problems. For all 3 videos, the prompt is Describe what is going on in this video in great detail. and the generation config is

temperature: float = 0.0
top_p: Optional[float] = None
top_k: Optional[float] = None
max_gen_len: int = 1024
max_tokens: int = 11264
until: List[str] = field(default_factory=list)
compile_prefilling: bool = False 
reduce_generation_overhead: bool = False
show_progress: bool = True
dtype: Optional[str] = "bf16"
device: Optional[str] = "cuda"

1NcGHbFWBFA

A video of a compilation of different animals and people interacting with them. The video is set to music and has a blue background. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips. The video is a montage of different clips

-c8eATXUui8

A man is walking down a street in a city, eating food. He is wearing a green hoodie and glasses. He is holding an umbrella. He is eating a sandwich, then a cookie, then a hotdog, then a slice of pizza, then a taco, then a slice of cake, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza again, then a slice of pizza

_aVHf_jmWk8.

This video features an animated cartoon of a girl with purple hair and wings. The video starts with a close up of a clock, then the girl's face, then her back with wings. The video then cuts to a butterfly flying, then the girl's face again, then her back with wings again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair,

[dbolya]

About PE Lang: no worries, my earlier comment was a bit convoluted 😅

are you saying that this PE-Lang checkpoint released on HF came from the Stage 2 of an unreleased version of PLM, which was configured as PE-G+Llama 3.2 3B?

Yes, basically the "approach" to construct PE Lang is to train PLM without SFT and then extract the encoder. The PE-G w/ Llama 3.2 3B combination produced the most general PE Lang, but the resulting PLM model was not so useful. Note that all the MLLM results in the PE paper (including tiling, refcoco, and different LLMs) are using that model except for Table 12 (system-level comparison).

Then, is it correct to say that the PE-Lang checkpoint mentioned by the comment [...] will come from the Stage 2 of the PLM-8B currently released on HF?

Yes, this is correct. It will be an intermediate checkpoint of the released PLM-8B model.

Lastly, in page 16, table 12 of the PE paper, [...] does the PLM-8B at the last row refer to the same PLM-8B currently released on HF?

Yes, that is correct. That table in the PE paper is an abridged version of Tables 3 and 4 in the PLM paper.

[cheongalc]

Hi authors, thank you very much for your time. I would also like to clarify the decoding settings when doing eval on PLM-RTLoc and Charades-STA, as I'm interested in use cases where we ask PLM to output specific timestamps for a certain activity in the video.

I note that PLM was trained with PLM-STC, and every video in PLM-STC has a blue-colored segmentation mask to identify the subject of interest. But when PLM is evaluated on PLM-RTLoc, I see in the lmms-eval code that a red rectangle bbox is drawn onto the video frames instead before the whole video is sent to PLM.

May I know what the decision was behind the different kind of annotation during eval?

Moreover, for Charades-STA, I don't see any segmentation mask or bbox being applied to the video before inference is conducted. Does that mean that when PLM is evaluated on Charades-STA, there is no annotation on the video? And, the only assistance it receives is the text prompt with this kind of format:

Please find the visual event described by a sentence in the video, determining its starting and ending times. The format should be: 'The event happens in the start time - end time'. For example, The event 'person turn a light on' happens in the 24.3 - 30.4 seonds. Now I will give you the textual sentence: INSERT_MY_SENTENCE_HERE. Please return its start time and end time.

I find it really impressive that PLM-8B can get 58.6 mIOU on Charades-STA, given that

• The Charades-STA prompt has a small typo (seonds),
• The 1-shot example The event 'person turn a light on' happens in the 24.3 - 30.4 seonds is not dynamically updated to guide the model on how long the incoming video is,
• PLM received no guiding mask/bbox on which subject to focus on,
• and PLM only sampled 32 video frames uniformly, so for the videos in Charades-STA that were more than 32s long, the sampling skipped some frames.

If I wanted PLM-8B to identify specific timestamps of activities in a video, what would be the ideal decoding configuration?

• Does PLM-8B need to see a segmentation mask (specifically a blue one)? I think it doesn't since it can do well on RT-Loc and Charades-STA, both of which don't have this style of annotation?
• Should I ask the model to output the frame number, or the video time in seconds as the timestamp?
• If my video is long, should I cut it into chunks of 32 seconds each so that when I sample 32 frames, I cover every second of the video chunk? Otherwise the space between each frame could be quite big, then how would the model do fine grained localization? Is it doing "mental math" to figure out roughly where events start and end?

Lastly, suppose I wanted to use the PE-Lang-G-448 encoder with a different LLM decoder. Which PE-Lang checkpoint would be the best? The current one on HF which is based off PE-G+Llama 3.2 3B, or the new one which will come out in a few weeks?

Thank you so much!

[shuminghu]

I can comfirm the looping outputs for the same prompt from PLM-8B.
The looping outputs don't appear for PLM-1B or PLM-3B.
With PLM-8B, if you set temperature = 1.0,top_p = 0.95, you get this output for the following video:

Prompt 1: Describe what is going on in this video in great detail.
Generated Text: This video features cartoon-style animation of two young women. At the beginning, a clock hand moves from 10 to 11, and at the end, the word "thanks for watching" appears, followed by another text in the middle of the screen that states, "social networks in description."
There are close-ups of young woman's eyes and hair, and then we see a girl with pink hair float, and a red ribbon flying, and then two women facing each other. Following this, we see a closeup of a young woman's back, with wings and vines on her back, and then several sequences of the young women facing each other. In one instance, it shows the woman with wings, and in another, the two women run into each other's outstretched arms and hug.
It cuts to a light blue background, with a dark gray blue diamond shape moving along a pinkish hue line. We see the women facing each other again, except here the young woman has purplish hair. More digital art designs appear, with black shapes and angular patterns. There's a closeup of a young woman's face, and then the woman's back, with a sweater, but this time, we see red drops of blood coming out of her veins.
Tokens per second: 25.57

[_aVHf_jmWk8](https://www.youtube.com/watch?v=_aVHf_jmWk8).

This video features an animated cartoon of a girl with purple hair and wings. The video starts with a close up of a clock, then the girl's face, then her back with wings. The video then cuts to a butterfly flying, then the girl's face again, then her back with wings again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair, then a girl with purple hair, then a girl with pink hair again, then a girl with purple hair again. The video then cuts to a girl with pink hair,

[shuminghu]

For PLM-STC data, PLM is trained with the same video setup as eval, i.e., with red rectangle boxes drawn .

CharadsSTA is evaluated with prompt "<video>\nPlease provide the start and end frame of this event in a 32 frame video: {question}\nRespond in the format [start_frame, end_frame]{answer}." The model outputs start_frame and end_frame in terms of integers within [1, 2, ... , 32] (not timestamp/secconds).

I think you can start with PE-Lang-G-448 released, where the newer one is probably better but many folks are on PTO now.

My collegues can confirm and provide more context on this.

[cheongalc]

Thank you for the clarification! May I know where the code for the Charades_STA eval is? Since the prompt template is different, was lmms-eval used (but the YAML config was changed)?

[shuminghu]

I think we missed porting this task to lmms-eval.
Essentially given a video duration, we converted predicted frame index to timestamp by fetching from the following array

sampled_frames_ts = np.linspace(0, duration, 32)

before computing IoU.

You can try implementing this yourself. We will probably add it to lmms-eval later.