Training Details

[zc-zhao]

Thanks for your great work! I misunderstand the training data in the main paper.

In first paragraph, you use the same data as LLaVA, so does this mean that this model (using Vicuna as LLM) only features the image understanding not includes video?

In the second paragraph, you use the video data. Is it used when using LLaMA-3.1-8B or also when using Vicuna-7B？Does this mean that all the experiments on video evaluation in your paper were conducted based on LLaMA-3.1-8B?

Could you release more training details and code？

[zc-zhao]

@zhangshaolei1998

[zc-zhao]

Additionally, when I add the video-chat dataset during training, the program process hangs and results in the NCCL watchdog timeout error

[Rank 4] NCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=29875, OpType=_ALLGATHER_BAS
E, NumelIn=75264, NumelOut=602112, Timeout(ms)=1800000) ran for 1800089 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corr
upted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 7] NCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=29875, OpT
ype=_ALLGATHER_BASE, NumelIn=75264, NumelOut=602112, Timeout(ms)=1800000) ran for 1800096 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
what(): [Rank 7] NCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=29875, OpType=_ALLGATHER_BAS
E, NumelIn=75264, NumelOut=602112, Timeout(ms)=1800000) ran for 1800096 milliseconds before timing