Regarding the fine-tuning Whisper models for streaming capabilities in other languages

[IbratDO]

Hello, I came across your work which is truly great! I am wondering if it is possible to fine-tune a whisper model for streaming capability for other languages. For example, I have fine-tuned whisper model for Uzbek language. this model performs well, on batch transcription, but I need streaming capability. so I was wondering if your framework can help me do that. if yes, how can I do that? I find some codes in examples section of your github repo, but I am not sure that it will help me train whisper model for streaming. Thank you in advance

[robin1001]

You can just modify the training configtrain.yaml, setting use_dynamic_chunk to true.

[IbratDO]

@robin1001 that is it? does this work in other languages? such as I want to fine tune whisper-large-v3 model on Uzbek language and make the model streaming. is changing the train.yaml config, setting use_dynamic_chunk to True help me achieve that capability?

[robin1001]

Yes, that's all. CTC is used for streaming in wenet, and autogressive decoding is used for non-streaming result.

[robin1001]

Please see https://github.com/wenet-e2e/wenet/blob/main/wenet/models/whisper/whisper.py

[IbratDO]

@robin1001 Ok, thank you. Definitely, try it out!