How to force transcribe only?

[turbanisch]

I'm using the following CLI command to transcribe an audio file:

swift run whisperkit-cli transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3" --audio-path "podcast.wav"

The file is a language-learning podcast consisting of the hosts talking (in a mix of English and Chinese) and a short dialogue entirely in Chinese. While the hosts' words are accurately transcribed in their respective languages, the Chinese-only dialogue is automatically translated into English. Is there a way to avoid this and get the Chinese transcription of the dialogue instead? As far as I can tell, "transcribe" is already set as the default task, rather than "translation".

[ZachNagengast]

Would you be able to share the audio or a representative sample? Detect language is true by default via the CLI, but it may be some artifact of the audio that the model is interpreting as english instead of Chinese. You can also try the flag --use-prefill-prompt to see if it helps (this will ensure the language token is set for each window)

[ZachNagengast]

https://github.com/Vaibhavs10/translate-with-whisper here is a reference on how sometimes the model will translate when it is supposed to transcribe.

[bernie43]

I think this must be a configuration issue, when using the Whisper OpenAI Cloud API it doesn't translate by default.

If you run WhisperKit with this custom prompt it also transcribes correctly without translating:
whisperkit-cli transcribe --model-path "..." --audio-path ".." --prompt "Always transcribe, never translate"

After running some tests, the issue seems really weird:
So I've created some sample files in German.
When I spoke normally (German_Sample1.m4a), it would translate to English (without the custom prompt). When I spoke the same sentence very clearly and stilted (German_Sample2.m4a), it would transcribe German correctly.
And now comes the weird part: I converted the samples from m4a to mp3, because Github didnt't accept m4a files... and when using the mp3 versions of the exact same audio, both files get transcribed correctly instead of being translated. 🤯

audio_samples.zip