Automatic Speech Recognition pipline raises error when torchcodec is installed but not valid

[litagin02]

System Info

• transformers version: 4.57.3

• Platform: Windows-11-10.0.26200-SP0

• Python version: 3.12.9

• Huggingface_hub version: 0.36.0

• Safetensors version: 0.7.0

• Accelerate version: not installed

• Accelerate config: not found

• DeepSpeed version: not installed

• PyTorch version (accelerator?): 2.9.1+cpu (NA)

• Tensorflow version (GPU?): not installed (NA)

• Flax version (CPU?/GPU?/TPU?): not installed (NA)

• Jax version: not installed

• JaxLib version: not installed

• Using distributed or parallel set-up in script?: No

• torch: 2.9.1

• torchcodec: 0.8.1

• ffmpeg: version 8.0.1-full_build-www.gyan.dev

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. uv init
2. uv add torch torchcodec transformers
3. Run the following: https://huggingface.co/docs/transformers/main/en/task_summary#automatic-speech-recognition

from transformers import pipeline

transcriber = pipeline(
    task="automatic-speech-recognition", model="openai/whisper-small"
)
transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")

Then I will get the following Error:

$ uv run ./main.py
Device set to use cpu
Traceback (most recent call last):
  File "C:\Users\user\transformers-debug\main.py", line 6, in <module>
    transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\automatic_speech_recognition.py", line 275, in __call__
    return super().__call__(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\base.py", line 1459, in __call__
    return next(
           ^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\pt_utils.py", line 126, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\pt_utils.py", line 271, in __next__
    processed = self.infer(next(self.iterator), **self.params)
                           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torch\utils\data\dataloader.py", line 732, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torch\utils\data\dataloader.py", line 788, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torch\utils\data\_utils\fetch.py", line 33, in fetch
    data.append(next(self.dataset_iter))
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\pt_utils.py", line 188, in __next__
    processed = next(self.subiterator)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\transformers\pipelines\automatic_speech_recognition.py", line 381, in preprocess
    import torchcodec
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\__init__.py", line 10, in <module>
    from . import decoders, samplers  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\decoders\__init__.py", line 7, in <module>
    from .._core import AudioStreamMetadata, VideoStreamMetadata
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\_core\__init__.py", line 8, in <module>
    from ._metadata import (
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\_core\_metadata.py", line 16, in <module>
    from torchcodec._core.ops import (
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\_core\ops.py", line 84, in <module>
    load_torchcodec_shared_libraries()
  File "C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\_core\ops.py", line 69, in load_torchcodec_shared_libraries
    raise RuntimeError(
RuntimeError: Could not load libtorchcodec. Likely causes:
          1. FFmpeg is not properly installed in your environment. We support
             versions 4, 5, 6, and 7 on all platforms, and 8 on Mac and Linux.
          2. The PyTorch version (2.9.1+cpu) is not compatible with
             this version of TorchCodec. Refer to the version compatibility
             table:
             https://github.com/pytorch/torchcodec?tab=readme-ov-file#installing-torchcodec.
          3. Another runtime dependency; see exceptions below.
        The following exceptions were raised as we tried to load libtorchcodec:

[start of libtorchcodec loading traceback]
FFmpeg version 8: Could not load this library: C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\libtorchcodec_core8.dll
FFmpeg version 7: Could not load this library: C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\libtorchcodec_core7.dll
FFmpeg version 6: Could not load this library: C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\libtorchcodec_core6.dll
FFmpeg version 5: Could not load this library: C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\libtorchcodec_core5.dll
FFmpeg version 4: Could not load this library: C:\Users\user\transformers-debug\.venv\Lib\site-packages\torchcodec\libtorchcodec_core4.dll
[end of libtorchcodec loading traceback].

I think the cause is, in Windows, even if ffmpeg and torchcodec are installed, just import torchcodec raises an error, without using conda or manually setting os.add_dll_directory("path/to/ffmpeg/dll/dir) for shared builds of ffmpeg.
So the following part raises an error (because torchcodec is available but importing yields an error):

transformers/src/transformers/pipelines/automatic_speech_recognition.py

Lines 375 to 376 in cac0a28

	if is_torchcodec_available():
	import torchcodec

Expected behavior

Raises no error. Actually, without torchcodec, we have

$ uv run ./main.py
Device set to use cpu
`return_token_timestamps` is deprecated for WhisperFeatureExtractor and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
Transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.

which is fine.

[Rocketknight1]

Hmmn - I understand the issue, but I'm not sure we should try to catch that error! If users have a misconfigured torchcodec that crashes on load, the solution is to uninstall it or install a correct version, not for us to try/except load it and have a whole alternate path for that, right? It feels like it adds code bloat for an environment issue that should always be avoidable.

[litagin02]

I basically agree, but some libraries require torchcodec (e.g., pyannote.audio). In the case of pyannote.audio, the library actually handles the error gracefully:
https://github.com/pyannote/pyannote-audio/blob/ad15cf41dde06f833646b9b73091887d60c82576/src/pyannote/audio/core/io.py#L42-L52

I also want to highlight that in the current version of torchcodec, most Windows users (even with ffmpeg installed) run into this issue unless they’re using conda. Torchcodec on Windows basically requires a conda setup, or you need to manually add the DLL directory from a shared ffmpeg build. So this might be an issue with torchcodec rather than transformers, but I really appreciate that transformers can handle the error gracefully.

This means that the situation where torchcodec is present in the environment but raises an error when imported is common at the moment.

For now, I’m using a quick monkey patch like:

from transformers.utils import import_utils

import_utils._torchcodec_available = False

and that works for me.

[Rocketknight1]

cc audio team @eustlb @ebezzam in that case - WDYT?

[ebezzam]

thanks @litagin02 for bringing this up, torchcodec installation can indeed be annoying.

I do feel like the current error from libtorchcodec is informative enough? Namely try reinstalling torchcodec (and sometimes within a fresh environment)

While pyannote catches the error to give a "cleaner" suggestion, I'm not sure if this type of error catching is typically done in Transformers? @Rocketknight1 probably knows better!

I know @zucchini-nlp and @xenova were discussing torchcodec installation at one point? I'm not sure if they have a better solution/around? Or if there's a suggested version/setup of torchcodec that we can put somewhere, e.g. in the readme

[zucchini-nlp]

Yeah, we had issues in the past with torchcodec + chat templates, and they were resolved by safe guarding the import, i.e. check the version and lazy import only when user calls an audio LLM. In any case, we allow users to uninstall torchcodec if they don't like it, and the decoding will happen with librosa

The current issue seems to come from datasets + torchcodec dependency because it is the only option for loading audios from the hub. I see that the pipeline is trying to decode a torchcodec.AudioDecoder object back to an audio array and fails. Usually it is pretty straightforward to install unless it is for Windows. IMO we could handle it gracefully for windows users and print an actionable message

Since we can't get away from dataset's dependency on torhcodec, maybe we can guide users to download and decode the audio manually?

[Rocketknight1]

Ideally we'd just get torchcodec to resolve the Windows issues so we don't need error catching in Transformers! We'll do it if it's unavoidable, but I'd prefer not to if we don't have to.

[zucchini-nlp]

Windows is still an experimental feature for torchcodec apparently, so kinda not surprised that it errors out 😅

[eustlb]

It is true that for now torchcodec can be a bit of a pain to use for some users, especially since in the situation where you cannot update the available ffmpeg version on your system, you'll necessarily have to use a conda env to install the correct one.

For this reason, I fear these users end up a bit blocked trying to solve the torchcodec installation issue while, when it comes to transformers, we can always fallback to librosa/ direct ffmpeg call (here None of those libs are required to have the pipeline working, so the error is uselessly blocking).

I the same time as stated above we don't want to log custom stuff and try catch error ourselves. If torchcodec import fails, then it just fails and log its error.

Nonetheless the easy way to avoid blocking user in the pipelines here is just to try import only if the inputs makes it necessary

if is_torchcodec_available() and type(inputs).__module__.startswith('torchcodec.'):
            import torchcodec

            if isinstance(inputs, torchcodec.decoders.AudioDecoder):
                _audio_samples = inputs.get_all_samples()

                # torchcodec always returns (num_channels, num_samples)
                # while before (datasets < 4.0) we had (2, num_samples) if stereo, (num_samples,) if mono
                _array = _audio_samples.data
                _array = _array[0] if _array.ndim == 2 and _array.shape[0] == 1 else _array
                inputs = {"array": _array, "sampling_rate": _audio_samples.sample_rate}

opening a PR!