[BUG] Flashinfer with jit-cache and cubin still attempts to jit

[bbartels]

I encountered an issue with flashinfer attempting to create a directory in order to cache JIT output for qwen3-coder-480b-a35b. I would have assumed in a situation with flashinfer-jit-cache and flashinfer-cubin it would not additionally require jitting.
This was found using the vllm Dockerfile which includes both dependencies.
For details please refer to: vllm-project/vllm#28440

2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/core.py", line 818, in cutlass_fused_moe
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m     return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/flashinfer/fused_moe/core.py", line 286, in get_cutlass_fused_moe_module
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m     module = gen_cutlass_fused_moe_sm90_module(use_fast_build).build_and_load()
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/flashinfer/jit/fused_moe.py", line 77, in gen_cutlass_fused_moe_sm90_module
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m     return gen_cutlass_fused_moe_module(nvcc_flags, "90", use_fast_build)
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m   File "/usr/local/lib/python3.12/dist-packages/flashinfer/jit/fused_moe.py", line 111, in gen_cutlass_fused_moe_module
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m     raise RuntimeError(f"Failed to generate Cutlass kernels: {e}") from e
2025-11-11T02:42:39.924015483Z
�[1;36m(EngineCore_DP0 pid=523)�[0;0m RuntimeError: Failed to generate Cutlass kernels: [Errno 13] Permission denied: '/usr/local/lib/python3.12/dist-packages/flashinfer/data/csrc/nv_internal/tensorrt_llm/cutlass_instantiations'
2025-11-11T02:42:39.924015483Z

This is the location where flashinfer attempts to create a directory:

flashinfer/flashinfer/jit/fused_moe.py

Line 103 in 6765cad

output_dir.mkdir(parents=True, exist_ok=True)

[yzh119]

What's the output of flashinfer show-config and flashinfer list-modules btw?

[bbartels]

flashinfer show-config

root@machine:/vllm-workspace$ flashinfer show-config
=== Version Info ===
FlashInfer version: 0.5.2
flashinfer-cubin version: 0.5.2
flashinfer-jit-cache version: 0.5.2+cu129
=== Torch Version Info ===
Torch version: 2.9.0+cu129
CUDA runtime available: Yes

=== Environment Variables ===
FLASHINFER_CACHE_DIR: /.cache/flashinfer
FLASHINFER_CUBIN_DIR: /usr/local/lib/python3.12/dist-packages/flashinfer_cubin/cubins
FLASHINFER_CUDA_ARCH_LIST: {(9, '0a')}
FLASHINFER_CUDA_VERSION: 12.9
FLASHINFER_CUBINS_REPOSITORY: https://edge.urm.nvidia.com/artifactory/sw-kernelinferencelibrary-public-generic-local/
CUDA_VERSION: 12.9
CUDA_HOME: /usr/local/cuda
NVCC found: Yes

=== Artifact Path ===
TRTLLM_GEN_FMHA: 463def7494c9fc6792b5aa5b5beef34025e247ac/fmha/trtllm-gen/
TRTLLM_GEN_BMM: 23daeee32b60bde7947ce1ee7a58d4ab701f134b/batched_gemm-0d28130-add42d1
TRTLLM_GEN_GEMM: 1fddc48b7b48af33914d040051b3e2ee9ba4701e/gemm-145d1b1-9b113e3
CUDNN_SDPA: a72d85b019dc125b9f711300cb989430f762f5a6/fmha/cudnn/
DEEPGEMM: a72d85b019dc125b9f711300cb989430f762f5a6/deep-gemm/

=== Downloaded Cubins ===
Traceback (most recent call last):
  File "/usr/local/bin/flashinfer", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/__main__.py", line 160, in show_config_cmd
    status = get_artifacts_status()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/artifacts.py", line 255, in get_artifacts_status
    return tuple(_check_file_status(file_name) for file_name, _ in cubin_files)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/artifacts.py", line 255, in <genexpr>
    return tuple(_check_file_status(file_name) for file_name, _ in cubin_files)
                                                                   ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/artifacts.py", line 170, in get_subdir_file_list
    checksums = get_checksums(cubin_dirs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/artifacts.py", line 147, in get_checksums
    download_file(uri, checksum_path)
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/jit/cubin_loader.py", line 80, in download_file
    with lock:
         ^^^^
  File "/usr/local/lib/python3.12/dist-packages/filelock/_api.py", line 376, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.12/dist-packages/filelock/_api.py", line 332, in acquire
    self._acquire()
  File "/usr/local/lib/python3.12/dist-packages/filelock/_unix.py", line 44, in _acquire
    fd = os.open(self.lock_file, open_flags, self._context.mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.12/dist-packages/flashinfer_cubin/cubins/463def7494c9fc6792b5aa5b5beef34025e247ac/fmha/trtllm-gen/checksums.txt.lock'

flashinfer list-modules

root@machine:/vllm-workspace$ flashinfer list-modules
No modules found. Registering default modules...
❌ Module registration failed: Failed to generate Cutlass kernels: [Errno 13] Permission denied: '/usr/local/lib/python3.12/dist-packages/flashinfer/data/csrc/nv_internal/tensorrt_llm/cutlass_instantiations'

[bbartels]

Here is the output of find . in /usr/local/lib/python3.12/dist-packages/flashinfer_cubin/cubins
https://gist.github.com/bbartels/940176ed78e8e682bf7099e94b1d7f4c

We do operate in an air-gapped environment, if that makes a difference

[yzh119]

Looks weird, what's the easiet way to access the docker image btw?

[bbartels]

@yzh119

docker pull vllm/vllm-openai:nightly-0b25498990f01ea2553c02731d6e2ce2d550156a

For reference, this is how flashinfer is installed in the docker image: https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile#L362

We are running things on H200 nodes

[bbartels]

@yzh119 Did you get a chance to have a look? :) Happy to help if you can point me in the right direction

[yzh119]

Hi @bbartels we can't reproduce this issue (even inside the container), seems you don't have write permission to /usr/local/lib/python3.12/dist-packages/, may I ask how did you set up the entire vllm dev environment (e.g. did you have any volume mapping for the docker)?

[bbartels]

Hi @bbartels we can't reproduce this issue (even inside the container), seems you don't have write permission to /usr/local/lib/python3.12/dist-packages/, may I ask how did you set up the entire vllm dev environment (e.g. did you have any volume mapping for the docker)?

This only happened to us specifically with qwen3-coder-480b-a35b. Deepseek r1 works without issues. Additionally, only when the VLLM_FLASHINFER_MOE_FP16 flag is set. Have a look at the linked vllm issue where all my cli args and env vars are denoted.

[yongwww]

@bbartels It would be great if there were a small test case (e.g., a small model to test on a single GPU) to reproduce this. I don’t currently have access to an 8-GPU node to run Qwen3-Coder-480B-A35B as you mentioned in vllm-project/vllm#28440

[bbartels]

@bbartels It would be great if there were a small test case (e.g., a small model to test on a single GPU) to reproduce this. I don’t currently have access to an 8-GPU node to run Qwen3-Coder-480B-A35B as you mentioned in vllm-project/vllm#28440

The problem is we have only observed this with Qwen3-Coder-480B-A35B so far. Other models we are deploy work without issues. I can try a couple other models to see if I can reproduce it.

[bbartels]

Is there some flag that i can enable to produce some debug/trace logs for you that could help in the meantime?

[yongwww]

I tested this on an 8×H200 node but was not able to reproduce the issue in the vllm/vllm-openai:nightly-0b25498990f01ea2553c02731d6e2ce2d550156a container on my node with the command below:

export VLLM_SKIP_P2P_CHECK=1
export VLLM_DISABLE_COMPILE_CACHE=1
export RUNAI_STREAMER_DIST=1
export RUNAI_STREAMER_CHUNK_BYTESIZE=4194304
export VLLM_USE_FLASHINFER_MOE_FP16=1
vllm serve /Qwen3-Coder-480B-A35B-Instruct --disable-log-requests --load-format=runai_streamer \
  --port=5000 --tensor-parallel-size=8 --served-model-name=gresearch/qwen3-coder-480b-a35b \
  --uvicorn-log-level=warning --block_size=32 --enable-server-load-tracking --gpu-memory-utilization=0.90 \
  --max-model-len=262144 --max-num-seqs=1024 --enable-expert-parallel --enable-auto-tool-choice \
  --tool-call-parser=qwen3_coder --enable-prefix-caching --max-num-batched-tokens=4096 --dtype=bfloat16 

[bbartels]

I tested this on an 8×H200 node but was not able to reproduce the issue in the vllm/vllm-openai:nightly-0b25498990f01ea2553c02731d6e2ce2d550156a container on my node with the command below:

export VLLM_SKIP_P2P_CHECK=1
export VLLM_DISABLE_COMPILE_CACHE=1
export RUNAI_STREAMER_DIST=1
export RUNAI_STREAMER_CHUNK_BYTESIZE=4194304
export VLLM_USE_FLASHINFER_MOE_FP16=1
vllm serve /Qwen3-Coder-480B-A35B-Instruct --disable-log-requests --load-format=runai_streamer \
  --port=5000 --tensor-parallel-size=8 --served-model-name=gresearch/qwen3-coder-480b-a35b \
  --uvicorn-log-level=warning --block_size=32 --enable-server-load-tracking --gpu-memory-utilization=0.90 \
  --max-model-len=262144 --max-num-seqs=1024 --enable-expert-parallel --enable-auto-tool-choice \
  --tool-call-parser=qwen3_coder --enable-prefix-caching --max-num-batched-tokens=4096 --dtype=bfloat16 

Hmm, very interesting, i wonder its because we run on k8s and our containers dont run as root, meaning they arent allowed to create directories. Additionally we have an air-gapped setup if that makes any difference. I'll try to reproduce this outside of k8s on a node with internet connection and get back to you. thank you for the investigation so far