Nvidia 50xx with Talos 1.10.3 - Nvidia modules are not found

[flashpixx]

Hello,

I try to make it work that my new computer (one node) can run within the pods my Nvidia 5090.
I follow the NVIDIA GPU (Proprietary drivers) and prepare with the image factory (online / running locally with docker compose) my own ISO bare-metal image. My customizing to generate the ISO is

customization:
    systemExtensions:
        officialExtensions:
            - siderolabs/nonfree-kmod-nvidia-production
            - siderolabs/nvidia-container-toolkit-production

and on the controlplane.yamlI set for installing the system

machine:
    sysctls:
        net.core.bpf_jit_harden: 1

    kernel:
        modules:
            - name: nvidia
            - name: nvidia_uvm
            - name: nvidia_drm
            - name: nvidia_modeset

But I get the error on dmesg the no nvidia model is found. I Have tested it with 1.10.1. 1.10.2, 1.10.3. Did I something wrong?

[rothgar]

What does talosctl get pcidevices show?

[flashpixx]

II have grep-ed the Nvidia parts only:

192.168.20.7   hardware    PCIDevice   0000:01:00.0   1         Display controller          VGA compatible controller        NVIDIA Corporation                       GB202 [GeForce RTX 5090]
192.168.20.7   hardware    PCIDevice   0000:01:00.1   1         Multimedia controller       Audio device                     NVIDIA Corporation

I have got a typo in my first post, I get on dmesg the error message that the "nvidia kernel modules (drm, uvm, modset etc) are not found"

[smira]

From what I can tell (and you can do this research yourself to verify my points), even latest 570.x doesn't support your GPU https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-570-148-08/index.html

It's Blackwell, but seems that RTX 5xxx is supported only with Ada Lovelace. So you might need to figure out first what is the NVIDIA supported driver in stable series (or is there going to be one?).

[flashpixx]

I have seen this article https://medium.com/@ttio2tech_28094/how-to-install-rtx-5090-5080-5070-ti-5070-drivers-on-linux-detailed-guide-d7069c7a0db7 for supporting the GPU, but I'm not quite sure that your link reference the correct hardware, because it is for data center GPU, but maybe I could be wrong

[flashpixx]

The driver seems to be the version 575.57.08
https://www.nvidia.com/en-us/drivers/details/245537/

[smira]

That is not a production version yet within NVIDIA, only 570.x is the production release. So I guess once backported it might be available, but also keep in mind, this is more consumer GPU, not a server GPU, so NVIDIA might support it different way.

[flashpixx]

Sure, but will this support by Talos itself, if the driver is available?
I think there are a lot of options to use not a server GPU, in general it would be nice if Talos supports both

[smira]

We ship two version of drivers that are marked by NVIDIA as Production and LTS. At the moment it is 570.x and 535.x.

[flashpixx]

So I have to wait for the 575 update, is this correct?

[smira]

Please see my response above. Sorry, we don't have any insight into the way NVIDIA supports your particular GPU in their drivers series, neither on the topic of production releases/versions.

[delprofundo]

Hey Flashpixx et al,

glad others are playing in this space.

We have some 5070's that we can target directly with a Runtime class which is positive but we explode when trying to install the Nvidia device plugin or operator, 'cant find' as is reported in a lot of places.

Thats fine I get bleeding edge and all that, but im just unclear on the version discourse.

In this conversation I see 'two sets of drivers, one for production and one for lts', that makes total sense to me, but what I am struggling to work out is what is 'production'. Ive spent a few hours pouring over all the Nvidia driver docs I can find and I dont see anywhere a reference to a 'production' version, but I do see that version 580.76.05 is now 'recommended'.

My qs:

• is recommended the same as production, if thats the case we should expect these drivers soon?
• if its not the case, is this something that we can use the factory container to create ourselves? if there was somewhere that had guidance on that id happily try and contribute, we are just poc-ing for a sovereign data centre project at the moment so we probably have some time and I would love to contribute to talos, the best k8s platform Ive worked with.
• is there a place someone knows of to see 'this is the production and LTS versions' so I can pin the link?

[frezbo]

https://docs.nvidia.com/datacenter/tesla/drivers/ this cleary says 570 is production, 580 is the lts version

[delprofundo]

clear to you. thanks for the link