Questions about the Necessity of NVIDIA Driver Config Parameters

[michaelchen1996]

I have a question regarding the necessity of specific NVIDIA configuration parameters in the current version of DeepEP.

options nvidia NVreg_EnableStreamMemOPs=1 NVreg_RegistryDwords="PeerMappingOverride=1;"

However, in our tests, after removing these two parameters from initramfs, we still observe the following outputs:

NIC buffer will be on GPU memory
NIC handler will be GPU

From my observations, the version of cuStreamWaitValue64 used in nvshmem is 11070_v2, as indicated by:

LOAD_SYM(table, cuStreamWriteValue64, 11070, _v2, 1);
LOAD_SYM(table, cuStreamWaitValue64, 11070, _v2, 1);

According to the NVIDIA CUDA Driver API documentation, the NVreg_EnableStreamMemOPs=1 parameter is only required for v1 of these functions.

Additionally, does the PeerMappingOverride parameter only take effect in the case of non-root users?

So, are the parameters NVreg_EnableStreamMemOPs=1 and NVreg_RegistryDwords="PeerMappingOverride=1;" still necessary in the current DeepEP?

What impact (if any) would there be without these parameters—for instance, on performance?

Thank you!

[sphish]

I’m not an expert on CUDA drivers, but I suspect this might be related to recent updates in NVSHMEM. You could try asking about it in the NVSHMEM repository.