Require nvidia-persistenced for nvidia-cdi-refresh

[elezar]

No description provided.

[steven-bellock]

Suggested change

	# We difine nvidia-perisistenced.service as an (After) Requisite to ensure that this
	# serivce only starts if that is already started.
	# Specify nvidia-perisistenced.service as a dependency to ensure that this service
	# only starts if that has started. This is required for Confidential Compute.

[steven-bellock]

@yf23 this solution requires nvidia-persistenced to be started by systemd as outlined in the deployment guide https://docs.nvidia.com/cc-deployment-guide-tdx.pdf. Does this seem reasonable?

[elezar]

@yf23 this solution requires nvidia-persistenced to be started by systemd as outlined in the deployment guide https://docs.nvidia.com/cc-deployment-guide-tdx.pdf. Does this seem reasonable?

It is often recommended on our end, but I don't know whether we can call this a requirement. Is there a way to ONLY add this requirement IF it is actuall present?

[ArangoGutierrez]

@elezar Thanks for this PR, Could you provide a PR description detailing the motivation behind this change? is the goal to address a specific user need, eg CoCO/CC, or a more broad motivation behind making the cdi-refresh service more stable?

[yf23]

@yf23 this solution requires nvidia-persistenced to be started by systemd as outlined in the deployment guide https://docs.nvidia.com/cc-deployment-guide-tdx.pdf. Does this seem reasonable?

Hi @steven-bellock,

Thanks for putting up the solution. Overall it looks good to me, with one small question:
Does this dependency make sure the CDI refresh happens after Nvidia persistence daemon starts OR after Nvidia persistence mode enablement finishes? One example execution order (espcially on HGX with 8 GPUs when persistence mode enablement takes longer time) would be:

Nvidia persistence daemon starts
Nvidia persistence daemon enables persistenced mode starts
Nvidia CDI refresher daemon starts
Nvidia CDI refresher triggers CDI refresh (which calls nvidia-smi)
Nvidia persistence daemon enables persistenced mode finishes

In this case it could still have the problem

[steven-bellock]

Thanks for putting up the solution.

All credit goes to @elezar.

Execution of nvidia-persistenced will not return until all GPUs have been initialized. https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html says (emphasis mine)

After= ensures the opposite, that the listed unit is fully started up before the configured unit is started.

so it sounds like that should be OK.

[yf23]

Execution of nvidia-persistenced will not return until all GPUs have been initialized.
so it sounds like that should be OK.

Thanks @steven-bellock @elezar for the solution & explanation! Looks good to me and appreciate your help.