NCCL timeout while different ranks execute DeepEP and NCCL communications in different order

[defei-coder]

Reproduced code

if rank % 2 == 0:
    with torch.cuda.stream(test_alt_stream):
        dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)

buffer.low_latency_dispatch(current_x, topk_idx, num_tokens, num_experts,
                            use_fp8=dispatch_use_fp8, round_scale=round_scale, use_ue8m0=use_ue8m0,
                            cumulative_local_expert_recv_stats=cumulative_local_expert_recv_stats,
                            async_finish=not return_recv_hook, return_recv_hook=return_recv_hook)

if rank % 2 != 0:
    with torch.cuda.stream(test_alt_stream):
        dist.all_gather_into_tensor(all_topk_idx, topk_idx, group=group)
default_stream.wait_stream(test_alt_stream)

@sphish mentioned in #414: ' If each rank executes DeepEP and NCCL communications in the same order, there should be no conflicts. ' What is the reason for this restriction?

[sphish]

DeepEP’s low-latency kernels use cooperative launch to attempt launching a large number of SMs simultaneously. If NCCL occupies some of the SMs, it may prevent DeepEP’s kernels from being launched, which could result in a deadlock.

[defei-coder]

DeepEP’s low-latency kernels use cooperative launch to attempt launching a large number of SMs simultaneously. If NCCL occupies some of the SMs, it may prevent DeepEP’s kernels from being launched, which could result in a deadlock.

@sphish Thanks for your reply. I set low-latency kernel just use 64 Blocks while NCCL uses 2 Blocks, timeout still occur. In case, I disabled the cluster feature by setting attr[1].val.clusterDim.x = 1. On the other hand, for each rank, ag is executed first, followed by deepEP. The torch profiler shows that ag and deepEP can be executed in parallel on the two streams. SM resources are a potential risk, but not the root cause of the current problem.

[defei-coder]

A question about “If NCCL occupies some of the SMs, it may prevent DeepEP’s kernels from being launched“, Why can't deepEP wait for NCCL to finish？

[xiaofanl-nvidia]

Hi @defei-coder can you confirm if both the allgather and low_latency_dispatch operations are collective operations for all ranks?
I'm suspecting the above issue is simply caused by doing two collective operations in the wrong order.

E.g. collective ops require all ranks to participate to make progress. If the even ranks doing allgather is waiting for odd ranks to participate, meanwhile odd ranks are doing another collective operations and waiting for even ranks, it would naturally deadlock..

Let me know if that makes sense.. It's a guess since I'm not super familiar with low_latency_dispatch yet. :)

[defei-coder]

Thaks for your reply @xiaofanl-nvidia . In my view, low_latency_dispatch differs from all_gather in that low_latency_dispatch accomplishes communication waiting through hook functions, eliminating the need for all ranks to execute simultaneously. @sphish @LyricZhao Can you help me answer this question?

Hi @defei-coder can you confirm if both the allgather and low_latency_dispatch operations are collective operations for all ranks? I'm suspecting the above issue is simply caused by doing two collective operations in the wrong order.

E.g. collective ops require all ranks to participate to make progress. If the even ranks doing allgather is waiting for odd ranks to participate, meanwhile odd ranks are doing another collective operations and waiting for even ranks, it would naturally deadlock..

Let me know if that makes sense.. It's a guess since I'm not super familiar with low_latency_dispatch yet. :)