rdma: Re-enable eager messages

While the eager protocol did not have an impact on nccl-tests
performance, it did have a sizeable (30% difference in step time for
Maxtext Llama2 70B on P5) impact on applications.  So re-enable the
eager protocol, and adjust the early completion detection to
automatically adjust to eager enablement.

We need to come back to the early completion protocol and find a
way to have early completion and eager co-exist in the general
case, but that's future work.

Signed-off-by: Brian Barrett <[email protected]>

Author: bwbarrett

include/nccl_ofi_param.h

@@ -342,7 +342,7 @@ OFI_NCCL_PARAM_INT(net_latency, "NET_LATENCY", -1);
  * Eager message size limit when using RDMA protocol. Message sizes greater than
  * this limit will always be sent using RDMA write instead of eagerly.
  */
-OFI_NCCL_PARAM_INT(eager_max_size, "EAGER_MAX_SIZE", -1);
+OFI_NCCL_PARAM_INT(eager_max_size, "EAGER_MAX_SIZE", 8192);
 
 /*
  * Decide whether or not mutexes should default to errorcheck mode.

src/nccl_ofi_rdma.cpp

@@ -8109,7 +8109,17 @@ int nccl_net_ofi_rdma_init(const char *provider_filter,
 	* - Provider must use FI_PROGRESS_AUTO data progress model
 	*/
 	if (ofi_nccl_early_completion() < 0) {
-		early_completion = data_progress_auto;
+		if (!data_progress_auto) {
+			NCCL_OFI_TRACE(NCCL_INIT | NCCL_NET,
+				       "Early completion disabled due to progress model");
+			early_completion = false;
+		} else if (ofi_nccl_eager_max_size() >= 0) {
+			NCCL_OFI_TRACE(NCCL_INIT | NCCL_NET,
+				       "Early completion disabled because eager is enabled");
+			early_completion = false;
+		} else {
+			early_completion = true;
+		}
 	} else if (ofi_nccl_early_completion() == 0) {
 		early_completion = false;
 	} else {