[EP] low latency EFA connectionless mode

[MaoZiming]

Description

Please include a summary of the changes and the related issue.

EP= 16

# Node 0
[rank 0] Dispatch bandwidth: 28.92 GB/s, avg_t=259.77 us | Combine bandwidth: 54.70 GB/s, avg_t=265.75 us
[rank 1] Dispatch bandwidth: 28.17 GB/s, avg_t=266.63 us | Combine bandwidth: 56.28 GB/s, avg_t=258.28 us
[rank 4] Dispatch bandwidth: 27.80 GB/s, avg_t=270.16 us | Combine bandwidth: 57.44 GB/s, avg_t=253.07 us
[rank 6] Dispatch bandwidth: 29.61 GB/s, avg_t=253.73 us | Combine bandwidth: 54.98 GB/s, avg_t=264.41 us
[rank 5] Dispatch bandwidth: 28.13 GB/s, avg_t=267.04 us | Combine bandwidth: 57.10 GB/s, avg_t=254.60 us
[rank 7] Dispatch bandwidth: 30.47 GB/s, avg_t=246.56 us | Combine bandwidth: 52.96 GB/s, avg_t=274.50 us
[rank 2] Dispatch bandwidth: 28.41 GB/s, avg_t=264.41 us | Combine bandwidth: 56.00 GB/s, avg_t=259.58 us
[rank 3] Dispatch bandwidth: 27.56 GB/s, avg_t=272.60 us | Combine bandwidth: 57.68 GB/s, avg_t=252.03 us

# Node 1
[rank 9] Dispatch bandwidth: 38.30 GB/s, avg_t=196.14 us | Combine bandwidth: 40.31 GB/s, avg_t=360.64 us
[rank 12] Dispatch bandwidth: 38.65 GB/s, avg_t=194.37 us | Combine bandwidth: 40.08 GB/s, avg_t=362.73 us
[rank 10] Dispatch bandwidth: 38.28 GB/s, avg_t=196.24 us | Combine bandwidth: 40.18 GB/s, avg_t=361.77 us
[rank 8] Dispatch bandwidth: 39.63 GB/s, avg_t=189.55 us | Combine bandwidth: 40.47 GB/s, avg_t=359.20 us
[rank 14] Dispatch bandwidth: 39.63 GB/s, avg_t=189.56 us | Combine bandwidth: 40.27 GB/s, avg_t=360.99 us
[rank 15] Dispatch bandwidth: 39.57 GB/s, avg_t=189.83 us | Combine bandwidth: 40.93 GB/s, avg_t=355.13 us
[rank 13] Dispatch bandwidth: 50.47 GB/s, avg_t=148.83 us | Combine bandwidth: 40.45 GB/s, avg_t=359.42 us
[rank 11] Dispatch bandwidth: 50.31 GB/s, avg_t=149.32 us | Combine bandwidth: 40.75 GB/s, avg_t=356.70 us

Before (main)

# Node 0
[rank 1] Dispatch bandwidth: 26.76 GB/s, avg_t=280.73 us | Combine bandwidth: 60.20 GB/s, avg_t=241.47 us
[rank 2] Dispatch bandwidth: 25.91 GB/s, avg_t=289.88 us | Combine bandwidth: 62.63 GB/s, avg_t=232.09 us
[rank 6] Dispatch bandwidth: 27.68 GB/s, avg_t=271.37 us | Combine bandwidth: 59.02 GB/s, avg_t=246.29 us
[rank 3] Dispatch bandwidth: 26.46 GB/s, avg_t=283.89 us | Combine bandwidth: 61.18 GB/s, avg_t=237.60 us
[rank 7] Dispatch bandwidth: 27.80 GB/s, avg_t=270.24 us | Combine bandwidth: 58.99 GB/s, avg_t=246.42 us
[rank 4] Dispatch bandwidth: 28.57 GB/s, avg_t=262.88 us | Combine bandwidth: 56.68 GB/s, avg_t=256.46 us
[rank 0] Dispatch bandwidth: 26.17 GB/s, avg_t=287.06 us | Combine bandwidth: 62.77 GB/s, avg_t=231.59 us
[rank 5] Dispatch bandwidth: 26.49 GB/s, avg_t=283.55 us | Combine bandwidth: 61.84 GB/s, avg_t=235.06 us

# Node 1
[rank 12] Dispatch bandwidth: 38.64 GB/s, avg_t=194.38 us | Combine bandwidth: 38.35 GB/s, avg_t=379.06 us
[rank 14] Dispatch bandwidth: 39.74 GB/s, avg_t=189.02 us | Combine bandwidth: 38.02 GB/s, avg_t=382.31 us
[rank 10] Dispatch bandwidth: 38.70 GB/s, avg_t=194.11 us | Combine bandwidth: 38.63 GB/s, avg_t=376.33 us
[rank 13] Dispatch bandwidth: 39.18 GB/s, avg_t=191.75 us | Combine bandwidth: 38.60 GB/s, avg_t=376.63 us
[rank 11] Dispatch bandwidth: 38.81 GB/s, avg_t=193.57 us | Combine bandwidth: 38.81 GB/s, avg_t=374.56 us
[rank 15] Dispatch bandwidth: 39.32 GB/s, avg_t=191.03 us | Combine bandwidth: 39.23 GB/s, avg_t=370.57 us
[rank 9] Dispatch bandwidth: 49.44 GB/s, avg_t=151.94 us | Combine bandwidth: 38.66 GB/s, avg_t=375.99 us
[rank 8] Dispatch bandwidth: 51.52 GB/s, avg_t=145.79 us | Combine bandwidth: 38.27 GB/s, avg_t=379.89 us

EP= 24

[rank 30] Dispatch bandwidth: 12.21 GB/s, avg_t=615.31 us | Combine bandwidth: 19.63 GB/s, avg_t=740.47 us
[rank 24] Dispatch bandwidth: 9.68 GB/s, avg_t=775.97 us | Combine bandwidth: 24.79 GB/s, avg_t=586.37 us
[rank 27] Dispatch bandwidth: 12.17 GB/s, avg_t=617.27 us | Combine bandwidth: 19.71 GB/s, avg_t=737.60 us
[rank 26] Dispatch bandwidth: 11.73 GB/s, avg_t=640.42 us | Combine bandwidth: 20.03 GB/s, avg_t=725.86 us
[rank 25] Dispatch bandwidth: 11.69 GB/s, avg_t=642.46 us | Combine bandwidth: 20.17 GB/s, avg_t=720.54 us
[rank 31] Dispatch bandwidth: 12.27 GB/s, avg_t=612.04 us | Combine bandwidth: 19.56 GB/s, avg_t=743.31 us
[rank 29] Dispatch bandwidth: 12.17 GB/s, avg_t=617.15 us | Combine bandwidth: 19.72 GB/s, avg_t=737.14 us
[rank 28] Dispatch bandwidth: 12.17 GB/s, avg_t=617.10 us | Combine bandwidth: 19.69 GB/s, avg_t=738.24 us

Before (main)

[rank 7] Dispatch bandwidth: 10.95 GB/s, avg_t=685.95 us | Combine bandwidth: 20.65 GB/s, avg_t=704.11 us
[rank 5] Dispatch bandwidth: 10.95 GB/s, avg_t=686.21 us | Combine bandwidth: 28.14 GB/s, avg_t=516.49 us
[rank 2] Dispatch bandwidth: 10.80 GB/s, avg_t=695.80 us | Combine bandwidth: 20.84 GB/s, avg_t=697.66 us
[rank 0] Dispatch bandwidth: 11.00 GB/s, avg_t=683.11 us | Combine bandwidth: 20.52 GB/s, avg_t=708.35 us
[rank 6] Dispatch bandwidth: 11.05 GB/s, avg_t=679.53 us | Combine bandwidth: 20.42 GB/s, avg_t=711.97 us
[rank 3] Dispatch bandwidth: 10.84 GB/s, avg_t=693.06 us | Combine bandwidth: 20.77 GB/s, avg_t=699.95 us
[rank 1] Dispatch bandwidth: 10.91 GB/s, avg_t=688.44 us | Combine bandwidth: 20.68 GB/s, avg_t=703.05 us
[rank 4] Dispatch bandwidth: 10.94 GB/s, avg_t=686.92 us | Combine bandwidth: 20.71 GB/s, avg_t=702.05 us

Type of Change

• Bug fix
• New feature
• Documentation update

How Has This Been Tested?

Include any tests here.

• Unit tests
• Integration tests
• Manual testing

Checklist

• My code follows the style guidelines, e.g. format.sh.
• I have run build_and_install.sh to verify compilation.
• I have removed redundant variables and comments.
• I have updated the documentation.
• I have added tests.