Implement DualPipe training for moe model

[skydoorkai]

Is your feature request related to a problem? Please describe.
DualPipe is an efficient pipeline parallel implementation for moe models. This feature request is to implement moe model training using DualPipe .

Describe the solution you'd like
Given an example of moe dualpipe training and put it under examples/moe_dualpipe/
Useful modules or APIs can put in suitable atorch path, such as atorch/modules/moe/

Use All2All for moe token communication first, then consider using deep_ep dispatch/combine for optimization.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

[xhx1022]

@skydoorkai I'd like to work on this issue

[skydoorkai]

Step 1: Write a moe model (similar to this example ), which has the same interface with megatron model_provider, and write a training code to train it on single-GPU with dummy data, run it successfully on 1 GPU. ( 1 week)

Step 2: Add a naive implementation of overlapped_forward_backward in the moe model, make it functional (no need to consider performance, so no need to consider forward/backward compute/communication overlapping), and run it successfully in multi-gpu dualpipe training. (2 weeks)

Step 3: Optimize overlapped_forward_backward implementation by break the forward/backward computation into different compute calls according to DeepSeek's training profiling to make computation/communication partly overlapping. ( 3 week)

Step 4: Use DeepEP for moe communication, to make better computation/communication overlapping. Megatron's Flex Token dispatcher is use deepep for moe. Use it as reference. (2 week)