[model] Support Qwen3next

[sufubao]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @sufubao, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the model support by integrating the Qwen3next model. This integration required developing a new model architecture, implementing specialized inference logic for its hybrid attention and Gated Delta Net components, and creating highly optimized Triton kernels for performance. Additionally, the underlying memory management system was enhanced to accommodate the new model's state requirements and improve modularity across existing models.

Highlights

• Qwen3next Model Integration: This pull request introduces comprehensive support for the Qwen3next model, including its unique hybrid architecture that combines traditional full attention layers with Gated Delta Net (GDN) layers. This involves new model definition, layer inference logic, and specialized weight management classes.
• Specialized Triton Kernels: To optimize performance for Qwen3next's GDN layers, several new Triton kernels have been added. These include kernels for causal 1D convolution, fused GDN gating, gated RMSNorm, Gemma RMSNorm, and a suite of Flash Linear Attention (FLA) operations for efficient chunked and recurrent computations.
• Memory Management Enhancement: The core memory manager has been refactored to introduce a get_kv_buffer method, centralizing access to KV buffers. This change has been propagated across various existing models (Bloom, Deepseek2, Llama, etc.) to ensure consistent and flexible memory handling. A dedicated Qwen3NextMemoryManager is also introduced to manage both KV and Mamba-style states.
• New Weight Parameter Types: New ParameterWeight and TpParameterWeight meta-weight classes have been added to handle specific parameter loading and tensor parallelism splitting, likely tailored for the Qwen3next model's architecture.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for the Qwen3-Next model, a complex architecture featuring a mix of standard attention and Gated Delta Net (GDN) layers, along with a Mixture-of-Experts (MoE) design that includes a shared expert. The implementation is comprehensive, adding new layer inference and weight classes, a specialized memory manager for handling both KV cache and SSM-like states, and numerous Triton kernels tailored for the model's unique operations. Additionally, the PR includes a valuable refactoring across the codebase to encapsulate KV cache access, improving code structure. My review identifies a minor design issue regarding state management in the transformer layers, but overall, the changes are well-structured and robust.