feat(megatron): Add support for Gated Delta Net (GDN) & Kimi Delta Attention (KDA) - #676

Open

clairesonglee wants to merge 36 commits into

mainfrom

clairlee/kda-optimized-training-patch

Commits on Jan 29, 2026

initial commit

clairesonglee
authored and
Lee
committed
set self.lr_warmup_steps < self.lr_decay_steps

clairesonglee
authored and
Lee
committed
unwrap model to remove loss_mask parameter

clairesonglee
authored and
Lee
committed
add zebra-llama (hybrid mla mamba model) support

Mingyuyang-1
authored and
Lee
committed
add Zebra-Llama 3B configurations

Mingyuyang-1
authored and
Lee
committed
add Zebra-Llama 1B configs and remove unused configs

Mingyuyang-1
authored and
Lee
committed
remove unused configs

Mingyuyang-1
authored and
Lee
committed
Set submodule mamba to track enable-primus-hybrid-models branch
clairesonglee
committed

Commits on Feb 3, 2026

Commits on Feb 4, 2026

Commits on Feb 5, 2026

adjust zebra-llama architecture and training
Mingyuyang-1
committed

Commits on Mar 11, 2026

add gdn support
Mingyuyang-1
committed

Commits on Mar 12, 2026

Commits on Mar 19, 2026

added patch.sh to fix fla autotune kernel bug for mi300.
vanshbhatia-amd
committed

Commits on Apr 21, 2026

working on matching h100 and primus losses
vanshbhatia-amd
committed

Commits on Apr 24, 2026

training matches fla for gdn
vanshbhatia-amd
committed

Commits on May 6, 2026

Commits on May 7, 2026

adding patch
vanshbhatia-amd
committed

Commits on May 19, 2026

Add bash-docker script and GDN training documentation
vanshbhatia-amd
committed

Commits on May 20, 2026

Add KDA training documentation and configurations for FLA parity
vanshbhatia-amd
committed

Commits on May 25, 2026

Add KDA training documentation and configurations for FLA parity
vanshbhatia-amd
committed

Commits on Jun 8, 2026

Commits on Jul 15, 2026

Commits on Jul 20, 2026

Merge branch 'main' into clairlee/kda-optimized-training-patch
clairesonglee
authored