AMD unveils fully open-source MoE large language model Instella-MoE; its 16-billion parameter scale challenges mainstream open-source models.

AMD has announced the launch of Instella-MoE, a fully open-source Mixture of Experts (MoE) model. Boasting a total of 16 billion parameters with 2.8 billion activated per token, it is claimed to deliver leading performance among open-source language models of the same scale. AMD stated that Instella-MoE was trained entirely from scratch using its own AMD Instinct MI300X and MI325X GPUs, along with the ROCm software stack. It adopts architectural innovations including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective to enhance training and inference efficiency. Performance tests show that the base model Instella-MoE-16B-A3B has an average score of 76.7, ranking among top open-source models, outperforming models such as SmolLM3-3B and OLMo-3-7B, and competing with larger-scale models despite activating only 2.8 billion parameters. Additionally, the model supports 64K-token long context processing, and has completed a full training pipeline covering pre-training, mid-training, long context extension, supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (RL). AMD is simultaneously releasing all of Instella-MoE’s model weights, training configurations, data ratios, intermediate checkpoints, and inference code to advance open AI model research and reproducibility. The company noted that Instella-MoE demonstrates the capability to train large-scale MoE models based on AMD hardware and its open software ecosystem, and will continue developing larger-scale, more capable, and more efficient open-source language models in the future.