AI Systems Engineering
Mixture of Experts, Explained: How Sparse Models Work
If you have watched two AI labs ship huge models in the same week and wondered how either of them is affordable to run, the answer is almost certainly Mixture of Experts, or MoE. It is the single architectural idea that let Alibaba ship Qwen3.8-Flash (125 billion parameters, only 6 billion active per token) and Z.ai ship GLM-5.3-Flash (320 billion parameters, only 18 billion active per token) in the same week in late August 2026, both while keeping inference costs low enough to sell for cents per million tokens.
This post goes past the one-line explainer. We will look at why dense models hit a wall, how the router actually decides which expert handles which token, why load balancing is the hardest part of training these models, and when MoE is the wrong choice entirely.
The problem dense models run into
Every standard transformer layer has two main parts: attention, and a feed forward network (FFN). The FFN is usually the majority of a model's parameters. In a dense model, every single parameter in that FFN gets used for every single token you feed the model, whether the token is "the" or a gnarly piece of Python.
That is wasteful. Language is not uniformly difficult. Some tokens need heavy reasoning, some are trivial completions the model has seen a million times. A dense model can't tell the difference. It always pays full price.
This creates a scaling problem. If you want a smarter model, the traditional lever is "make it bigger." But making a dense model bigger means every token costs proportionally more compute, forever, at both training and inference time. Costs and latency scale directly with total parameter count. That is the wall MoE was built to get around.
What MoE actually changes
Instead of one FFN that every token passes through, a Mixture of Experts layer replaces that single FFN with many smaller FFNs, called experts, sitting side by side. A small neural network called the router looks at each token and decides which experts are relevant to it. Only those experts fire. The rest sit idle for that particular token.
This is the core trick: total parameters (everything stored in the model) and active parameters (what actually computes for a given token) become two different numbers. Qwen3.8-Flash's 125 billion total parameters and 6 billion active parameters is exactly this split in action. GLM-5.3-Flash's 320 billion total and 18 billion active is the same idea at a different scale.
The router and top-k selection
Here is roughly what happens inside an MoE layer for a single token:
- The token's vector representation is passed through the router, a simple learned linear layer
- The router outputs a score for every expert, essentially "how relevant is this expert to this token"
- The system picks the top-k highest scoring experts (k is usually a small number, like 1, 2, or 8, depending on the model)
- The token is processed by only those k experts
- Their outputs are combined, weighted by how confident the router was in each one
This idea traces back to Noam Shazeer and colleagues' 2017 paper "Outrageously Large Neural Networks," which introduced sparsely gated MoE layers and top-k routing for language models, along with an auxiliary loss to keep expert usage balanced. Google's Switch Transformer in 2021 simplified this further by showing that routing to a single expert (k=1) worked well and cut routing overhead substantially. Mixtral 8x7B, released by Mistral in December 2023, brought sparse MoE into mainstream open-weight use with top-2 routing across 8 experts per layer, and proved a sparse model could match or beat much larger dense models while activating only a fraction of the parameters per token.
Why load balancing is the hard part
There is an obvious failure mode here: what stops the router from just always picking the same two or three experts and ignoring the rest? Nothing, unless you explicitly stop it.
If a handful of experts get all the traffic, those experts become overloaded bottlenecks while the rest of the model sits there trained on almost nothing, wasting parameters you paid to train. This is called router collapse, and it was one of the first big problems MoE researchers had to solve.
The fix is an auxiliary loss added during training, on top of the normal language modeling loss, that penalizes uneven expert usage and pushes the router toward distributing tokens more evenly across all available experts. Shazeer's original 2017 paper introduced this idea, and Switch Transformer and Google's GShard both built on it with their own versions of a load balancing loss. Getting the weighting of this auxiliary loss right matters a lot: too weak and you get collapse, too strong and it fights the main training objective and hurts model quality.
Fine-grained experts: the next refinement
An important later development came from DeepSeek's DeepSeekMoE architecture, introduced in January 2024. Instead of a handful of large experts, DeepSeekMoE splits each expert into many smaller, more fine-grained experts by dividing the FFN's hidden dimension, while activating proportionally more of them per token. This lets the router mix and match a much larger combinatorial set of specializations for each token, instead of being stuck choosing between a few coarse, generalist experts.
DeepSeekMoE also introduced shared expert isolation: a small number of experts are always active for every token, regardless of what the router picks. These shared experts are meant to absorb common knowledge that would otherwise get redundantly duplicated across many routed experts, freeing the routed experts to specialize more sharply. Modern frontier MoE models, including GLM-5.3-Flash's architecture of 288 routed experts with 8 selected per token, build directly on this fine-grained, shared-expert-plus-routed-experts pattern.
Reading the numbers correctly: Qwen3.8-Flash and GLM-5.3-Flash
This is where the theory becomes concrete. Both models released in the same week in late August 2026 are useful side by side because they sit at very different scales but solve the same problem.
Qwen3.8-Flash-Next, released by Alibaba's Qwen team, has 125 billion parameters in its main MoE body, plus an additional 51 billion parameter N-gram embedding table (a separate mechanism that stores common word groups and can run on system RAM rather than GPU memory), while activating only about 6 billion parameters per token. Alibaba describes it as an early architecture preview of the upcoming Qwen4 generation, built on a hybrid attention design combining Gated DeltaNet linear-attention layers with Qwen Sparse Attention.
GLM-5.3-Flash, released by Z.ai, has roughly 320 billion total parameters across 45 layers, routing each token through 8 of 288 experts, for about 18 billion active parameters per token. It is Z.ai's first natively multimodal model in the GLM-5 series and combines linear attention with sparse attention layers to reduce long-context serving costs, supporting up to a 1 million token context window.
The active-to-total ratio is the number that actually matters for your compute bill. Qwen3.8-Flash runs at roughly 20:1 (125B total to 6B active), and GLM-5.3-Flash at roughly 18:1 (320B total to 18B active). Both are, in compute terms per token, closer to a 6 billion and 18 billion parameter dense model respectively, while carrying the knowledge capacity of something many times larger.
The tradeoff nobody skips mentioning enough: memory still counts
Here is the part that trips people up. Active parameters govern compute cost and latency. They do not govern memory. Every expert in an MoE model, whether the router picks it for a given token or not, has to be loaded into memory somewhere, because the router's decision changes token by token and you cannot predict in advance which experts you will need.
This means GLM-5.3-Flash, despite only computing with 18 billion active parameters per token, still needs its full 320 billion parameters resident in memory, roughly 320GB at FP8 precision. That is not a laptop workload. It requires serious multi-GPU infrastructure or CPU offloading frameworks built specifically for this, which is exactly why tools like KTransformers exist. The same logic applies to Qwen3.8-Flash: its FP8 checkpoint is over 170GB even though it computes like a 6 billion parameter model.
So the honest framing is: MoE trades compute cost for memory footprint. You get frontier-level capability at a fraction of the FLOPs per token, but you pay for it in VRAM or system RAM to hold the full parameter set. Anyone evaluating an MoE model for self-hosting needs to check both numbers, not just the flattering "active parameters" figure in the marketing copy.
When MoE is the wrong choice
Following the same standard I hold every lesson to: knowing when not to use something matters as much as knowing how it works.
Small-scale or edge deployment. If you cannot fit the full parameter set in memory, the compute savings of MoE are irrelevant. A smaller dense model that fully fits your hardware will usually serve you better than an MoE model you have to awkwardly offload.
Distributed training overhead. Routing decisions in MoE models require all-to-all communication between devices when experts are spread across a training cluster, since a token processed on one device might need an expert that lives on another. This communication overhead is real and can eat into the theoretical compute savings if your infrastructure is not built for it.
When dense is simply good enough. If your workload is narrow and a smaller dense model already hits your quality bar, the extra architectural and serving complexity of MoE, router training instability, load balancing tuning, expert-parallel deployment, is added engineering cost for a benefit you do not need.
The practical takeaway
The releases of Qwen3.8-Flash and GLM-5.3-Flash in the same week are not really about either model individually. They are evidence that MoE has moved from a research curiosity to the default architecture for any lab trying to ship a large, capable, and affordably-served model. The pattern is now standard enough that the interesting question for developers is no longer "is this model using MoE," it is "what is this model's active-to-total ratio, and does my infrastructure fit the memory requirement that ratio implies."
FAQ
What is the difference between total parameters and active parameters in an MoE model?
Total parameters are everything stored in the model and required in memory. Active parameters are the subset that actually computes for a given token, selected by the router. A model like GLM-5.3-Flash has 320 billion total parameters but only computes with about 18 billion of them per token.
Why do MoE models still need a lot of memory if only a few experts are active?
Because the router's choice of experts changes with every token, and you cannot know in advance which experts will be needed. All experts must be loaded into memory to be available, even though only a fraction compute on any single token.
Is Mixture of Experts the same as ensemble learning?
No. Ensemble learning runs multiple full models and combines their outputs, using more compute per prediction. MoE routes each input to a small subset of specialized sub-networks within a single model, using less compute per prediction than a dense model of equivalent size.
Do Qwen3.8-Flash and GLM-5.3-Flash use the same MoE design?
No. They share the core idea of sparse routing but differ in architecture. GLM-5.3-Flash routes through 288 experts, 8 per token, and combines linear and sparse attention. Qwen3.8-Flash-Next uses a hybrid Gated DeltaNet and Qwen Sparse Attention stack alongside a separate N-gram embedding component.
Should I use an MoE model for a small side project?
Usually not, unless a hosted API handles the serving for you. Self-hosting an MoE model still requires memory for its full total parameter count, which is often far more demanding than a dense model with similar active compute.