Your feed is explaining Mixture of Experts to you right now. Threads with colourful diagrams, videos with dramatic music, "here's the architecture behind the new models". Sounds like a breakthrough from last week.
The idea is from 1991.
Back then four researchers — including Geoffrey Hinton, who won the 2024 Nobel Prize in Physics — described a method called "Adaptive Mixtures of Local Experts". In 2017 a Google team turned it into the sparse version everyone uses today. About as new as the combustion engine.
And yet the excitement has a real core. It just sits somewhere other than the threads claim.
What Mixture of Experts actually is
Picture a large hospital. Hundreds of specialists, every department with its own field. You walk in with a broken arm — and all four hundred of them do not come storming into the treatment room. Someone at reception decides in seconds: trauma surgery, second floor.
The hospital is enormous. Your appointment is still short.
That is exactly how an MoE model works. It consists of many sub-networks, the "experts". In front of them sits a small component, the router — reception, in our picture. For every scrap of text the router decides which two, four or eight experts get to work. The rest stay switched off.
The opposite design is called "dense". There, the complete model runs on every single request. Every word, every time, all of it. Expensive.
What is actually new
This is where it gets interesting — and where most explainer threads stop.
An MoE model has two numbers: the total size, meaning how many parameters are in there at all, and the active parameters, meaning how many of them actually do the maths for each answer. For years those two sat reasonably close together. Right now they are drifting apart brutally:
- DeepSeek V4 Flash: 284 billion parameters in total, roughly 13 billion of them active.
- Qwen 3.8 Max: 2.4 trillion in total, around 95 billion of them active.
- Kimi K3: 896 experts on board, 16 of them doing the work.
That is roughly 2 to 5% of the model per answer. So the models are not simply getting bigger — they are getting emptier. Providers keep stacking more knowledge onto the shelf while the compute per request barely moves.
That is the real trend your feed is celebrating without naming it.
Why this concerns you
Two things, quite practical.
Parameter counts in headlines barely mean anything anymore. When you read "2.4 trillion parameters", that is warehouse size, not thinking power per answer. If you want to compare models, look for the active number — it is often right there in the name. A label like 35B-A3B means: 35 billion parameters total, 3 billion of them active.
This is why AI is getting cheaper without getting worse. Prices per request have fallen lately while quality went up. That is not marketing magic but, to a good extent, exactly this architecture — more on that in What AI really costs.
What the hype threads leave out
Two caveats that rarely show up when someone sells MoE as a silver bullet.
Only the computing is sparse, not the memory. Those 284 billion parameters have to be fully loaded, even though only 13 billion are working. Which means: MoE does not suddenly make self-hosting easy. You still need room for the whole hospital, even if you only visit trauma surgery.
The "experts" are not fields of expertise. This is the most stubborn misconception from the explainer videos. There is no law expert sitting next to a maths expert. What one of these experts has learned usually maps to no clean human topic at all — the model worked out the division by itself during training, along patterns nobody handed it. "Expert" is a technical term, not a job title.
Do you need to know this?
Honestly: no. You can use ChatGPT, Gemini or DeepSeek without ever having heard the word router.
But it explains two things you keep running into: why the parameter counts in headlines keep getting more absurd — and why your AI bill does not grow along with them.
So next time someone sells you MoE as the revolution of the month: the architecture is older than most of the people posting about it. The only new thing is how "empty" the models have become.