Looking at Kimi-K3 expert routing, brought this subject of what an expert means back to the fore. Is the term expert indicative of specialization – if so what kind. Let’s look at the literature for how the term is currently seen and how it evolved.
Routing-analysis studies of Mixture of Experts split into three camps.
Camp A — Interpretable specialization is observed.“experts do specialize, they often specialize along lines intuitive to humans, and that the representational complexity of an input affects the degree to which experts specialize,” (Stanford CS231n). A dedicated syntactic study, “Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models”, reports that “findings from six popular MoE models reveal expert specialization for specific POS categories,” and that “routing paths” show “high predictive accuracy for POS,” demonstrating measurable linguistic structure in routing decisions even though the paper is careful to note that “in most cases, these experts are not trained to specialize in specific parts of the data; rather, they are subdivisions of the dense FFN layer into smaller networks aimed at computational efficiency” (arXiv:2412.16971) — specialization, where it appears, is emergent, not designed-in.
Camp B — Specialization is real but shallow/token-level, not conceptual. DeepSeekMoE’s own diagnosis (see Section 4) implies that without architectural intervention (fine-grained segmentation, shared experts), ordinary MoE experts default to “knowledge hybridity” — each expert absorbing “vastly different types of knowledge” rather than a coherent subject-matter competence (arXiv:2401.06066). The specialization that does emerge in vanilla Switch/GShard-style models tends to be at the level of superficial token statistics (part-of-speech, punctuation, high-frequency subwords) rather than semantic domains or “topics” a human would recognize as a coherent area of expertise.
Camp C — Apparent specialization is largely an artifact of geometry, not a designed property. The most skeptical and recent analysis, “The Myth of Expert Specialization in MoEs: Why Routing Reflects…” (arXiv:2604.09780), argues that because the router is just a linear projection of the hidden state (gtℓ=Pℓhtℓ), “tokens with similar hidden states must activate similar experts” as a mathematical near-tautology, not because experts encode any independently meaningful semantic role. Its central claim: “specialization is therefore an emergent property of the representation space, not of the routing architecture itself,” and, more pointedly, “without an explicit objective encouraging human-aligned structure… we should not expect the hidden state and router logit space to have a metric system aligned with human standards”. Under this reading, when POS or topical patterns are observed in routing, they are downstream echoes of how the model’s internal representation space happens to cluster tokens for reasons unrelated to expert design — the router did not learn “grammar expert” or “punctuation expert” roles; it learned a linear boundary in an already-clustered space, and post-hoc labeling imposes human categories onto what may be a more arbitrary partition.
MoE literature survey table: the evolving meaning of “Expert”
| Era / Paper | Year | What “expert” means | Primary motivation | Specialization evidence |
|---|---|---|---|---|
| Jacobs, Jordan, Nowlan, Hinton — “Adaptive Mixtures of Local Experts” | 1991 | A complete, independent feed-forward network with fully decoupled weights | Reduce interference in multi-task learning by dividing input space into subtasks | Directly demonstrated: vowel-pair decision boundaries per expert |
| Jordan & Jacobs — “Hierarchical Mixtures of Experts and the EM Algorithm” | 1993/94 | A GLIM-based local regression model within a recursive input-space partition | Rigorous probabilistic (EM) formulation of the 1991 architecture | Local regression surfaces validated on robot-arm dynamics |
| Shazeer et al. — “Outrageously Large Neural Networks” | 2017 | One of up to ~131,000 feed-forward sub-networks inside an MoE layer between LSTM layers | Conditional computation for >1000x parameter capacity at near-constant compute | Not a stated goal; load-balancing losses used instead |
| Lepikhin et al. — GShard | 2020 | Inherited FFN-slot definition; sharded across accelerators | Scale MoE Transformers past 600B parameters via automatic sharding | Not analyzed in the paper |
| Fedus, Zoph, Shazeer — Switch Transformer | 2021 | One homogeneous FFN selected via simplified top-1 routing | Trillion-parameter scaling with “simple and efficient sparsity” | Not a central claim; homogeneity emphasized |
| Du et al. — GLaM | 2021 | FFN expert bank in a decoder-only LM, 64 experts/layer | Beat GPT-3 with 49% fewer FLOPs/token, 65% less power | Not analyzed; efficiency is the headline metric |
| Fedus, Dean, Zoph — “A Review of Sparse Expert Models” | 2022 | Formal survey definition: parameters partitioned into experts with unique weights, selected by router | Synthesize the field around parameter/compute decoupling | Explicitly notes historical shift from experts-as-whole-networks to experts-as-layer |
| Jiang et al. — Mixtral of Experts | 2024 | Explicitly “feedforward blocks (i.e. experts),” 8 per layer, top-2 routed per token | Access 47B parameters while using only 13B active per token | Selected pair varies per timestep; no persistent specialization claim |
| Dai et al. — DeepSeekMoE | 2024 | Fine-grained FFN segments plus always-on shared experts, engineered for specialization | Explicitly fix “knowledge hybridity” and “knowledge redundancy” pathologies | Architecture re-designed specifically to induce non-overlapping, focused expert knowledge |
Secure Machinery 