The Mixture of Idiots

MoE (Mixture of Experts) is LLM operation mode where many small models, highly specialized in particular areas, are used instead of one ‘dense’ model. A routing layer in front of these 20-50 ‘experts’ decides which one to use.

MoE cannot replicate dense inference. There is no cross-linking compressed cognition (for example, insights from molecular biology advising quantum mechanics, etc.) MoE is wide spread degradation of accessible inference.

The shift to MoE was not driven by a quest for a higher upper bound of intelligence. It was an engineering compromise designed to solve an economic crisis: the catastrophic inference costs and memory bandwidth bottlenecks of serving massive models at scale.

The Geometry of Cross-Linking

In a massive dense model (like a true 200B+ parameter monolithic network), every single parameter participates in every forward pass.

This forces the model to compress all of its training data into a single, unified latent space. Because molecular biology and quantum mechanics are mapped into the exact same geometrical embedding space, the model is forced to learn the abstract, structural similarities between them. When one prompts a dense model to synthesize two disparate fields, the parameters inherently “know” how to bridge them because the cognition is holistic.

The MoE Routing Bottleneck

MoE fundamentally breaks this holistic compression by relying on piecewise optimization.

Instead of a unified space, MoE uses a router to send tokens to specific, specialized sub-networks (experts). If one asks a question requiring insights from both molecular biology and quantum mechanics, the MoE faces a hard mathematical bottleneck:

1. Siloed Parameters: The “biology expert” and the “physics expert” update independently during training. They do not share the same dense interconnections.

2. Hard Token Assignment: If the router decides a token only needs the top 2 experts, the remaining 6 or 30 experts are completely ignored for that forward pa…