Routing is the real problem

I have been building a small mixture-of-experts model from scratch, mostly to feel where it breaks. The headline idea is simple: instead of one big network answering everything, you train several smaller "experts" and a router that picks which ones handle each token. The promise is more capacity for the same compute, because only a few experts fire at a time.

What surprised me is how little of the difficulty lives in the experts.

The experts mostly take care of themselves

Give a sub-network a narrow enough slice of the distribution and it learns its slice. That part behaves. The experts are not where the model embarrasses you.

The router is. The router has to decide, per token, who is competent here, and it has to do that early, with less information than the experts will eventually have. Get it wrong and you either starve good experts of training signal or send tokens to experts that have no idea. Both failures are quiet. The loss still goes down. The model just gets worse in ways that do not show up until you look at which expert fired where.

Load balancing is a values problem in disguise

Left alone, a router collapses. It finds two or three experts that are slightly better early on and routes everything to them. The rest never get gradient, never improve, and the "mixture" becomes a single overworked network with dead weight attached.

The standard fix is an auxiliary loss that punishes imbalance, nudging the router to spread tokens out. But that is a tax on the thing you actually want, which is the right expert, not a fair expert. You are trading specialization for utilization, on purpose, and the coefficient on that trade is a knob nobody can set from first principles.

Why this rhymes with agent systems

This is the same shape as the multi-agent work I do at AWS, one layer down.

An agent system routes between LLM personas at the orchestration layer. A mixture-of-experts model routes between sub-networks at the parameter layer. Different altitudes, identical question: given an input, who should handle it, and how do you keep the answer from always being "whoever happened to be good last week."

In both cases the experts are the easy part. The routing is where the system is actually intelligent, or actually broken. I find that clarifying. The capability you can buy. The allocation you have to design.