Mixture of Experts (MoE)

A mixture of experts (MoE) is a model architecture that splits the network into many specialized sub-networks and routes each input to only a few of them, activating a fraction of total parameters per token.

By using a router to engage only the relevant experts, an MoE model can have a very large total parameter count while keeping the compute per token comparable to a much smaller dense model.

This sparse activation is a way to raise capability without a proportional rise in inference cost, a trade-off relevant to the price of AI labor.