A mixture of experts (MoE) LLM is an architecture that allows models to grow to very large parameter counts, potentially trillions of parameters, without a proportional increase in the cost of using them. Instead of processing information with one large neural network layer, an MoE model uses many smaller "expert" sub-networks and a "router" that selects which experts to use for each piece of incoming data.

This approach dramatically reduces the computation needed per token, but introduces new challenges in memory management, network communication, and system complexity.

Key takeaways

  • A mixture of experts LLM replaces some dense, monolithic layers with a set of smaller, parallel expert layers.
  • A router network directs each token to a small subset of experts (e.g., the best 2 out of 8), making computation "sparse."
  • This sparsity reduces the active parameter count and arithmetic computation for each token, which can lead to faster inference for certain workloads, though wall-clock time savings are not automatic due to system complexities.
  • The primary trade-off is lower compute cost per token in exchange for much higher memory requirements, as all experts must be loaded in VRAM across the system.
  • MoE models rely on complex strategies like expert parallelism and load balancing to manage memory and ensure efficient hardware use.

What is a Mixture of Experts (MoE) LLM?

A mixture of experts (MoE) LLM is a type of Transformer model that replaces some of its standard feed-forward network layers with MoE layers. An MoE layer contains two key components: a set of smaller neural networks called "experts" and a "router" network. For each token the model processes, the router dynamically chooses a small number of experts to handle it. This conditional computation is the defining feature of MoE models.

In a standard Transformer architecture, every token passes through the same sequence of layers and calculations. If a feed-forward layer has a large number of parameters, all of those parameters are used for every single token.

An MoE model works differently. It might replace that single large layer with, for example, 8 smaller expert layers. When a token arrives, the router might decide that only Expert #3 and Expert #7 are needed. The token is processed only by those two experts, and their outputs are combined. The other six experts remain idle for that token.

The result is that the model still has a large total parameter count, but the active parameter count—the number used for any given token—is much smaller. The experts themselves are typically standard feed-forward networks (also called multi-layer perceptrons or MLPs) (Source name). Other parts of the model, like the self-attention layers, remain unchanged.

How does a Mixture of Experts model work? (Step-by-step)

An MoE layer activates a small, data-dependent portion of its network for each token. This process involves routing the token to selected experts, processing it in parallel, and then combining the results. The key is that this entire sequence happens inside what would otherwise be a single, dense feed-forward block in a standard Transformer.

Here is a step-by-step breakdown of a token's journey through an MoE layer.

  1. Input: A token's vector representation (its hidden state) arrives at the MoE layer from the preceding layer (e.g., a self-attention block).
  2. Routing: The router, a small neural network, takes the token's vector as input. It outputs a score for each available expert, indicating how suitable that expert is for processing this specific token.
  3. Selection: The router applies a function, like softmax, to its scores to create a probability distribution. It then selects the experts with the highest scores. This is often called top-k routing, where 'k' is the number of experts to select. Many modern MoE models use top-2 routing, meaning each token is sent to two experts (Source name).
  4. Dispatch: The token's vector is sent only to the selected experts. This strategy, known as token dispatch, is a common implementation for MoE execution. In a distributed system where experts are spread across multiple GPUs, this step requires network communication, often an "all-to-all" operation where each GPU sends tokens to the GPUs holding the chosen experts (Source name). An alternative approach is masked execution, where all experts compute on the data but only the results from the selected experts are preserved.
  5. Expert Processing: The one or two selected experts each process the token's vector through their own feed-forward network. Each expert has its own unique weights and will transform the vector differently.
  6. Combination: The outputs from the selected experts are combined. This is usually done through a weighted sum, where the weights are the scores originally calculated by the router. This gives more influence to the output of the expert that the router deemed most suitable.
  7. Output: The final, combined vector is passed to the next component of the model, such as the next self-attention block or layer normalization.

Why are MoE models cheaper per token but heavy on memory?

Mixture of Experts models are computationally cheaper per token because they only use a fraction of their total parameters for any given calculation. However, they are heavy on memory because the parameters for all experts must be loaded and ready in the GPUs' VRAM. This is the central trade-off of the MoE architecture: sacrificing memory to save on computation.

Think of it like a library full of specialized reference books. A dense model is like having one enormous, general-purpose encyclopedia. To answer any question, you must load and consult the entire book. An MoE model is like having a library with hundreds of smaller, specialized books (e.g., on physics, history, poetry).

  • Cheaper Per Token (Computation): To answer a physics question, a librarian (the router) directs you to read only the physics and mathematics books (the top-2 experts). You perform far less reading (computation) than if you had to read the entire encyclopedia. Similarly, an MoE model only activates a few experts, keeping the Floating Point Operations (FLOPs) per token low (Source name).
  • Heavy on Memory (Storage): To be able to answer any type of question, the entire library of specialized books must be available on the shelves. You cannot send the books you aren't using into storage. Likewise, an MoE model must keep the weights for every expert loaded in memory, because the router might need to access any of them at any time (Source name).

To manage this massive memory requirement, MoE models use expert parallelism. This strategy splits the experts across different GPUs. For example, in an 8-expert model running on 8 GPUs, each GPU might hold the weights for just one expert. When a token needs to be processed by an expert on another GPU, its data is sent over the network.

MoE in action: a worked example of token routing

To make this concrete, let's walk through an illustrative example. Imagine an MoE layer with 8 experts, and our goal is to process the token for the word "learning." The model is configured to use top-2 routing, meaning it will select the two best experts for this token.

Let's assume the token "learning" arrives at the MoE layer as a vector.

  1. Routing: The router network processes this vector and produces a score (a logit) for each of the 8 experts.

    • Expert 1: 1.2
    • Expert 2: -0.5
    • Expert 3: 4.5
    • Expert 4: 0.1
    • Expert 5: 2.3
    • Expert 6: -1.8
    • Expert 7: 3.9
    • Expert 8: 1.9
  2. Selection: The router identifies the two highest scores. In this case, Expert 3 (score 4.5) and Expert 7 (score 3.9) are the top 2.

  3. Gating Weights: The router then applies a softmax function to these scores to turn them into weights that sum to 1. For simplicity, let's say the resulting weights are:

    • Weight for Expert 3: 0.65
    • Weight for Expert 7: 0.35
  4. Expert Processing: The original "learning" token vector is sent to both Expert 3 and Expert 7. Each processes it independently.

    • Output_3 = Expert_3("learning_vector")
    • Output_7 = Expert_7("learning_vector")
  5. Combination: The final output of the MoE layer is the weighted sum of the outputs from the selected experts.

    • Final_Output = (0.65 * Output_3) + (0.35 * Output_7)

This Final_Output vector is then passed on to the next layer of the model. All other experts (1, 2, 4, 5, 6, 8) were not used for this specific token, saving a significant amount of computation.

Here's what that logic looks like in simple pseudocode:

Python
# Illustrative example
def moe_layer(token_vector, experts, router):
    # 1. Get scores from the router
    router_scores = router.predict(token_vector)  # Example: [1.2, -0.5, 4.5, ..., 3.9, 1.9]

    # 2. Select top 2 experts and get their weights
    top_k_indices, top_k_weights = find_top_k(router_scores, k=2) # Indices: [3, 7], Weights: [0.65, 0.35]

    # 3. Process with selected experts
    expert_outputs = []
    for i in top_k_indices:
        expert_outputs.append(experts[i].process(token_vector))

    # 4. Combine outputs with weights
    final_output = 0
    for i in range(len(expert_outputs)):
        final_output += top_k_weights[i] * expert_outputs[i]

    return final_output

If each of the 8 experts had 2 billion parameters, the total parameter count for this layer's experts would be 16 billion. However, for the token "learning," we only activated two of them, for an active parameter count of just 4 billion.

How do MoE models handle expert load balancing?

MoE models require load balancing mechanisms because the router might naturally learn to favor a few popular experts, leaving others underutilized. This imbalance is inefficient, as it wastes the memory holding the idle experts and can create computational bottlenecks at the overworked ones. Systems use two main techniques to manage this: an auxiliary loss function and expert capacity.

During training, a load balancing loss is often added to the model's primary loss function. This auxiliary loss penalizes the router for sending a disproportionate number of tokens in a batch to any single expert. It encourages the router to learn a gating policy that distributes tokens more evenly across the pool of available experts (Source name).

The second mechanism is expert capacity. This is a hard limit on the number of tokens an expert can accept within a single processing batch. The capacity is often set slightly higher than what would be needed for a perfectly uniform distribution. If the router sends more tokens to an expert than its capacity allows, the "overflow" tokens are dropped. These dropped tokens typically bypass the expert layer completely and are passed directly to the next layer through their residual connection. This prevents any single expert from becoming a severe bottleneck, but dropping too many tokens can degrade the model's quality.

MoE vs. dense models: which is better?

Neither architecture is strictly better; they represent different trade-offs between performance, cost, and complexity. An MoE model can achieve the quality of a much larger dense model with fewer computational flops per token during inference. However, this comes at the cost of higher memory usage, increased network communication, and more complex serving infrastructure.

Here's a comparison of the key differences:

Feature Dense Model MoE Model
Per-Token Compute High. All parameters in a layer are active for every token. Low. Only parameters of selected top-k experts are active.
Total Parameters Corresponds directly to memory and compute. Very large, but decoupled from per-token compute cost.
Memory (VRAM) High, proportional to parameter count. Very high. All experts must be loaded in memory across the system.
Communication Lower. Primarily between layers on the same or different GPUs. Higher. Requires all-to-all communication to dispatch tokens to experts on different GPUs.
Inference Latency More predictable. Can be variable due to routing, load imbalance, and network latency.
Training Complexity Simpler. Focus is on standard distributed training. More complex. Requires managing router stability and load balancing.
Serving Complexity Simpler. The model is a more uniform compute graph. More complex. Requires specialized parallelism and careful hardware provisioning.

When not to use a Mixture of Experts model

Despite their benefits, MoE models are not always the right choice. Their complexity and high memory overhead make them unsuitable for certain applications.

You may want to avoid an MoE architecture when:

  • Memory is highly constrained. On consumer hardware or edge devices with limited VRAM, loading all the experts is often impossible. A smaller dense model is a more practical choice.
  • Low-latency single-request inference is critical. The communication overhead of expert parallelism (the all-to-all dispatch) can add latency. While MoE models have high throughput with large batches, a dense model that fits on a single node might serve a single user's request faster.
  • The engineering team is small or lacks distributed systems expertise. Successfully training and deploying MoE models requires a deep understanding of model parallelism, network interconnects, and performance tuning. The operational complexity is significantly higher than for dense models.
  • The task does not benefit from a massive parameter count. For simpler tasks or domains where a model with tens of billions of parameters is sufficient, the added complexity of MoE may not provide a meaningful quality improvement to justify the costs.

Ultimately, the decision to use an MoE model is a systems co-design problem, involving trade-offs between model quality, available hardware budget, and the engineering resources needed to manage a complex distributed system.

What to learn next

Sources and further reading