Mixture-of-Experts explained: why the biggest models are mostly idle

Imagine an AI model with one trillion parameters, but only 32 billion of them participate in processing any particular token. The remaining 968 billion parameters are effectively sitting out that computation.

That is not a thought experiment. It describes the approximate relationship between total and active parameters in Kimi K2.5, released in January 2026.

This is the central idea behind Mixture-of-Experts (MoE): increase a neural network’s total capacity without increasing the computation required for every token by the same proportion.

Instead of sending every token through one enormous feed-forward network, an MoE model uses a learned router to choose a small subset of specialized networks called experts.

The result is an architecture with two very different parameter counts:

• Total parameters: everything stored in the model.

• Active parameters: the subset involved in processing one token.

Understanding that distinction helps explain how modern language models can grow to hundreds of billions or trillions of parameters while keeping per-token computation far below what an equally large dense model would require.

In this tutorial, we’ll examine the architecture, build a working sparse router using NumPy, measure expert utilization, and explore why sparse computation creates new challenges for memory, training, and distributed inference.

1. Dense models versus sparse models

How a traditional dense Transformer works

A conventional dense Transformer processes token representations through a repeated series of computational blocks.

A simplified block contains:

1. An attention mechanism that exchanges information between tokens.

2. A feed-forward network (FFN) that transforms each token’s representation.

3. Normalization and residual connections that help maintain stable information flow.

The feed-forward network typically expands the hidden representation into a larger intermediate dimension, applies a nonlinear transformation, and projects it back to the original width.

In a dense Transformer, every token passes through the same feed-forward weights at a particular layer.

Suppose an FFN contains 100 million parameters. Whether the input is about programming, travel, or mathematics, that FFN is evaluated for every token reaching the layer.

This simplicity has advantages. Dense operations are comparatively straightforward to batch, optimize, and distribute.

But it creates a constraint: increasing the size of the FFN generally increases both model capacity and per-token computation.

What an MoE model changes

An MoE layer replaces one dense FFN with multiple smaller FFNs.

These independent networks are called experts.

A router examines each token representation and selects which experts should process it. Usually, only a small number of experts are activated.

For example, imagine a layer with:

• 8 available experts.

• 2 selected experts per token.

• 50 million parameters in each expert.

The layer contains 400 million expert parameters, but each token uses only 100 million expert parameters.

That is four times the expert capacity of one 100-million-parameter dense FFN, with approximately the same expert weight computation per token, excluding routing and other overhead.

The experts are not necessarily manually assigned categories such as mathematics, language, or programming. The training process learns how to distribute different token representations across them.

The key innovation is conditional computation: the model chooses which computation to perform based on the input.

2. What sparse expert routing actually looks like

Simplified top-2 routing through one MoE layer:

token hidden state
        |
        v
     router  -> scores for experts E0..E7, keep the top 2
        |
   +----+----+----+----+----+----+----+----+
   E0   E1   E2   E3   E4   E5   E6   E7
             [*]                 [*]          <- only 2 of 8 experts run
             w=0.6               w=0.4
        |
        v
weighted sum of the 2 expert outputs -> next layer

Only two of eight expert networks execute for this token. Another token can select a completely different pair.

A sparse MoE layer has three main components.

Component A: The router

The router receives a token’s hidden representation and generates an expert score vector.

For example, with four experts, a token might receive these scores:

If top-k equals 2, the router selects Expert 2 and Expert 0.

The score is not necessarily a probability. A common implementation applies softmax to the selected scores to obtain mixture weights.

For the example above, those weights are approximately 0.711 and 0.289.

The router must be inexpensive relative to the expert computation it controls. Otherwise, the routing process itself would undermine the savings.

Component B: The experts

Each expert is a neural network that processes its assigned token representations.

In many Transformer architectures, an expert is a relatively small feed-forward network, often using a gated activation function such as SwiGLU.

Different experts maintain different learned weights.

Importantly, they receive contextual hidden representations, not merely raw words. The same word in different contexts can therefore receive different expert assignments.

Component C: The weighted combination

After the selected experts compute their outputs, the MoE layer combines them using the router weights.

In our four-expert example:

output = 0.711 × Expert₂(x) + 0.289 × Expert₀(x)

Only the two selected experts are evaluated.

The output has the same hidden dimension as the input, allowing the MoE block to fit into the surrounding Transformer architecture.

Some implementations normalize only selected scores, while others use different routing score functions, normalization rules, or shared experts. The exact weighting scheme is an architectural choice rather than a universal law.

3. Why top-k routing matters

The two key routing configuration values are:

• E: the total number of experts.

• k: the number of routed experts selected for each token.

The fraction of routed experts selected per token is:

Active expert fraction = k / E

Consider an MoE layer with 64 experts and top-2 routing.

Only 2 / 64 = 3.125% of the routed experts execute for a given token.

This sounds extraordinarily efficient, but there is an important qualification: 3.125% of experts does not mean 3.125% of the entire model’s parameters.

The model still contains always-active components, including attention projections, normalization, and potentially shared experts, embeddings, or other modules.

A useful simplified parameter estimate is:

Total MoE FFN parameters ≈ E × parameters per expert + router parameters

Active MoE FFN parameters per token ≈ k × parameters per expert + router parameters

These estimates assume equally sized routed experts, no shared expert, and no biases.

The total active parameter count of the complete model also includes its dense components.

That distinction becomes particularly important when interpreting the marketing or architectural specifications of large models.

4. Hands-on: build a sparse router with NumPy

Now let’s implement the central routing mechanism ourselves.

We will create a small MoE layer with eight experts and top-2 routing. Our implementation will:

1. Generate synthetic token representations.

2. Initialize eight independent two-layer experts.

3. Calculate router scores for every token.

4. Select the two highest-scoring experts.

5. Dispatch tokens to the selected experts.

6. Combine their outputs using routing weights.

7. Measure how many tokens reach each expert.

This is an inference-only demonstration with randomly initialized weights. It teaches the mechanics of sparse routing, not how to train a useful language model.

Step 1: Prepare the environment

We only need Python and NumPy.

Create a virtual environment on Linux or macOS:

python -m venv .venv
source .venv/bin/activate
python -m pip install "numpy>=2.0,<3"

Save the following code as moe_demo.py.

Step 2: Implement the router and experts

import numpy as np

rng = np.random.default_rng(42)

num_tokens = 12
model_width = 16
expert_width = 32
num_experts = 8
top_k = 2

# Synthetic token representations.
x = rng.normal(
    size=(num_tokens, model_width)
).astype(np.float32)

# A lightweight router.
router_w = rng.normal(
    scale=1 / np.sqrt(model_width),
    size=(model_width, num_experts),
).astype(np.float32)

# Independent two-layer MLP weights for each expert.
expert_up = rng.normal(
    scale=1 / np.sqrt(model_width),
    size=(num_experts, model_width, expert_width),
).astype(np.float32)

expert_down = rng.normal(
    scale=1 / np.sqrt(expert_width),
    size=(num_experts, expert_width, model_width),
).astype(np.float32)

# Step 1: Compute scores for all experts.
scores = x @ router_w

# Step 2: Select top-k experts for each token.
chosen = np.argsort(scores, axis=1)[:, -top_k:][:, ::-1]

chosen_scores = np.take_along_axis(
    scores, chosen, axis=1
)

# Step 3: Softmax over selected experts.
shifted = chosen_scores - chosen_scores.max(
    axis=1, keepdims=True
)

gates = np.exp(shifted)
gates /= gates.sum(axis=1, keepdims=True)

# Step 4: Allocate the MoE output and load counters.
output = np.zeros_like(x)
load = np.zeros(num_experts, dtype=np.int64)

# Step 5: Dispatch to experts and combine outputs.
for expert_id in range(num_experts):
    token_ids, slot_ids = np.where(
        chosen == expert_id
    )

    if token_ids.size == 0:
        continue

    expert_input = x[token_ids]

    # Each expert is a two-layer ReLU MLP.
    hidden = np.maximum(
        expert_input @ expert_up[expert_id],
        0.0,
    )

    expert_output = (
        hidden @ expert_down[expert_id]
    )

    # Weighted accumulation back into token order.
    output[token_ids] += (
        gates[token_ids, slot_ids, None]
        * expert_output
    )

    load[expert_id] = token_ids.size

# Step 6: Verify the routing invariants.
assert output.shape == x.shape
assert load.sum() == num_tokens * top_k
assert np.allclose(
    gates.sum(axis=1), 1.0
)

# Step 7: Calculate illustrative parameter counts.
parameters_per_expert = (
    2 * model_width * expert_width
)

router_parameters = (
    model_width * num_experts
)

total_parameters = (
    num_experts * parameters_per_expert
    + router_parameters
)

active_per_token = (
    top_k * parameters_per_expert
    + router_parameters
)

print(
    "First token -> expert IDs:",
    chosen[0].tolist(),
)

print(
    "First token -> mixture weights:",
    ", ".join(f"{v:.3f}" for v in gates[0]),
)

print("Expert assignments:", load.tolist())
print("Assignments checked:", int(load.sum()))
print("Output shape:", output.shape)

print(
    "Toy layer total parameters:",
    total_parameters,
)

print(
    "Toy layer active parameters per token:",
    active_per_token,
)

print(
    "Active fraction:",
    f"{active_per_token / total_parameters:.1%}",
)

Step 3: Run the example

python moe_demo.py

With the seeded random initialization, the output is:

First token -> expert IDs: [6, 4]
First token -> mixture weights: 0.530, 0.470
Expert assignments: [2, 1, 4, 4, 4, 2, 2, 5]
Assignments checked: 24
Output shape: (12, 16)
Toy layer total parameters: 8320
Toy layer active parameters per token: 2176
Active fraction: 26.2%

Step 4: Understand what happened

Our synthetic batch contains 12 token representations, each with 16 features.

Every token is assigned to exactly two experts, producing 24 expert assignments.

The first token selects experts 6 and 4. Their outputs are combined with approximately 53% and 47% mixture weights.

Notice the important property: although the MoE layer contains 8,320 parameters, only 2,176 participate in the computation for any one token, counting the router and the two selected experts.

Total versus active parameters in our toy layer:

Counts include the always-evaluated router. No attention, biases, shared experts, or residual-layer parameters are included in this toy example.

Our sparse layer computes only two expert networks for each token, even though all eight are stored in memory.

A dense computation that evaluated all eight expert MLPs for every token would require approximately four times as much expert matrix multiplication work.

That does not imply that our Python program runs four times faster. NumPy loop overhead, small matrix operations, routing work, and memory access can dominate at this scale.

The purpose of this example is to verify sparse execution, not to benchmark optimized inference kernels.

5. The challenge hiding inside routing: expert imbalance

Look again at the expert assignment counts.

Our seeded example assigns one expert five tokens and another only one.

Expert utilization in the toy router

Assignments for the seeded 12-token example. Each token selects two of eight experts.

Average assignments per expert: 3. Real training systems must manage this imbalance across far larger batches and distributed devices.Uneven expert utilization is one of the defining operational challenges of MoE.

Imagine a distributed model with 64 experts spread across eight GPUs.

Suppose one expert attracts a disproportionately large number of tokens. The GPU responsible for that expert could become a bottleneck while other GPUs finish their work earlier.

Even though the model is sparse, its effective speed is constrained by how well the computation is distributed.

What expert collapse means

A router can learn to repeatedly favor a small subset of experts.

This can cause several problems:

• Popular experts become overloaded.

• Underused experts receive fewer useful training updates.

• Hardware utilization becomes uneven.

• Distributed inference suffers from stragglers.

• The model may fail to exploit its full parameter capacity.

Expert collapse is not inevitable, but it is an important training failure mode.

How modern systems address imbalance

One common solution is to add an auxiliary load-balancing objective during training.

The training process then optimizes both the primary language-modeling objective and a term that encourages more balanced expert use.

Another strategy adjusts routing preferences based on recent expert loads.

Modern implementations may also support capacity limits, token dropping, or dropless routing.

Each option changes the engineering tradeoff:

• Auxiliary losses encourage balance but alter the optimization objective.

• Dynamic expert biases can steer routing away from overloaded experts.

• Capacity limits bound work per expert but may require dropping or otherwise handling overflowed assignments.

• Dropless routing preserves assignments while requiring the system to handle uneven token counts.

NVIDIA’s current Megatron Core MoE documentation describes top-k routing, balancing losses, expert parallelism, and dropless execution as supported building blocks.

The key lesson is that a good MoE router is not just a classifier that picks experts. It is also part of a resource-allocation system.

6. Does each expert learn a specific skill?

It is tempting to imagine a large language model with a Python expert, a mathematics expert, a French expert, and a history expert.

That is an intuitive analogy, but it is too literal.

Experts are usually not manually told which subject to handle. They learn from gradients produced by the training process.

The learned divisions may reflect distinctions such as:

• Different patterns in contextual representations.

• Structural or syntactic properties of tokens.

• Language-specific statistical regularities.

• Semantic or computational features that are difficult to interpret.

These are possibilities, not guaranteed one-to-one mappings.

An expert might respond to patterns appearing across several languages and tasks. Conversely, one task may require many experts across different layers.

And crucially, routing happens independently at each MoE layer.

A token might select experts 3 and 7 in an early layer, then 11 and 19 in a later one. It does not receive a permanent expert assignment that persists throughout the entire network.

This is one reason expert specialization is difficult to interpret from router statistics alone.

Observing that an expert frequently receives code-related tokens does not prove it has exclusively learned programming.

To investigate specialization, researchers can examine routing frequencies, controlled input categories, activation patterns, and the performance consequences of disabling individual experts.

7. Training an MoE model is different from running one

During inference, routing determines which experts execute for a particular token.

During training, routing also influences which parameters receive gradient updates.

For a standard hard top-k mechanism:

1. The router produces a score for every expert.

2. The system selects the highest-scoring experts.

3. Those selected experts compute outputs.

4. The training loss propagates gradients through the selected expert computations and their mixture weights.

The selection operation itself is discrete. The selected routing weights can participate in differentiation, but ordinary hard top-k does not provide a smooth gradient through every possible routing choice.

Training systems must therefore manage the router’s learning dynamics carefully.

Under ordinary sparse routing, an expert that receives no tokens in a training step receives no data-driven gradient from that step.

That makes sustained expert underutilization particularly problematic.

Why training all those parameters can still make sense

Sparse activation does not imply that most parameters are permanently unused.

Different tokens select different experts, and different batches activate different subsets of the model.

Over many optimization steps, the training process can update the entire collection of expert parameters.

This allows the model to accumulate much greater representational capacity than the parameters used for any single token.

However, the exact benefit depends on data quality, routing stability, expert utilization, model size, and the amount of training computation.

Increasing the number of experts is not a guarantee of better performance.

8. The hidden cost: sparse compute does not mean sparse memory

This is arguably the most important practical distinction in the entire architecture.

Suppose a model has one trillion total parameters but activates only 32 billion per token.

The computation for a token may involve only a fraction of the expert weights.

But the system still needs access to the complete model, because future tokens could select different experts.

At BF16 precision, each parameter typically occupies two bytes of raw weight storage.

A trillion parameters therefore require approximately:

1,000,000,000,000 × 2 bytes = 2 terabytes

That is around 2 TB of raw weight storage before accounting for additional inference memory.

Reducing the active parameter count does not automatically reduce that storage requirement.

Quantization can shrink the raw weights substantially, but it does not erase the distinction between total and active parameters.

What memory is needed during inference?

A serving system may need resources for:

• The full model weights, potentially distributed across devices.

• Key-value caches for attention, where applicable.

• Intermediate token activations.

• Router outputs and expert assignment metadata.

• Temporary communication and kernel buffers.

• Concurrent requests and scheduling overhead.

Offloading expert weights to CPU memory or other storage tiers is possible, but it introduces data-transfer costs and potentially substantial latency.

This explains why a sparse trillion-parameter model should not be confused with an ordinary dense 32-billion-parameter model.

Their active parameter counts may be similar, while their memory requirements and deployment complexity differ dramatically.

9. Expert parallelism: how MoE scales across GPUs

Large MoE models require more than efficient matrix multiplication. They also need efficient communication between devices.

A common approach is expert parallelism, in which different GPUs own different sets of experts.

Consider an eight-GPU system hosting 64 experts.

Each GPU might own eight experts. When a router selects an expert on another GPU, the token representation must reach the device holding that expert.

A simplified execution cycle is:

1. Compute routing decisions.

2. Group token representations by their destination experts.

3. Transfer representations to the appropriate GPUs.

4. Execute expert networks locally.

5. Transfer results back when needed.

6. Reconstruct the outputs in the original token order.

This often involves all-to-all collective communication.

Simplified expert-parallel execution:

tokens on each GPU --all-to-all dispatch--> GPUs that own the selected experts
  GPU 0: E0 E1     GPU 1: E2 E3     GPU 2: E4 E5     GPU 3: E6 E7
        <--all-to-all combine-- expert outputs return in original token order

Conceptual four-GPU illustration. Each GPU stores a subset of experts, and token representations travel to the devices that own their selected experts.

The memory and computation can be distributed, but the movement of token representations is not free.

Communication performance depends on the interconnect, placement of experts, batch size, routing distribution, and implementation details.

Small batches are especially challenging because each expert may receive only a few tokens, resulting in inefficient small matrix operations.

That is why high-performance MoE inference uses techniques such as grouped matrix multiplication, fused routing operations, and specialized token dispatch.

The vLLM expert-parallel deployment guide documents how expert parallelism is used to distribute experts across GPUs.

In May 2026, the vLLM project also described elastic expert parallelism, highlighting the importance of adjusting distributed MoE serving resources as workloads change.

Sparse arithmetic alone is not enough. Efficient serving requires aligning the routing algorithm with the hardware and communication topology.

10. Cherry on the cake: a trillion-parameter model that activates about 3%

Kimi K2.5

Released January 2026 by Moonshot AI: a trillion-parameter-class model with unusually sparse per-token execution.

Approximate published figures. Sources: Moonshot AI’s model card and NVIDIA’s February 2026 technical overview.

According to Moonshot AI’s published configuration, Kimi K2.5 uses 384 routed experts and selects eight per token, alongside a shared expert.

That means only 8 of the 384 routed experts are selected for an individual token at an MoE layer.

NVIDIA reports approximately 32.86 billion active parameters, or about 3.2% of the model’s roughly one-trillion-parameter total.

The exact reported active count can vary slightly with parameter accounting, but the architectural point is unchanged: the overwhelming majority of stored parameters are not evaluated for a single token.

And here’s the surprising nuance: this does not mean 96.8% of the model remains unused throughout a conversation.

Different tokens can activate different experts. Across a batch of many tokens, a large proportion of the model’s experts may perform useful work.

The sparsity is primarily per token, not necessarily across an entire request or serving batch.

A second 2026 example: Qwen3.5

Alibaba’s Qwen3.5-397B-A17B offers another instructive comparison.

Its published configuration lists:

• 397 billion total parameters.

• 17 billion activated parameters.

• 512 routed experts.

• 10 selected routed experts plus one shared expert.

Its active parameter fraction is approximately 4.3%.

The model also combines sparse MoE layers with a hybrid attention architecture, illustrating how conditional computation can coexist with other methods for reducing inference costs.

These specifications are documented in the official Qwen3.5 model card.

The two models demonstrate that active parameter count, total parameter count, and number of selected experts are distinct architectural choices.

11. Why not make every model a mixture of experts?

MoE is powerful, but it is not a free upgrade.

A dense model and an MoE model with the same number of active parameters can have very different deployment requirements and performance characteristics.

An MoE architecture is attractive when a system benefits from much greater parameter capacity without proportionally increasing per-token arithmetic.

But it may be less attractive when memory is constrained, batches are very small, networking is slow, or deployment simplicity is paramount.

The inference bottleneck changes with the workload

During large-batch processing, efficient expert kernels can process many routed tokens together.

During autoregressive decoding with few concurrent requests, the system may become more sensitive to memory bandwidth, dispatch overhead, and small expert batches.

Long-context workloads introduce additional attention and cache considerations, even when the FFN portion is sparse.

Therefore, two MoE models with similar active parameter counts may have very different real-world throughput.

Model efficiency must be measured at the system level, not inferred from parameter counts alone.

12. Five practical experiments to deepen your understanding

Our NumPy implementation is deliberately small, making it a useful starting point for experiments.

Try the following changes and observe what happens.

1. Increase the expert count. Change from eight experts to 16 while keeping top-k at two. Calculate how the total and active parameter counts diverge.

2. Change top-k. Compare selecting one, two, and four experts. Observe how many assignments are created and how much expert work is needed.

3. Introduce artificial router bias. Add a large positive offset to one expert’s scores before selecting experts. Watch how quickly utilization becomes skewed.

4. Increase the batch size. Generate hundreds or thousands of token representations instead of 12. Examine whether the distribution of assignments becomes smoother.

5. Add a shared expert. Implement an additional FFN that processes every token, then combine its output with the sparse expert mixture. Recalculate active parameter counts.

The important metrics to track are not just execution time or the number of parameters.

Measure routing utilization, assignment imbalance, total versus active parameters, and the amount of expert computation.

For a real benchmark, the next step would be to implement the layer in a differentiable framework such as PyTorch, train it on a neutral synthetic task, and compare dense and sparse variants with controlled model dimensions and hardware conditions.

13. The mental model to keep

The biggest conceptual mistake when learning MoE is treating parameter count as a direct measure of the work done for every token.

For dense models, total and active parameter counts are much more closely related.

For sparse MoE models, they are fundamentally different.

Remember these principles:

• Routing is conditional. Different tokens can select different experts, even within the same sequence.

• Experts are learned networks. Their specialization emerges from training rather than being assigned fixed human-readable jobs.

• Sparse computation is not sparse storage. Unselected expert weights still have to be available to the serving system.

• Routing quality affects efficiency. Poorly balanced experts can waste hardware resources and hurt training.

• Total capacity and active compute are different dimensions. Large parameter counts do not automatically imply equally large per-token computational costs.

The trillion-parameter models of 2026 demonstrate how far this separation can go. A model can hold vastly more learned weights than it uses for any particular token, while still requiring significant infrastructure to store and coordinate them.

That is the real significance of mixture-of-experts: not simply making networks larger, but changing how model capacity translates into computation.

Your next step

Run the NumPy router, change top-k from two to one, and then increase the number of experts from eight to 32.

Record the total parameters, active parameters, and assignments per expert.

Then ask the question that drives real MoE engineering: How much extra model capacity can you add before memory, routing imbalance, and communication overhead cancel out the benefits of sparse computation?

Once you can answer that question with measurements, you’re no longer just reading about mixture-of-experts. You’re reasoning like an AI model architect.