Activation functions in 2026: why ReLU won and what’s replacing it

The 2026 answer in one sentence

ReLU remains the cheapest dependable baseline, but modern transformer feed-forward blocks increasingly favor smooth, gated designs—especially GELU, SiLU, and SwiGLU—because they preserve a gradient path around zero and can learn which features to pass.

That does not make ReLU obsolete. Activation choice is now architectural: a compact CNN, a binary output head, and a large mixture-of-experts transformer should not automatically use the same nonlinearity.

What an activation function changes

Without nonlinear activations, stacked linear layers collapse into one linear transformation. Activations create nonlinear decision boundaries and shape both forward signals and backward gradients.

Activation	Definition	Practical behavior
Sigmoid	sigmoid(x) = 1 / (1 + e⁻ˣ)	Compresses values into 0–1; saturates at both extremes
ReLU	ReLU(x) = max(0, x)	Passes positives; hard-zeros negatives
GELU	GELU(x) = x · Φ(x)	Smoothly weights inputs instead of thresholding them
SiLU	SiLU(x) = x · sigmoid(x)	Smooth, self-gated scalar activation
SwiGLU	SiLU(gate(x)) ⊙ value(x)	Learned gate plus a separate value stream

Sigmoid still belongs in independent binary probability outputs. Inside hidden layers, however, its derivative is at most 0.25 and approaches zero for large positive or negative inputs, so deep stacks can learn slowly.

ReLU avoids positive-side saturation: its derivative is 1 for positive inputs. Its weakness is the negative half-line, where the derivative is zero.

GELU and SiLU smooth the transition around zero. PyTorch exposes both as stable modules, while BCEWithLogitsLoss combines sigmoid and binary cross-entropy more stably than applying them separately.

Why ReLU won

ReLU solved several practical problems at once:

• Useful gradients: positive activations keep a derivative of 1 instead of progressively flattening.

• Minimal compute: the core operation is essentially a maximum, not an exponential or cumulative distribution.

• Exact zeros: negative preactivations become zero, creating hard feature selection.

• Easy deployment: every major training and inference stack supports it efficiently.

Those zeros do not automatically accelerate dense matrix multiplication. Speedups require kernels or hardware that can exploit the resulting sparsity pattern.

ReLU therefore remains an excellent first baseline for small MLPs, conventional CNNs, mobile models, and latency-sensitive systems. It won because it was simple, fast, and difficult to misuse—not because it is theoretically perfect.

What is replacing ReLU—and where

GELU: the smooth drop-in

GELU usually replaces ReLU without changing layer shapes or parameter count. Slightly negative values can still contribute, while large positive values pass almost unchanged.

That smoothness can make optimization less brittle around zero. The trade-off is extra activation work, although real cost depends on compiler fusion, tensor shape, precision, and hardware.

SiLU and SwiGLU: gating instead of thresholding

SiLU is a scalar activation. SwiGLU is a feed-forward structure:

1. Project the input into a value stream.

2. Project it again into a gate stream.

3. Apply SiLU to the gate.

4. Multiply gate and value element-wise.

5. Project back to the model dimension.

The extra projection increases compute and expressiveness. Production models often reduce the intermediate width so the comparison is closer in parameter or FLOP budget.

What current public models actually use

Public code shows a strong 2026 trend, but not a universal rule.

Qwen3.5 text configurations declare hidden_act: "silu". The current Transformers implementation applies that activation to gate_proj(x), multiplies it by up_proj(x), and sends the result through down_proj—a SwiGLU-style MLP rather than a plain SiLU layer.

OpenAI’s public gpt-oss implementation also uses SwiGLU inside its expert blocks, but modifies the textbook formula: it clamps both branches, uses a scaled sigmoid with alpha 1.702, and adds 1 to the linear branch before multiplication.

The practical picture is therefore:

• ReLU remains the simple baseline.

• GELU is the smooth, parameter-neutral replacement.

• SwiGLU-style gating is common in large transformer MLPs.

• Real production implementations may alter the textbook activation for numerical or architectural reasons.

Closed models often disclose too little architecture to justify claims about every frontier system. Public configuration and source code are evidence, not a universal census.

Run the comparison

PyTorch 2.13.0 was released on July 8, 2026, and the current Matplotlib documentation is 3.11.1. Pin both so future installs do not silently change the experiment.

python -m pip install "torch==2.13.0" "matplotlib==3.11.1"

The reference run reported this complete environment:

Component	Reference value
Python	3.13.5
PyTorch	2.13.0
Matplotlib	3.11.1
CPU	AMD EPYC 9V74 80-Core Processor
Operating system	Linux 6.12.13, x86-64, glibc 2.41

Save this as activation_benchmark.py. It builds a noisy two-moons dataset, trains four approximately parameter-matched networks, saves loss curves, prints environment metadata, and then deliberately kills a ReLU layer.

import math
import platform
import sys

import matplotlib
import matplotlib.pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F

SEED = 7
EPOCHS = 300
EXPECTED_TORCH = "2.13.0"
EXPECTED_MATPLOTLIB = "3.11.1"

if torch.__version__.split("+", 1)[0] != EXPECTED_TORCH:
    raise RuntimeError(
        f"Expected PyTorch {EXPECTED_TORCH}, got {torch.__version__}"
    )

if matplotlib.__version__ != EXPECTED_MATPLOTLIB:
    raise RuntimeError(
        f"Expected Matplotlib {EXPECTED_MATPLOTLIB}, "
        f"got {matplotlib.__version__}"
    )

torch.manual_seed(SEED)
torch.set_num_threads(1)
torch.use_deterministic_algorithms(True)


def cpu_name() -> str:
    if platform.system() == "Linux":
        try:
            with open("/proc/cpuinfo", encoding="utf-8") as handle:
                for line in handle:
                    if line.startswith("model name"):
                        return line.split(":", 1)[1].strip()
        except OSError:
            pass

    return platform.processor() or platform.machine()


def print_environment() -> None:
    print("Environment")
    print(f"Python     : {sys.version.split()[0]}")
    print(f"PyTorch    : {torch.__version__}")
    print(f"Matplotlib : {matplotlib.__version__}")
    print(f"CPU        : {cpu_name()}")
    print(f"OS         : {platform.platform()}")


def make_moons(n_samples: int = 1200, noise: float = 0.12):
    if n_samples % 2:
        raise ValueError("n_samples must be even")

    half = n_samples // 2
    angle = torch.rand(half) * math.pi

    first = torch.stack(
        (torch.cos(angle), torch.sin(angle)),
        dim=1,
    )
    second = torch.stack(
        (1.0 - torch.cos(angle), 0.5 - torch.sin(angle)),
        dim=1,
    )

    features = torch.cat((first, second), dim=0)
    features += noise * torch.randn_like(features)
    targets = torch.cat((torch.zeros(half), torch.ones(half)))

    order = torch.randperm(n_samples)
    return features[order], targets[order]


features, targets = make_moons()
split = int(0.8 * len(features))

x_train, x_val = features[:split], features[split:]
y_train, y_val = targets[:split], targets[split:]

mean = x_train.mean(dim=0, keepdim=True)
std = x_train.std(dim=0, keepdim=True)

x_train = (x_train - mean) / std
x_val = (x_val - mean) / std


class PlainMLP(nn.Module):
    def __init__(self, activation: nn.Module, width: int = 32):
        super().__init__()
        self.hidden = nn.Linear(2, width)
        self.activation = activation
        self.output = nn.Linear(width, 1)

    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
        hidden = self.activation(self.hidden(inputs))
        return self.output(hidden).squeeze(1)


class SwiGLUMLP(nn.Module):
    def __init__(self, width: int = 18):
        super().__init__()
        self.value = nn.Linear(2, width)
        self.gate = nn.Linear(2, width)
        self.output = nn.Linear(width, 1)

    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
        gated = self.value(inputs) * F.silu(self.gate(inputs))
        return self.output(gated).squeeze(1)


def parameter_count(model: nn.Module) -> int:
    return sum(
        parameter.numel()
        for parameter in model.parameters()
    )


def train_model(
    model: nn.Module,
    learning_rate: float = 0.03,
):
    loss_fn = nn.BCEWithLogitsLoss()
    optimizer = torch.optim.AdamW(
        model.parameters(),
        lr=learning_rate,
        weight_decay=1e-4,
    )
    validation_curve = []

    for _ in range(EPOCHS):
        model.train()
        optimizer.zero_grad(set_to_none=True)

        loss = loss_fn(model(x_train), y_train)
        loss.backward()
        optimizer.step()

        model.eval()
        with torch.no_grad():
            validation_curve.append(
                loss_fn(model(x_val), y_val).item()
            )

    with torch.no_grad():
        predictions = (
            torch.sigmoid(model(x_val)) >= 0.5
        ).float()
        accuracy = (
            (predictions == y_val)
            .float()
            .mean()
            .item()
        )

    return validation_curve, accuracy


factories = {
    "Sigmoid": lambda: PlainMLP(nn.Sigmoid()),
    "ReLU": lambda: PlainMLP(nn.ReLU()),
    "GELU": lambda: PlainMLP(nn.GELU()),
    "SwiGLU": lambda: SwiGLUMLP(),
}

print_environment()
print("\nActivation comparison")

results = {}

for name, factory in factories.items():
    torch.manual_seed(100)

    model = factory()
    curve, accuracy = train_model(model)
    results[name] = (curve, accuracy)

    print(
        f"{name:7s} | params={parameter_count(model):3d} "
        f"| loss={curve[-1]:.4f} "
        f"| val_acc={accuracy:.1%}"
    )

plt.figure(figsize=(8, 5))

for name, (curve, _) in results.items():
    plt.semilogy(curve, label=name)

plt.xlabel("Epoch")
plt.ylabel("Validation BCE loss")
plt.title("Approximately parameter-matched activations")
plt.legend()
plt.tight_layout()
plt.savefig("activation_loss_curves.png", dpi=160)
plt.close()


def dying_relu_demo(
    activation: nn.Module,
    steps: int = 400,
):
    torch.manual_seed(123)

    model = PlainMLP(
        activation=activation,
        width=32,
    )

    with torch.no_grad():
        model.hidden.weight.normal_(
            mean=0.0,
            std=0.02,
        )
        model.hidden.bias.fill_(-3.0)

    initial_weight = (
        model.hidden.weight.detach().clone()
    )

    loss_fn = nn.BCEWithLogitsLoss()
    optimizer = torch.optim.Adam(
        model.parameters(),
        lr=0.1,
    )

    optimizer.zero_grad(set_to_none=True)
    loss = loss_fn(model(x_train), y_train)
    loss.backward()

    first_gradient = (
        model.hidden.weight.grad.norm().item()
    )
    optimizer.step()

    for _ in range(steps - 1):
        optimizer.zero_grad(set_to_none=True)

        loss = loss_fn(model(x_train), y_train)
        loss.backward()
        optimizer.step()

    with torch.no_grad():
        hidden_delta = (
            model.hidden.weight - initial_weight
        ).norm().item()

        predictions = (
            torch.sigmoid(model(x_val)) >= 0.5
        ).float()
        accuracy = (
            (predictions == y_val)
            .float()
            .mean()
            .item()
        )

    return first_gradient, hidden_delta, accuracy


print("\nDying-ReLU stress test")

stress_tests = {
    "ReLU": nn.ReLU(),
    "LeakyReLU": nn.LeakyReLU(
        negative_slope=0.1
    ),
}

for name, activation in stress_tests.items():
    gradient, delta, accuracy = dying_relu_demo(
        activation
    )

    print(
        f"{name:9s} "
        f"| first_grad={gradient:.6f} "
        f"| hidden_delta={delta:.6f} "
        f"| val_acc={accuracy:.1%}"
    )

Reading the results

The pinned reference environment produced the following fixed-seed CPU results:

Hidden block	Parameters	Final validation loss	Validation accuracy
Sigmoid	129	0.0239	99.2%
ReLU	129	0.0087	100.0%
GELU	129	0.0082	100.0%
SwiGLU	127	0.0053	100.0%

The experiment is approximately, not exactly, parameter-matched: the plain networks have 129 parameters and SwiGLU has 127. Runtime is not matched either, because SwiGLU performs two input projections.

The useful lesson is not that SwiGLU always wins. This tiny dataset shows that:

• Sigmoid can learn, but its loss falls more slowly.

• ReLU and GELU are close once optimization gets moving.

• Gating can add expressiveness at a similar parameter count.

• Final accuracy can hide convergence differences visible in the loss curves.

The script verifies the two pinned library versions and prints Python, PyTorch, Matplotlib, CPU, and operating-system details before training. Treat that metadata as part of the result. Deterministic algorithms and one CPU thread reduce variation, but they do not promise bit-for-bit equality across every processor, build, or operating system.

Cherry on the cake: kill every ReLU live

The stress test initializes every hidden bias to −3 and uses tiny weights. Every training sample therefore lands on the negative side of every ReLU.

Dying-ReLU stress test
ReLU      | first_grad=0.000000 | hidden_delta=0.000000 | val_acc=46.7%
LeakyReLU | first_grad=0.027166 | hidden_delta=20.605633 | val_acc=99.6%

The ReLU network cannot revive its hidden layer. Its output bias can move, but the hidden weights receive no gradient because ReLU’s derivative is zero throughout the observed input region.

The LeakyReLU control starts from the same hostile initialization. Its negative slope preserves a gradient path, so hidden weights move and the model recovers. Reporting weight change is more reproducible than counting how many units remain non-positive, which can vary with low-level numerical details.

Real training is usually less hostile. Sensible initialization, normalized inputs, residual connections, normalization layers, and conservative learning rates all reduce the risk. The demo exaggerates the conditions so the dying-ReLU failure is unmistakable.

A practical selection guide

Use ReLU for a strong baseline, compact MLPs or CNNs, minimal activation overhead, and straightforward deployment.

Use GELU when you want a smooth drop-in replacement without changing parameter count.

Use SwiGLU for transformer feed-forward blocks when learned gating is worth an extra projection and you can tune intermediate width for the actual compute budget.

Use sigmoid mainly to convert logits into independent binary probabilities. During training, prefer logits plus BCEWithLogitsLoss rather than manually applying sigmoid first.

Run the script, inspect activation_loss_curves.png, then replace one activation in a model you already train. Keep the parameter budget close, record convergence speed, validation quality, peak memory, and inference latency, and choose the activation that wins on your workload—not the one with the newest name.