-
Activation functions in 2026: why ReLU won and what’s replacing it
ReLU remains the cheapest dependable baseline, but modern transformer feed-forward blocks increasingly favor smooth, gated designs—especially GELU, SiLU, and SwiGLU—because they preserve a gradient pa