Recurrent nets to attention: why sequence models dropped the loop

LEARN · NEURAL NETWORKS FROM THE GROUND UP

For years, the standard approach to understanding a sequence was straightforward: read one element, update an internal memory, move to the next element, and repeat.

Recurrent neural networks followed this pattern. Long short-term memory networks made the memory more sophisticated, allowing information to survive across longer sequences. These architectures became important tools for language modeling, machine translation, speech recognition, and time-series forecasting.

Then attention changed the underlying assumption.

Instead of forcing every piece of information through a chain of hidden states, attention allowed a model to retrieve information directly from other positions in a sequence.

That sounds like a small architectural adjustment. In practice, it changed how neural networks learn relationships, how their computations can be parallelized, and how sequence models scale.

The important question is not simply whether attention performs better. It is why removing the recurrent loop can make certain learning problems dramatically easier.

In this tutorial, we’ll explore that question from first principles, reproduce a long-range memory failure in PyTorch, and connect the result to the design of modern transformers.

By the end, you’ll understand:

Why recurrent hidden states struggle to preserve information over long distances.
How vanishing and exploding gradients arise during backpropagation through time.
Why gated recurrent architectures helped, but didn’t eliminate sequential computation.
How query-key-value attention creates shorter paths between distant tokens.
How to train a small recurrent model and an attention-based model on exactly the same synthetic task.
What the experiment does—and does not—tell us about transformer architectures.

1. The original idea: turn a sequence into a running memory

Consider a sequence of tokens:

START → A → B → C → D → QUERY

Suppose the model must answer a question about token A after processing everything else.

A conventional recurrent neural network maintains a hidden state. At each time step, it combines the current token representation with the previous hidden state.

A simplified recurrent update is:

hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + b)

Here:

xₜ is the input representation at step t.
hₜ is the updated hidden state.
hₜ₋₁ is the previous hidden state.
Wₓ and Wₕ are learned weight matrices.
b is a bias vector.
tanh introduces a nonlinear transformation.

The hidden state is effectively a compressed summary of everything the network has processed so far.

Recurrent information flow:

h0 -> [h1] -> [h2] -> [h3] -> ... -> [hT] -> prediction
       ^       ^       ^              ^
       x1      x2      x3             xT

Each hidden state depends on the previous one. Information from the first token must pass through every intermediate state to reach the final prediction.

The architecture has an appealing property: it can process sequences incrementally.

If you’re reading sensor measurements arriving one at a time, you don’t necessarily need the entire history in memory. You can update the current state and discard older activations during inference.

But this introduces a difficult learning problem.

The hidden-state bottleneck

Imagine the model must remember the first token across 100 subsequent tokens.

Information about the first token must survive 100 repeated transformations. At each step, the hidden state is updated using both existing memory and new information.

That means the network must learn two operations simultaneously:

Preserve information that will be needed later.
Update information that matters for the current token.

These objectives can conflict.

Suppose the model reads a product identifier followed by a long sequence of unrelated shopping events. At the end, it must identify the original product.

Some hidden-state dimensions must remember the product identifier while others process the intervening events.

A recurrent network can theoretically learn this behavior. The difficulty is discovering and maintaining it through gradient-based optimization.

The problem becomes more pronounced as the relevant information moves farther from the final prediction.

2. The deeper problem: gradients must travel backward through the loop

Neural networks learn by computing gradients of a loss with respect to their parameters.

For an ordinary feedforward network, gradients travel backward through layers. For a recurrent network, they must also travel backward through time.

This is called backpropagation through time, or BPTT.

Suppose the model makes its prediction at time T, but the relevant information appeared at time 1.

To determine how the early hidden state affected the final loss, the gradient passes through every intervening recurrent transition.

Conceptually:

∂Loss/∂h₁ = (∂Loss/∂hₜ) · (∂hₜ/∂hₜ₋₁) · … · (∂h₂/∂h₁)

The exact order of the matrix multiplications depends on the chosen gradient convention, but the important feature is the repeated multiplication.

Each recurrent step contributes a Jacobian matrix describing how a small change to one hidden state affects the next.

If these transformations repeatedly shrink gradients, the signal from distant tokens can become extremely small.

A numerical example

Imagine that each step retains approximately 90% of a particular gradient component.

After 10 steps:

0.9¹⁰ ≈ 0.349

After 40 steps:

0.9⁴⁰ ≈ 0.0148

After 80 steps:

0.9⁸⁰ ≈ 0.000218

Illustrative gradient decay across timeA hypothetical gradient component multiplied by 0.9 at each recurrent step; not a measured training curve.

The numerical example is intentionally simplified. Real recurrent gradients involve matrices, nonlinearities, multiple paths, and changing Jacobians. They don’t generally decay according to one fixed scalar.

Nevertheless, the calculation illustrates the central issue: repeatedly multiplying moderately contractive transformations can suppress distant learning signals exponentially.

If the transformations instead amplify particular directions, gradients can explode. Exploding gradients may produce unstable updates, very large parameter changes, or numerical overflow.

Gradient clipping can control the magnitude of those updates, but it doesn’t restore information that has already disappeared from a gradient.

That distinction is important.

The challenge isn’t only storing a useful value in a hidden state. It’s learning how to store that value when the training signal becomes weaker with temporal distance.

This observation motivated one of the most influential developments in sequence modeling, which we’ll revisit after our experiment.

3. A controlled experiment: remembering the first token

Let’s make the problem concrete.

We’ll train two small neural networks on the same synthetic sequence classification task:

Recurrent reader: A conventional tanh recurrent network that processes tokens sequentially and makes a prediction from its final hidden state.
Attention reader: A model that uses learned positional embeddings and a query to retrieve information from the entire input sequence.

Both models will solve the same problem, receive the same number of training updates, and be evaluated on fresh examples.

Define the task

Each training example is a sequence containing three possible token values:

0: a binary token.
1: another binary token.
2: a special end-of-sequence read marker.

The label is always the value of the first token.

For example:

Example: delayed binary recall

Position	First	2	3	4	5	6	7	Read
Token	1	0	1	1	0	0	1	2

Correct answer: the first token, 1.

All intermediate binary tokens are independent random distractors. Their values provide no useful information about the target.

For a sequence containing 80 tokens, the network must recover the first token after 78 intervening distractors and the final read marker.

Random guessing achieves approximately 50% accuracy because the targets are balanced in expectation.

This is a deliberately narrow memory test. It isolates the ability to access an early piece of information without requiring natural-language understanding, large datasets, or expensive hardware.

Why use a synthetic dataset?

Real-world sequence datasets often contain shortcuts.

For example, predicting a product category from browsing history might not actually require remembering the first product. Later browsing events may reveal the same category.

Our synthetic data avoids that ambiguity.

The distractor tokens are independently sampled, so a model cannot consistently infer the label from their contents. It must preserve or retrieve information from the first position.

We will test two sequence lengths:

Configuration	Tokens	Purpose
Short sequence	8	Check whether both architectures can learn the task
Long sequence	80	Stress long-range information access

Each length is trained as a separate experiment, rather than training at one length and testing generalization to another.

4. Set up the environment

The implementation uses PyTorch’s stable recurrent and multi-head attention modules. It does not require downloaded datasets, pretrained checkpoints, or GPU hardware.

Create a fresh Python environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install torch

The activation command above is for Linux and macOS shells. On Windows, use the corresponding activation command for your terminal.

For installation requirements and hardware-specific builds, consult the official PyTorch installation guide.

Create a file named sequence_memory.py.

Complete runnable implementation

import torch
from torch import nn
import torch.nn.functional as F

# Small CPU experiment; no external dataset required.
torch.set_num_threads(2)


def make_batch(size, length, generator):
    tokens = torch.randint(
        0,
        2,
        (size, length),
        generator=generator,
    )

    # Remember the first binary token.
    target = tokens[:, 0].clone()

    # Replace the final token with the READ marker.
    tokens[:, -1] = 2

    return tokens, target


class RecurrentReader(nn.Module):
    def __init__(self, width=32):
        super().__init__()

        self.token = nn.Embedding(3, width)

        self.rnn = nn.RNN(
            input_size=width,
            hidden_size=width,
            nonlinearity="tanh",
            batch_first=True,
        )

        self.classifier = nn.Linear(width, 2)

    def forward(self, tokens):
        embedded = self.token(tokens)

        _, final_state = self.rnn(embedded)

        # Shape: [batch_size, width]
        final_hidden = final_state[-1]

        return self.classifier(final_hidden)


class AttentionReader(nn.Module):
    def __init__(self, length, width=32):
        super().__init__()

        self.token = nn.Embedding(3, width)

        self.position = nn.Embedding(
            length,
            width,
        )

        # One learned query retrieves sequence information.
        self.read_query = nn.Parameter(
            torch.zeros(1, 1, width)
        )

        self.attend = nn.MultiheadAttention(
            embed_dim=width,
            num_heads=4,
            batch_first=True,
        )

        self.classifier = nn.Linear(width, 2)

    def forward(self, tokens, return_attention=False):
        offsets = torch.arange(
            tokens.size(1),
            device=tokens.device,
        )

        # Combine token content and positional identity.
        values = (
            self.token(tokens)
            + self.position(offsets).unsqueeze(0)
        )

        query = self.read_query.expand(
            tokens.size(0),
            -1,
            -1,
        )

        context, weights = self.attend(
            query,
            values,
            values,
            need_weights=return_attention,
        )

        scores = self.classifier(context[:, 0])

        if return_attention:
            return scores, weights

        return scores


def run(model_kind, length, steps=300):
    # Repeatable parameter initialization.
    torch.manual_seed(7)

    if model_kind == "RNN":
        model = RecurrentReader()
    else:
        model = AttentionReader(length)

    optimizer = torch.optim.AdamW(
        model.parameters(),
        lr=0.003,
    )

    # The same training examples are generated
    # for both architectures at a given length.
    train_rng = torch.Generator().manual_seed(2026)

    model.train()

    for _ in range(steps):
        x, y = make_batch(
            size=128,
            length=length,
            generator=train_rng,
        )

        predictions = model(x)
        loss = F.cross_entropy(predictions, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    # Evaluate on independently generated data.
    model.eval()

    test_rng = torch.Generator().manual_seed(99)

    with torch.inference_mode():
        x_test, y_test = make_batch(
            size=4096,
            length=length,
            generator=test_rng,
        )

        predicted = model(x_test).argmax(dim=1)

        accuracy = (
            (predicted == y_test)
            .float()
            .mean()
        )

        print(
            f"{model_kind:9s} "
            f"length={length:2d} "
            f"accuracy={accuracy.item():.3f}"
        )

        if model_kind == "Attention":
            _, weights = model(
                x_test[:64],
                return_attention=True,
            )

            first_weight = (
                weights[:, 0, 0]
                .mean()
                .item()
            )

            print(
                "  Mean attention on first token: "
                f"{first_weight:.3f}"
            )


if __name__ == "__main__":
    for length in (8, 80):
        for model_kind in ("RNN", "Attention"):
            run(model_kind, length)

Run the complete experiment:

python sequence_memory.py

The script uses the current documented interfaces for PyTorch’s recurrent layer and multi-head attention.

5. Results: where the long-range failure appears

Running the experiment on a CPU installation of PyTorch 2.10 produced the following output:

RNN       length= 8 accuracy=1.000
Attention length= 8 accuracy=1.000
  Mean attention on first token: 0.852
RNN       length=80 accuracy=0.502
Attention length=80 accuracy=1.000
  Mean attention on first token: 0.765

These are results from the specified seeded run, not universal performance guarantees. Different software builds, initialization choices, training budgets, and architectures can change the outcome.

Validation accuracy after 300 training stepsMeasured results from the synthetic delayed-recall experiment. Chance performance is approximately 50%.

Recurrent readerAttention readerThe short sequence is easy enough that both models learn the task perfectly in this run.

The long sequence creates a substantial difference.

The recurrent reader achieves approximately 50% accuracy. For this balanced binary problem, that’s consistent with chance-level performance.

Meanwhile, the attention reader continues to classify the examples correctly.

Why?

What happens inside the recurrent reader?

The relevant bit appears at position zero.

But the recurrent network makes its prediction from the hidden state after the final token.

The information path looks like this:

First token → hidden state 1 → hidden state 2 → … → hidden state 80 → prediction

Learning to store the first bit requires the optimizer to discover a representation that survives dozens of randomly varying inputs.

And, as we’ve already established, the gradient for that distant information must travel backward through the same chain of transitions.

For this initialization, architecture, and training budget, the network does not learn the necessary memory behavior at length 80.

That doesn’t mean recurrent networks are mathematically incapable of solving the task. A recurrent model with suitable weights can preserve a bit over many steps, and different training procedures may successfully find those weights.

The result demonstrates an optimization difficulty, not an impossibility theorem.

What happens inside the attention reader?

The attention model doesn’t need to push the first token through 79 successive recurrent state updates.

Instead, its learned query can assign a large weight directly to the representation at position zero.

The model learns something resembling the following procedure:

Recognize which position contains the relevant information.
Assign substantial attention weight to that position.
Retrieve its representation.
Classify the recovered bit.

The printed attention weight provides a useful clue.

For the 80-token experiment, the mean attention assigned to the first token was approximately 0.765.

That is large relative to the roughly 0.0125 weight associated with uniformly distributing attention across 80 positions.

The attention layer is doing more than blindly averaging all the tokens.

An important nuance: our code averages attention weights across four heads. A value of 0.765 does not imply that every head individually assigns that exact weight or that attention weights are a perfect explanation of model behavior.

Nevertheless, the concentration on the first position is consistent with the successful retrieval behavior observed in this experiment.

6. Understanding attention mathematically

The central attention operation uses three collections of vectors:

Queries (Q): Represent what information a model is looking for.
Keys (K): Represent features used to determine which information is relevant.
Values (V): Represent the information returned when a position receives attention.

For scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Here, dₖ is the dimensionality of a key vector.

Let’s unpack the operation.

Step 1: Calculate relevance scores

A query is compared against each key using a dot product.

If a query and key align strongly, their dot product tends to be larger.

The system learns projections that make useful relationships correspond to high compatibility scores.

In our experiment, the query is a learned vector representing the task of retrieving the relevant bit.

The keys are derived from the token representations combined with positional embeddings.

Step 2: Scale the scores

The scores are divided by the square root of the key dimension.

Without scaling, dot-product magnitudes can grow with vector dimensionality under common distributional assumptions.

Large score magnitudes can produce very peaked softmax outputs and poorly conditioned gradients.

Scaling helps control this effect.

Step 3: Normalize with softmax

Softmax converts the scores into nonnegative weights that sum to one over the available key positions.

For an illustrative four-token sequence, attention weights might look like this:

Position	Attention weight
0	0.88
1	0.05
2	0.04
3	0.03

These numbers are hypothetical, not measurements from our trained model.

The interpretation is straightforward: the query retrieves most of its information from position zero.

Step 4: Combine the value vectors

The output is a weighted sum of value vectors.

A position with a large attention weight contributes more strongly to the output representation.

The classifier can then use this representation to predict the target.

The important structural difference is that each query has direct access to all permitted key positions.

Attention-based information retrieval:

positions:   x1     x2     x3    ...    xT
weights:    0.85   0.01   0.02   ...   0.01
              \     |      |             /
               \    |      |            /
                learned query (read marker)

A learned query combines information from all source positions; here almost all of the weight lands on the first token.

The shortest information route between distant positions no longer requires crossing every intervening time step.

That makes it easier for training signals to influence representations at distant positions.

However, the architecture doesn’t guarantee that the correct position will be selected. The model still has to learn useful queries, keys, and values.

7. Why positional information matters

There is an easy-to-miss detail in our attention implementation.

We added a learned positional embedding to each token representation.

Why?

Because attention based only on token content does not automatically know which token came first.

Consider these two sequences:

1, 0, 1, 0, READ
0, 1, 1, 0, READ

Without positional information, a single learned query attending over the token representations sees the same multiset of token values in both examples.

It cannot reliably distinguish their order.

For our task, that would be disastrous because the target depends specifically on the first position.

The recurrent model gets ordering information naturally through the sequence in which it updates its state.

The attention model needs ordering information supplied through positional representations or another structural mechanism.

Our code uses learned absolute position embeddings.

This allows the model to associate a distinct representation with position zero and learn that it is important for the classification task.

A useful ablation experiment

Remove the positional embedding contribution from the attention reader, leaving only the token embedding.

Train the model again.

For this fixed-query architecture, the output becomes invariant to permutations of tokens that preserve the collection of token values.

It can no longer reliably distinguish the first bit from other identical-valued bits at different positions.

The resulting performance should deteriorate substantially.

This is a powerful lesson: attention solves information routing, while positional representations help specify which information should be routed.

A second limitation follows from using a learned lookup table for positions. Our model is configured for a particular sequence length and isn’t designed to extrapolate directly to unseen longer positions.

Production architectures must consider sequence length, positional representation, masking, and context generalization more carefully.

8. The historical breakthrough: memory cells that preserved gradients

Here’s the historical twist in the story.

The vanishing-gradient problem was recognized long before transformers became popular.

In 1997, Sepp Hochreiter and Jürgen Schmidhuber published Long Short-Term Memory in Neural Computation.

Their motivation was precisely the difficulty of learning dependencies across long time intervals.

The paper introduced specialized memory cells intended to sustain useful error signals over extended sequences.

One central mechanism was the constant error carousel.

Rather than repeatedly replacing memory through an ordinary nonlinear recurrent transformation, the architecture introduced a cell-state pathway designed to preserve information and support more stable gradient propagation.

This was a fundamentally different response to the problem we just observed.

Instead of removing the recurrent chain, the researchers redesigned the chain so information and gradients could survive along it.

The modern gated memory update

In a modern long short-term memory network, a simplified cell-state update can be written as:

cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜ

Where:

cₜ is the new cell state.
fₜ is the forget gate.
iₜ is the input gate.
gₜ is the candidate cell update.
⊙ denotes elementwise multiplication.

The forget gate controls how much existing information remains. The input gate controls how much new information enters the cell state.

When the forget gate remains close to one, existing memory can persist with much less attenuation along the cell-state pathway.

There is an important historical distinction: the original 1997 architecture introduced the constant-error mechanism, but the adaptive forget gate was a later extension described in Learning to Forget: Continual Prediction with LSTM, published by Gers, Schmidhuber, and Cummins in 2000.

That distinction matters because the original invention and the commonly used modern cell are not exactly identical.

Why gated recurrence was such a big improvement

Gates allow the model to learn behaviors such as:

Preserve an important early observation.
Incorporate new information when it becomes relevant.
Suppress irrelevant updates.
Clear outdated memory when appropriate.
Expose selected parts of its memory for predictions.

The cell-state pathway offers a more favorable route for gradients than ordinary repeated tanh transformations.

It does not eliminate every possible gradient difficulty. Forget gates below one can still attenuate memory, and gradients flowing through other parts of the architecture can behave differently.

But the overall design makes long-distance dependency learning substantially more practical.

This is why gated recurrent architectures remained important across numerous sequence-processing problems even after their original introduction.

What gated recurrence didn’t change

The network still has a recurrent dependency.

To compute the next hidden state, it needs the previous hidden state.

That means a recurrent layer cannot generally compute all time steps independently in parallel.

Even if the memory problem is improved, the sequential computation problem remains.

Attention-based transformer layers would address that second limitation through a different architectural approach.

9. From one attention query to a transformer

Our experiment uses a deliberately simple attention reader.

It has one learned query, which retrieves a summary from all input positions.

This is a form of learned-query attention pooling. More specifically, the query attends to a sequence of key-value representations.

It is not yet a full transformer.

A transformer uses attention as part of a deeper architecture with multiple learned projections, residual connections, normalization, feedforward transformations, and, often, repeated layers.

A defining ingredient is self-attention.

What changes with self-attention?

Instead of using one learned query for the entire sequence, self-attention creates queries from token representations themselves.

Each token can then retrieve information from other permitted tokens.

For example, suppose we have the sentence:

The package that arrived yesterday was damaged.

A token representation for “was” might use information from “package,” even though several other words intervene.

In a self-attention layer, the computation between those positions is direct.

The query, key, and value projections are learned so that the model can identify and combine context relevant to its task.

Multi-head attention

One attention head produces one set of relevance weights and one weighted representation.

Multi-head attention runs several attention operations over different learned projections.

Different heads can learn different ways of combining information.

For example, one head might focus on nearby context while another focuses on longer-range relationships.

However, such interpretations are possibilities, not guarantees. Head behavior can be distributed, redundant, or difficult to interpret.

The heads are concatenated and projected into an output representation.

Our experiment uses four heads, but all four share the same single query position.

A full transformer self-attention layer instead constructs query representations for all token positions.

Why this enabled parallel training

Suppose an input has 1,000 tokens.

A recurrent layer must propagate its hidden state through 1,000 ordered transitions. Although computation within each transition can be parallelized across channels and examples, the temporal dependency creates a sequential chain.

A standard transformer self-attention layer can compute query, key, and value projections across all available token positions using parallel matrix operations.

The attention scores between permitted token pairs can also be calculated in parallel.

That makes transformer training well suited to accelerator hardware.

The original 2017 paper Attention Is All You Need proposed a sequence-transduction architecture built around attention rather than recurrent or convolutional sequence processing.

It provided the foundational transformer design connecting attention-based information routing with highly parallelizable training.

10. The computational tradeoff: removing the loop isn’t free

It’s tempting to conclude that attention simply replaced recurrence because it’s better in every way.

That would be misleading.

The two architectures have different computational and memory characteristics.

Recurrent computation

For a conventional recurrent network, each time step updates a fixed-size hidden state.

At fixed hidden width, the amount of computation grows approximately linearly with sequence length.

But the updates have a sequential dependency.

During incremental inference, a conventional recurrent model can often maintain only a fixed-size state instead of storing the complete input history.

This is useful for streams that continue indefinitely.

Full self-attention

In dense self-attention, each of T query positions can compare itself with T key positions.

That produces T × T attention scores.

The attention computation therefore has a quadratic sequence-length component.

For illustration:

Sequence length	Potential query-key pairs
128	16,384
512	262,144
2,048	4,194,304
8,192	67,108,864

These counts refer to dense pairwise attention for one head, before accounting for masking, specialized sparsity patterns, or implementation optimizations.

They are not measurements of total model memory consumption.

Modern attention implementations can substantially reduce intermediate memory traffic and avoid materializing a full attention matrix, but dense pairwise computation still has an important scaling cost.

PyTorch’s scaled dot-product attention documentation describes backend selection and optimized implementations.

Why our tiny attention reader is cheaper

Our experiment is different from full self-attention.

It uses just one query for a sequence of length T.

Its attention score matrix therefore has shape 1 × T, rather than T × T.

The attention-weighting operation grows linearly with sequence length for a fixed query count and embedding width. The full module also performs learned projections, so its total work is more than just those dot products.

This architecture was chosen to illustrate direct information retrieval, not to replicate the full computational complexity of a transformer.

That distinction is essential when interpreting the benchmark.

Autoregressive generation is still sequential

There’s another nuance.

Transformers remove recurrence from the layer architecture, but autoregressive language generation still predicts tokens in order.

When generating the next token, the model ordinarily cannot condition on future tokens that have not yet been generated.

Causal attention masks enforce that constraint during training and inference.

During generation, key-value caching allows a transformer to reuse information from previous positions, reducing redundant computation.

But the cache occupies memory that grows with the amount of retained context, subject to the architecture and cache-management strategy.

So removing the recurrent loop does not mean every aspect of sequence generation becomes parallel.

11. Was this a fair comparison?

Our experiment demonstrates an instructive difference, but it is not a comprehensive benchmark of recurrent networks against transformers.

Several choices favor understanding over strict experimental equivalence.

First, the models do not have matched parameter counts. The attention reader contains positional embeddings and additional projection parameters. Both models use a width of 32, but that does not make their total capacities identical.

Second, both models receive the same number of optimizer updates, not a matched computational budget. Different architectures may require different amounts of computation per update.

Third, the task is particularly natural for attention. The correct answer is always stored at a known position, making direct positional retrieval an effective strategy.

Fourth, the recurrent baseline is intentionally simple. It uses a one-layer tanh network rather than a gated memory architecture, specialized initialization, or an engineered memory mechanism.

Finally, one seed and one training schedule are insufficient to establish general performance rankings. A serious study would evaluate multiple seeds, training durations, sequence lengths, parameter budgets, and task families.

Nevertheless, the result exposes a real architectural distinction:

A recurrent reader must learn to propagate information through many intermediate states.

An attention reader can learn to retrieve that information directly.

Both may represent the necessary function, but their optimization paths differ substantially.

12. Extend the experiment yourself

The most useful way to internalize this material is to modify the experiment rather than treating the reported accuracy as the final answer.

Here are four progressively more revealing exercises.

Exercise 1: Increase the dependency distance

Change the tested long sequence from 80 tokens to 160 or 320.

Keep the training budget fixed at 300 updates.

Observe whether either model’s validation accuracy changes.

Then increase the number of training updates and compare.

This separates the difficulty of learning a long-range dependency from the question of whether a model eventually has enough training to solve it.

You may find that larger positional tables, wider networks, or different learning rates influence the attention reader as well.

Exercise 2: Replace ordinary recurrence with gated memory

Implement a third reader using PyTorch’s nn.LSTM while retaining the same token embedding and final-state classifier.

For the default one-layer, unidirectional configuration, the final hidden state has the same high-level shape used by our recurrent classifier.

Unlike the ordinary recurrent layer, the long short-term memory layer also returns a final cell state.

Compare its behavior against both original readers.

Try different forget-gate initializations and examine whether long-distance learning improves.

The question is not whether the gated architecture wins every run. It is whether providing a more stable memory pathway changes the optimization behavior.

Exercise 3: Remove positional embeddings

Modify the attention reader to use token embeddings alone.

Retrain it on the original task.

Think about whether any permutation-invariant function of these token embeddings can always identify the first token.

This exercise helps demonstrate why order information is essential when attention is used for sequence modeling.

Then consider a redesigned task where a unique marker identifies the relevant token regardless of its position. That task could be solved through content-based attention without requiring the same absolute positional signal.

Exercise 4: Make the location unpredictable

Our task always stores the relevant bit at position zero.

A more interesting task would place a special key somewhere in the sequence and require the model to retrieve the bit following that key.

For example:

0, 1, KEY, 0, 1, 1, 0, READ

The correct answer would be 0 because it follows KEY.

Now the model must find a relationship between tokens rather than simply learn one absolute location.

For a full transformer experiment, implement a small self-attention encoder with token and positional embeddings, followed by a prediction head.

This extension creates a natural progression from learned-query attention pooling to genuine context-dependent self-attention.

Be careful to distinguish an architecture that reads the complete input before classifying it from a causal language model that must not access future positions.

13. When should you still consider recurrent models?

The transformer architecture is an extremely important approach to sequence processing, but a recurrent model can still be appropriate.

Consider recurrence when:

Input arrives as a continuous stream and decisions must be updated incrementally.
The system has strict memory constraints.
A compact hidden state is sufficient for the task.
Low-latency per-event processing matters more than parallel processing of long batches.
The application involves simple temporal relationships that don’t require expensive global token interactions.

Attention is particularly attractive when:

The model must combine information from distant parts of an available context.
Relationships between tokens are more important than maintaining only a compressed running summary.
Accelerator parallelism is available.
The task benefits from retrieving different context positions for different queries.
The architecture needs multiple layers of flexible information exchange.

Neither choice is determined by sequence length alone.

A long stream with simple local dynamics is a different modeling problem from a long document requiring references between distant passages.

The best architecture depends on the required information flow, computational environment, latency requirements, and training data.

14. The conceptual bridge to transformers

There is a useful way to understand the evolution of sequence architectures.

The early recurrent approach asked:

How do we compress everything observed so far into a useful state?

Gated recurrent memory asked:

How do we preserve important information and its learning signal across many updates?

Attention introduced a different question:

Why compress everything through one evolving memory if the relevant information can be retrieved directly?

These are not entirely competing ideas. Attention was also used alongside recurrent encoder-decoder architectures before transformers removed recurrence from the main sequence-processing blocks.

The transformer made direct, learned information exchange a central architectural operation.

Its power isn’t merely that it can look far backward in a sequence. It’s that each permitted query position can construct a context-dependent representation from many other positions, and this exchange can be repeated across layers.

That changes the effective computational graph.

A token no longer has to wait for information to pass through every intermediate recurrent time step before interacting with a distant token.

But the model must still learn which interactions matter, encode ordering appropriately, respect causal constraints when necessary, and manage the computational cost of broad access.

Those are the engineering problems that continue to shape attention-based architectures.

15. Final takeaways

The experiment gives us several durable lessons.

Long-range memory is an optimization problem as well as a representation problem. A model may be capable of storing information without reliably learning to do so.
Ordinary recurrence creates long gradient paths. Repeated transformations can weaken or amplify training signals.
Gated memory improves the recurrent pathway. The 1997 long short-term memory breakthrough addressed vanishing gradients by redesigning how information persisted.
Attention creates direct retrieval paths. A query can interact with an early token without transporting its representation through every intervening recurrent state.
Positional information and masking remain essential. Direct retrieval does not automatically imply an understanding of sequence order or permitted information access.
Transformers trade sequential recurrence for parallelizable attention operations, but introduce their own scaling costs.

The central lesson is that architecture determines more than what a neural network can theoretically calculate.

It determines how information travels, how gradients reach their targets, and how easily useful behavior can be learned.

Our tiny experiment illustrates that difference using a single bit of information separated from its prediction by dozens of distractions.

Modern transformers apply the same broad principle at a much larger scale: replace mandatory step-by-step information propagation with flexible, learned interactions between token representations.