Skip to main content

The First Question After the Transformer: Are RNNs Really Finished?

· 6 min read
p4r4d0xb0x
p4r4d0xb0x
Rustacean, AI, OSS Enthusiast

The 2017 Transformer changed a basic assumption of sequence models. Machine translation first demonstrated the claim that a model that "looks all at once and selects important relationships" could scale better than a model that "reads in order," and the Transformer subsequently became the standard architecture of the LLM era. A standard architecture always carries costs, however. As context grows longer, attention computation and memory use increase, while system design for inference centers on the cache needed to keep referring to tokens that have already passed.

Looking at the ICLR discussion of Transformer alternatives, an interesting first question is not "Should we make the Transformer larger?" but "What remains if we redesign the RNN that preceded the Transformer?" In Were RNNs All We Needed?, Leo Feng and colleagues propose minLSTM and minGRU, minimal versions of LSTM and GRU, and argue that the older recurrent family can achieve both parallel training and competitive performance.

Why Transformer Alternatives Matter Again​

The Transformer's strengths are clear. Every token in a sentence can see every other token directly, and the training stage is easy to parallelize. GPUs are more efficient when handling large matrix operations together than when performing sequential work, so the Transformer gained more advantage as model and dataset sizes increased.

Dark glass diagram contrasting all-to-all Transformer attention with a compact recurrent state path

The problem changes for long sequences. Self-attention fundamentally computes relationships between pairs of tokens. When the number of tokens doubles, the number of relationships to compare grows by roughly four times. Real systems apply various optimizations, but demand for inexpensive handling of long context has revived interest in architectures beyond the Transformer.

Term explainer: sequence model

A model that processes ordered data while considering relationships between earlier and later elements.

Example: when reading a sentence, it considers not only one word but also whether earlier words change the meaning of later ones.

Term explainer: self-attention

A method that calculates how much each word in a sentence should refer to the other words.

Example: in "I ate an apple. It was sweet," understanding what "it" means requires paying more attention to the earlier word "apple."

Why RNNs Fell Behind and What They Aim to Recover​

An RNN reads a sequence one step at a time. The next state depends on the previous state. This structure is natural, but difficult to parallelize during training. A sentence must be processed from the first word in order, making it difficult for a GPU to handle many calculations at once. This is why RNNs were quickly pushed to the margins when the Transformer appeared.

The core of Were RNNs All We Needed? is not simply reviving the RNN unchanged. The paper removes complex components from LSTM and GRU to create minLSTM and minGRU, explaining that this simplification can have three effects.

  1. They have fewer parameters than conventional LSTMs and GRUs.
  2. Full parallelization is possible during training.
  3. They show performance that can compete with recent models, including Transformers, across multiple tasks.

This claim does not mean that "RNNs are unconditionally better than Transformers." A more accurate interpretation is that "some of the reasons the Transformer won may have arisen not from the limits of the RNN idea, but from the computational structure of RNN designs at the time."

Term explainer: RNN

A neural-network architecture that carries what it has read so far like a small memory while processing the next input.

Example: it resembles reading a long story line by line, writing important information in a mental notebook, and consulting those notes to understand the next line.

Term explainer: parallelization

Processing several calculations at the same time rather than one by one in sequence.

Example: one teacher grading the tests of 30 students in order takes a long time, while several teachers dividing the tests and grading simultaneously finish much faster.

What “Alternative” Precisely Means​

The phrase "Transformer alternative" is often exaggerated. In actual research, one architecture does not replace every other architecture; different structures are selected for different workloads. Recurrent architectures or state-space models may be more attractive under conditions such as long context, streaming input, low latency, and limited memory. Conversely, the Transformer still has major advantages in general-purpose language-model training and ecosystem compatibility.

The significance of minLSTM and minGRU lies in structural simplicity. Contrary to the intuition that more complex models perform better, removing unnecessary dependencies and computations from an older architecture can create new competitiveness. This matters both for research and practice. The value of an alternative first comes not from "replacing the king" but from "expanding the design space."

Term explainer: parameter

A number that a model adjusts during training.

Example: if you repeatedly vary the amount of water, heat, and cooking time when making ramen to find the best combination, those adjustable values resemble a model's parameters.

Limitations and How to Read the Claim​

This work requires careful reading. First, the paper's results do not mean "replace every LLM with an RNN." Conclusions vary with experimental scope, dataset size, implementation optimizations, and hardware characteristics. Second, the Transformer ecosystem consists of more than the model architecture. Tokenizers, training recipes, inference stacks, distributed training, and serving frameworks have all been heavily optimized around Transformers.

Even so, this direction matters. The first good question after the Transformer is not "What is the next fashionable architecture?" but "At what point does the current standard architecture become expensive?" Once that point is identified precisely, even an old idea such as the RNN becomes a research candidate again.

Sources​

// COMMENTS

Comments