From RNNs to Transformers: RNN, LSTM, Seq2Seq

TL;DR: RNNs read a sequence one token at a time and fold history into a single hidden state, which is elegant but breaks on long inputs because gradients vanish over many steps and the computation cannot parallelize. LSTMs and GRUs add gates that let information skip past the squashing, and seq2seq with attention let the decoder look back at every encoder state instead of one fixed summary vector. That attention idea, plus dropping recurrence for full parallelism, is exactly what produced the transformer.

Recurrence and the hidden state

An RNN processes a sequence step by step. At position t it takes the current input x_t and the previous hidden state h_{t-1}, mixes them through a shared weight matrix, and emits a new hidden state h_t. That single vector h_t is the model's entire memory of everything seen so far. The same weights apply at every step, so the network handles variable-length input with a fixed parameter count, which was the appealing part.

The cost is that everything must run in order. You cannot compute h_100 before h_99, so training and inference are inherently sequential. On modern accelerators that is a hardware mismatch: GPUs want large parallel matrix multiplies, and a strict left-to-right loop starves them.

Why long dependencies break

The deeper problem is the gradient. Backpropagating an error at step 100 to a parameter that mattered at step 1 multiplies many Jacobians together. If the recurrent weight's effective scale is below 1, those products shrink toward zero (vanishing gradients) and the early signal never reaches the update. If above 1, they blow up (exploding gradients). Vanishing is the common case with the squashing nonlinearities RNNs used, so a plain RNN effectively forgets anything more than roughly 10 to 20 steps back. Subject-verb agreement across a long clause, or a pronoun referring to a name 60 tokens earlier, is out of reach.

Gates fix the carry, not the order

The LSTM adds a separate cell state that runs alongside the hidden state with mostly additive updates, plus three gates (forget, input, output) that decide what to erase, what to write, and what to expose. Because the cell state is updated by addition rather than repeated multiplication, gradients can flow across hundreds of steps without collapsing. The GRU is a lighter variant with two gates (reset, update) that performs comparably on many tasks with fewer parameters.

Gates fixed the vanishing-gradient problem. They did not fix the sequential bottleneck: an LSTM still runs one step at a time.

Seq2seq and the bridge to attention

Translation and summarization need to map one sequence to another of different length. The seq2seq design uses an encoder RNN to read the input into a final hidden state, then a decoder RNN to generate the output from that state. The flaw is obvious once you name it: the entire source sentence is crushed into one fixed-size vector, and long sentences lose detail at the front.

Attention (Bahdanau, 2014) removed that bottleneck: at each decoding step the decoder computes a weighted sum over all encoder states, learning where to look. Now position 1 of the source is directly reachable when generating any output token, no matter the distance. Once you have attention doing the heavy lifting of moving information between positions, the recurrence looks redundant. The transformer's move was to delete the RNN entirely, keep attention, and add positional encodings so order survives, which buys full parallelism over the sequence.

That final design is one block stacked N times: an attention sublayer that mixes information across tokens, a per-token feed-forward layer, each wrapped in a residual connection and a normalization step.

Why interviewers probe this

This is a screen for whether you understand why the transformer won, not just that it did. The strong-answer move is to name two distinct problems RNNs had, long-range gradient flow and sequential (non-parallel) computation, and show that gates solved the first while attention plus dropping recurrence solved the second. The held-back follow-up is usually "if attention scales quadratically with sequence length, why is that acceptable when RNNs were linear?" The answer: the quadratic cost is fully parallel matrix work that GPUs eat happily, whereas the RNN's linear cost is a serial dependency chain that hardware hates.

Common misconceptions

• "LSTMs solved long-range dependencies completely." They extend useful range to hundreds of steps, but information still degrades with distance and the model stays strictly sequential.
• "RNNs are slow because they are big." They are slow because each step depends on the previous one, so the work cannot be parallelized regardless of model size.
• "Attention was invented for transformers." Attention first appeared inside RNN seq2seq models years earlier; the transformer's contribution was removing the recurrence around it.
• "Vanishing and exploding gradients are the same bug." Same product-of-Jacobians cause, opposite symptom; exploding is patched with gradient clipping, vanishing needs gates or skip connections.

Key takeaways

• An RNN carries all history in one hidden state and runs strictly left to right, which kills parallelism.
• Plain RNNs forget past roughly 10 to 20 steps because gradients vanish across many multiplicative steps.
• LSTM and GRU gates use a mostly additive memory path so gradients survive over long ranges.
• Seq2seq with attention killed the single-vector bottleneck; deleting the recurrence and keeping attention is the transformer.