Long Short-Term Memory

Long Short-Term Memory, usually shortened to LSTM, is a type of recurrent neural network designed to learn from ordered data while preserving useful information across many time steps. It processes a sequence one step at a time and carries two forms of state forward: a cell state that acts as the longer-lived memory path and a hidden state that represents the current output and short-term working state.

Sepp Hochreiter and Jürgen Schmidhuber introduced LSTM in 1997 to address insufficient, decaying error flow in recurrent networks. The commonly taught modern LSTM includes a forget gate added by Felix Gers, Jürgen Schmidhuber, and Fred Cummins in later work. This historical distinction matters because “the LSTM” now usually means the updated input-gate, forget-gate, and output-gate design rather than the exact 1997 cell.

Why ordinary RNNs forget

A recurrent neural network reuses the same transformation at every position in a sequence. During training, backpropagation through time multiplies gradients through those repeated steps. If the relevant derivatives are mostly smaller than one, the gradient can shrink toward zero before it reaches an early event. If they are too large, it can explode.

The result is that a basic RNN may learn nearby relationships but struggle to connect events separated by many steps. For example, it may need an early subject to interpret a verb much later, an old sensor reading to explain a current anomaly, or an earlier musical motif to predict the next phrase.

LSTM creates a more direct, additive path through the cell state. Learned gates decide what to retain, write, and reveal. This improves gradient flow and gives the model a practical mechanism for selective memory.

How an LSTM cell works

At time step t, an LSTM receives the current input x_t, the previous hidden state h_(t-1), and the previous cell state c_(t-1). It computes several vectors with learned weights and biases:

1. The forget gate decides how much of the previous cell state to keep.
2. The input gate decides how much new candidate information to write.
3. A candidate update proposes new content derived from the input and previous hidden state.
4. The cell combines retained old state with gated candidate content to produce the new cell state.
5. The output gate decides which part of that cell state becomes the new hidden state.

A common formulation is:

f_t = sigmoid(W_f · [h_(t-1), x_t] + b_f)

i_t = sigmoid(W_i · [h_(t-1), x_t] + b_i)

g_t = tanh(W_g · [h_(t-1), x_t] + b_g)

c_t = f_t ⊙ c_(t-1) + i_t ⊙ g_t

o_t = sigmoid(W_o · [h_(t-1), x_t] + b_o)

h_t = o_t ⊙ tanh(c_t)

Here, ⊙ means element-by-element multiplication. Each gate is a vector, so the cell can retain one feature, erase another, and write a third at the same time. A sigmoid output near zero suppresses a component, while a value near one passes most of it through. Gates are soft controls, not binary switches.

The three LSTM gates

Gate	Main question	Effect
Forget gate	What old information should remain?	Multiplies the previous cell state before it is carried forward
Input gate	Which candidate information should be written now?	Controls the new contribution to the cell state
Output gate	Which stored information should affect the current output?	Controls the hidden state exposed to the next layer or time step

The cell state is not a database of readable facts. It is a learned continuous vector whose dimensions acquire whatever internal meanings help minimize the training loss. The model does not receive explicit rules saying which gate should remember a name or forget a noise spike.

Cell state vs hidden state

State	Role	Passed forward?
Cell state c_t	Longer-lived internal memory path updated additively through gates	Yes, to the next time step
Hidden state h_t	Current exposed representation and recurrent output	Yes, to the next step and usually to the next layer or prediction head

“Long short-term memory” does not mean unlimited permanent memory. The name describes a mechanism for keeping information longer than a basic short-term recurrent state. Capacity is finite, state can be overwritten, training often uses truncated sequences, and very long dependencies can still be lost.

How LSTMs are trained

LSTMs are usually trained with backpropagation through time. The recurrent computation is unrolled across a sequence, a loss is calculated from one or more outputs, and gradients flow backward through the unrolled steps and shared parameters. An optimizer then updates the weights.

Long sequences increase memory and compute because activations from many steps may be needed for the backward pass. Truncated backpropagation through time limits the number of steps used for each gradient update while carrying state between chunks. Gradient clipping is commonly used to control exploding gradients.

Training data can support several input-output patterns:

• Many-to-one, such as classifying an entire time series or sentence
• One-to-many, such as generating a sequence from one context vector
• Many-to-many aligned, such as labeling every audio frame or token
• Encoder-decoder, where one sequence is encoded before another sequence is generated

Padding, masking, sequence length, state resets, and whether state crosses batch boundaries are part of the model definition. A stateful LSTM can carry state between consecutive chunks, but it must not accidentally leak information between unrelated examples.

LSTM variants

Variant	What changes	Typical reason to use it
Stacked LSTM	Places multiple recurrent layers on top of one another	Learns higher-level temporal representations
Bidirectional LSTM	Processes the sequence forward and backward	Uses both past and future context when the full sequence is available
Peephole LSTM	Lets gates inspect the cell state directly	Gives timing-sensitive tasks additional state access
Convolutional LSTM	Replaces dense transformations with convolutions	Preserves spatial structure in video, weather, or image sequences
Projected LSTM	Projects the hidden output to a smaller dimension	Reduces recurrent computation and output size

A bidirectional LSTM is unsuitable for strictly causal streaming when future inputs do not yet exist. It can work well for offline tagging, transcription, or analysis where the complete sequence is available.

LSTM vs RNN vs GRU vs transformer

Architecture	Memory mechanism	Parallelism across sequence positions	Main tradeoff
Basic RNN	One recurrent hidden state	Low	Simple and small, but weak on long dependencies
LSTM	Cell state plus input, forget, and output gates	Low	Stronger selective memory with more parameters and computation
GRU	One gated state with update and reset gates	Low	Simpler than LSTM and often similarly effective, but not universally better
Transformer	Attention over token or patch representations	High during training	Scales well and connects distant positions directly, but attention cost and context storage can be large

Transformers displaced LSTMs as the default architecture for large language models because attention makes training much more parallel and gives each position a direct path to other positions. That does not make LSTMs obsolete. The better choice depends on data size, latency, memory, sequence length, hardware, and whether the application is streaming.

Where LSTMs are still useful

LSTMs remain practical for time-series forecasting, anomaly detection, speech and handwriting pipelines, biosignals, industrial sensors, financial sequences, embedded systems, and streaming classification. They can be attractive when data arrives one step at a time, the model must maintain a fixed-size state, training data is modest, or a compact recurrent model is easier to deploy than an attention model.

They are also useful as baselines. A complicated transformer that barely beats a tuned LSTM may not justify its extra memory, latency, or operational complexity.

Limitations

• Recurrence makes sequence positions difficult to process fully in parallel during training.
• Long unrolled sequences consume memory and increase training time.
• LSTMs mitigate vanishing gradients but do not guarantee retention over arbitrary distances.
• Cell capacity is finite and gates can learn to keep irrelevant information or forget important signals.
• Very long context, retrieval, or global pairwise relationships may be easier for attention-based models.
• Hidden state makes batching, state resets, masking, and deployment behavior more complicated.
• Results are sensitive to sequence construction, scaling, missing values, and temporal leakage.

For forecasting, random train-test splits can leak future information into training. Evaluation should preserve time order and compare against simple seasonal, persistence, and statistical baselines.

Bottom line

An LSTM is a recurrent neural network with a controlled memory path. Its gates learn how much old state to retain, how much new information to write, and what to expose at each step. It made long-range sequence learning far more practical, but its memory is selective and finite, and transformers are often a better fit for large-scale parallel sequence modeling.