TopolearnText, time series and event data are sequences; these are the building blocks before (and inside) transformers.
Sequence models start with an embedding: nn.Embedding(vocab_size, dim) maps each int64 token id to a learned vector, so a batch of shape (N, T) becomes (N, T, dim). Set padding_idx=0 so the padding token's vector stays zero and gets no gradient.
Batches need equal lengths, so shorter sequences are padded (nn.utils.rnn.pad_sequence does this in a collate_fn).
Recurrent layers nn.RNN, nn.GRU and nn.LSTM process the sequence step by step. Pass batch_first=True to use (N, T, features) like the rest of your code. An LSTM returns output (the hidden state at every step) and a tuple (h_n, c_n) holding the final states with shape (num_layers * directions, N, hidden).
With padding, the "final" state would include pad steps. Packing fixes that: pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False) lets the RNN skip pads, so h_n is the state after each sequence's last real token.
For new NLP work, transformers have largely replaced RNNs, but embeddings, padding and masks carry over directly, and GRUs and LSTMs remain useful for small streaming and time-series models.
Example
import torch
from torch import nn
emb = nn.Embedding(100, 8, padding_idx=0)
lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
tokens = torch.tensor([[5, 7, 9, 0],
[3, 4, 0, 0]]) # padded with 0
out, (h, c) = lstm(emb(tokens))
print(out.shape) # torch.Size([2, 4, 16]) every time step
print(h.shape) # torch.Size([1, 2, 16]) (layers, batch, hidden)
lengths = torch.tensor([3, 2])
packed = nn.utils.rnn.pack_padded_sequence(
emb(tokens), lengths, batch_first=True, enforce_sorted=False)
_, (h, _) = lstm(packed)
print(h[-1].shape) # torch.Size([2, 16]) state after the last real tokenRemember
- Embedding: (N, T) int64 ids -> (N, T, dim)
- Use
batch_first=True; h_n is (layers*dirs, N, hidden) - Pack padded batches so final states ignore padding
Watch out for
- Taking
output[:, -1]as the sequence summary when sequences are right-padded; it is often a pad step.