🤖 AI / ML

Transformer

transformer · transformer architecture · self-attention · multi-head attention · bert · gpt architecture

Definition

A transformer is a neural network architecture with a multi-head self-attention mechanism that models the dependencies between all tokens in a sequence in parallel.

Where the transformer came from

Before 2017 recurrent neural networks (RNN, LSTM) dominated: they processed text sequentially, word
after word. That created two limitations — slow training, because nothing could be parallelised, and
the tendency to forget context from the beginning of a long text.

The paper “Attention is All You Need” (Vaswani et al., 2017) proposed dropping recurrence
altogether: take the attention mechanism alone, apply it to every position at once, and add
positional encodings to preserve token order.

How self-attention works

For every token the model computes three vectors: Query (Q), Key (K) and Value (V). The
attention of token A to token B is the dot product of Q_A and K_B, normalised by softmax. The result
is a weighted sum of the V vectors of all tokens. Put another way: each token votes on which other
tokens matter for interpreting it.

Attention(Q, K, V) = softmax(QK^T / √d_k) · V

Multi-head attention runs this process in parallel across several projections and concatenates the
results, which lets the model capture different types of dependency at the same time.

Encoders and decoders in e-commerce

Type	Examples	E-commerce use
Encoder-only	BERT, RoBERTa	Semantic search, query-product matching
Decoder-only	GPT, Llama	AI Shopping Assistant, description generation
Encoder-decoder	T5, BART	Review summarisation, multilingual search

Search tasks usually call for encoders: they turn the shopper’s query and the product description
into vectors that can be compared by cosine similarity. Conversational scenarios call for decoders
or encoder-decoder models.

Typical implementation mistakes

Overestimating the model size. For recommendation tasks BERT-base (110M parameters) is often
enough — heavy models of 7B parameters and above do raise quality, but they lose on inference
latency.
Ignoring context length. Most transformers cap the input sequence length (512 or 2,048
tokens). Long product descriptions or user histories need truncation or a specialised
architecture.
Fine-tuning without domain data. A general BERT understands category-specific terminology
poorly. Fine-tuning on product descriptions and real search queries improves quality
substantially.

Frequently asked questions

Why is a transformer better than a recurrent network?

Recurrent networks process a sequence token by token — slowly, and they struggle to hold long-range dependencies. A transformer processes every token in parallel and uses attention to model the dependency between any two positions explicitly. That both accelerated training and improved quality on long texts.

What are the encoder and the decoder in a transformer?

The encoder turns an input sequence into a vector representation, an embedding, and is used for text-understanding tasks (BERT, search models). The decoder generates text token by token, attending to what it has already produced (the GPT family). Models such as T5 or BART use both blocks.

How are transformers used in recommender systems?

Directly, in session-based recommendations, where the shopper's click history is treated as a sequence of tokens and the transformer predicts the next product. Indirectly, through the embeddings produced by BERT-style models for semantic search and query-product matching.

What does multi-head attention mean?

The model applies several attention heads in parallel, each focusing on a different type of dependency. One head may learn syntactic relations, another semantic ones. The outputs of all heads are concatenated and projected.