Large Language Model Architecture and Training: Tokens, Attention and Pretraining

Tokenization is the first step in processing text through a large language model. A tokenizer splits input text into subword units called tokens. Modern tokenizers use byte-pair encoding (BPE) or SentencePiece algorithms, which learn a vocabulary of subword pieces from a training corpus. Common words become single tokens; rare words are split into multiple subword tokens.

The tokenizer is not part of the neural network—it is a deterministic preprocessing step. The same text always produces the same token sequence given the same tokenizer. The tokenizer's vocabulary is fixed at training time and cannot change without retraining the model's embedding layer.

Tokenization affects cost, context length and multilingual coverage. A model with a vocabulary optimised for English may produce more tokens for non-English text, increasing inference cost and reducing effective context length. This is an engineering consideration, not just a linguistic one.

After tokenization, each token ID is mapped to a learned embedding vector. The embedding is a high-dimensional real-valued vector (typically 768 to 12288 dimensions) that the model learns during training. Tokens that appear in similar contexts tend to have similar embedding vectors—a property that emerges from training, not from manual design.

Positional information is added because self-attention is permutation-invariant by default—without position information, the model cannot distinguish "cat sat on the" from "the on sat cat." Modern large language models use rotary position embeddings (RoPE) or similar methods that encode relative positions into the attention computation, allowing the model to generalise to longer sequences than seen during training.

The embedding layer and positional encoding together determine the model's context window—the maximum number of tokens the model can process in a single forward pass. Context length is a design decision with direct implications for memory, compute and cost.

Self-attention is the core mechanism of the Transformer. For each token in the input, self-attention computes a weighted combination of all other tokens' representations. The weights are determined by a similarity score between the current token (query) and each other token (key), scaled and normalised through softmax.

Causal self-attention applies a mask that prevents each token from attending to future tokens. This is essential for language modelling: when predicting the next token, the model must not see the answer. The causal mask sets attention scores for future positions to negative infinity before softmax, resulting in zero attention weight.

Multi-head attention runs multiple attention computations in parallel, each with its own learned query, key and value projections. This lets the model attend to different relationships simultaneously—one head might focus on syntactic dependencies, another on semantic similarity. The outputs of all heads are concatenated and projected back to the model dimension.

Pretraining is the process of training a large language model on a large corpus of text using the next-token prediction objective. The model receives a sequence of tokens and is asked to predict the next token at each position. The loss is the cross-entropy between the predicted probability distribution and the actual next token.

Pretraining requires significant compute. The model is trained on billions or trillions of tokens using stochastic gradient descent or AdamW optimiser, with a learning rate schedule (warmup followed by cosine decay). Training runs for multiple epochs over the corpus, typically on clusters of GPUs or TPUs with data and model parallelism.

The training data mixture matters. Most large language models are trained on a blend of web text, books, code and structured data. The composition of this mixture affects the model's capabilities—a model trained on more code will be better at code generation, while one trained on more books may produce more formal prose. Data quality, deduplication and filtering are as important as data quantity.