AI Interview Questions

Next-token prediction, also called causal language modeling, is a pretraining objective where the model learns to predict the next token in a sequence given all previous tokens. The model is trained to maximize the probability of the actual next token while being prevented from attending to future tokens through causal masking. This autoregressive objective teaches the model to understand language patterns and generate coherent text.

During pretraining, the model processes sequences of tokens (typically from massive text corpora) with a causal attention mask that ensures each position can only attend to previous positions. For each position t, the model computes a probability distribution over the vocabulary for the next token P(token_t+1 | tokens_1..t). The training loss is cross-entropy between this predicted distribution and the actual token at position t+1. Gradients are computed and backpropagated to update model weights. This is repeated on trillions of tokens over thousands of GPU-days. The model never sees future tokens - it must predict them solely from context. This forces it to learn grammar, facts, reasoning patterns, and world knowledge to make accurate predictions.

Next-token prediction is deceptively powerful - it's a simple objective that scales remarkably well. By learning to predict the next word, models implicitly learn vast amounts about the world: facts (to predict 'capital' after 'France'), reasoning (to predict 'therefore' after logical premises), language structure (to predict correct verb forms), and even coding patterns. This objective enables zero-shot learning - models can perform tasks they weren't explicitly trained on by simply continuing prompts. It's also the foundation for in-context learning, where models learn from examples in the prompt. The causal nature ensures the model can generate coherent text autoregressively at inference, matching training conditions exactly.

Training on sentence 'The capital of France is Paris.' At position 'France', model predicts next token distribution: P('is')=0.8, P('was')=0.1, P('has')=0.05, etc. Loss computed against actual 'is', weights updated. At position 'is', predicts next: P('Paris')=0.7, P('Lyon')=0.15, etc. Over trillions of such predictions, model learns that 'capital of France' predicts 'Paris', 'capital of Italy' predicts 'Rome', and generalizes to answer questions about any country it learned about during training.

Masked language modeling is a pretraining objective where random tokens in the input sequence are masked (replaced with [MASK] token), and the model learns to predict the original masked tokens using bidirectional context from both left and right. Unlike causal LM which only uses past context, MLM leverages full context to build deep bidirectional representations.

In BERT-style pretraining, 15% of tokens in each sequence are selected for masking. Of these, 80% are replaced with [MASK], 10% replaced with random token, and 10% left unchanged (this prevents mismatch between pretraining and fine-tuning where [MASK] doesn't appear). The model processes the modified sequence with full bidirectional attention (no causal mask) and predicts the original tokens at masked positions using a softmax over vocabulary. The loss is cross-entropy only on masked positions. This is combined with next sentence prediction (NSP) in original BERT, though later models found NSP unnecessary. Training uses massive corpora (BooksCorpus, Wikipedia) over many epochs.

MLM enables deep bidirectional understanding that causal LM cannot achieve. By seeing both left and right context, BERT builds representations where each token's meaning is informed by entire surrounding context. This is particularly powerful for understanding tasks: classification, named entity recognition, extractive QA, and any task requiring full context comprehension. MLM-trained models consistently outperform causal models on understanding benchmarks (GLUE, SuperGLUE) because bidirectional context provides richer representations. The trade-off is that MLM models cannot generate text naturally - they're encoder-only, designed for understanding rather than generation. This division of labor (BERT for understanding, GPT for generation) defined the NLP landscape for years.

Sentence 'The [MASK] of France is Paris.' BERT sees full context: 'The', 'of', 'France', 'is', 'Paris'. With bidirectional attention, it knows 'France' and 'Paris' indicate capital relationship, so predicts 'capital' with high confidence. If causal LM saw 'The' then [MASK], it would have much less information. For understanding task like sentiment analysis, BERT's bidirectional view of entire review gives richer representation than causal processing. This is why BERT dominated leaderboards until GPT-3 showed surprising understanding capabilities despite causal limitation.

Autoregressive objectives (like next-token prediction) train models to predict future tokens given past tokens in a sequential manner, using causal masking to ensure only left context is visible. Autoencoding objectives (like masked language modeling) train models to reconstruct original input from corrupted versions, using bidirectional context to predict masked tokens. These represent fundamentally different approaches to learning language representations.

Autoregressive models (GPT family) factorize sequence probability left-to-right: P(sequence) = ∏ P(token_t | tokens_<t). They're trained on natural text order, predicting each token from previous ones. During inference, they generate by sampling from predicted distributions. Autoencoding models (BERT family) corrupt input by masking tokens and learn to denoise: they predict masked tokens using full context. The objective is to reconstruct original uncorrupted text. The model sees corrupted input and must recover original tokens, learning bidirectional representations. Some models (T5, XLNet) combine aspects of both with different corruption strategies.

These objectives produce models with different strengths. Autoregressive models excel at generation - they're naturally suited for producing coherent text, code, and completions because they're trained exactly for sequential prediction. They also enable in-context learning where tasks are framed as continuations. Autoencoding models excel at understanding - they build richer representations for classification, NER, and tasks requiring deep comprehension because bidirectional context provides complete information. The choice determines architecture: autoregressive → decoder-only, autoencoding → encoder-only. Modern large models (GPT-4, Claude) are primarily autoregressive but large enough that they develop strong understanding capabilities despite causal limitation, blurring the distinction.

For sentiment classification of 'This movie was absolutely fantastic!', autoregressive model processes left-to-right: 'This' → 'movie' → 'was' → 'absolutely' → 'fantastic'. At 'fantastic', it has seen all previous words, so can build representation. Autoencoding model sees all tokens simultaneously with bidirectional attention, so 'fantastic' can directly attend to 'movie' and 'absolutely' equally, potentially building richer representation for classification. For generating a review, autoregressive model naturally produces text sequentially; autoencoding model cannot generate at all without modification. Different tools for different tasks.

Unsupervised pretraining on massive text corpora learns general-purpose language representations that capture syntax, semantics, world knowledge, and reasoning patterns without requiring labeled data. These pre-trained models can then be fine-tuned on downstream tasks with minimal labeled examples, transferring the knowledge acquired during pretraining. This paradigm revolutionized NLP by dramatically reducing dependence on task-specific labeled data.

During pretraining, models process trillions of tokens from diverse sources: books, websites, academic papers, code repositories. The pretraining objective (next-token prediction or masked LM) forces the model to learn patterns at multiple scales: word co-occurrence, syntactic structures, factual knowledge (to predict 'Paris' after 'capital of France'), reasoning chains (to predict 'therefore'), and even basic arithmetic (to predict '2' after '1+1='). These patterns are encoded in model weights through gradient updates. When fine-tuned on a downstream task (e.g., sentiment classification), the model already understands language; it only needs to learn the specific task mapping, requiring far fewer examples than training from scratch.

This approach solved the data scarcity problem that previously limited NLP. Before pretraining, building a decent sentiment classifier required 10k+ labeled examples. With BERT pretraining, 100 labeled examples often suffice because the model already understands 'great' is positive and 'terrible' negative. This democratized NLP - organizations can build high-quality systems with modest annotation budgets. Pretraining also enables transfer across domains and languages. The scale matters: larger models trained on more data capture more knowledge, which is why GPT-3 (175B params, 500B tokens) outperforms smaller models on few-shot learning. Scaling laws show continued improvement with more compute and data.

Medical NER system identifying diseases in clinical notes. Without pretraining: need 50k annotated notes (cost $500k, months of doctor time). With BioBERT pretrained on PubMed: need 500 annotated notes (cost $5k, weeks). The pretrained model already understands medical terminology, sentence structure, and entity boundaries from reading millions of medical papers. It only needs to learn the specific annotation scheme for your task. This 100× reduction in required data is why pretraining is the foundation of modern NLP.

LLM pretraining uses massive, diverse text corpora collected from public sources, typically totaling trillions of tokens. These datasets combine web crawls, books, academic papers, code repositories, and social media to provide broad coverage of human knowledge and language patterns. The composition and quality dramatically affect model capabilities.

Common sources include: Common Crawl (petabytes of web pages, filtered for quality), Books (copyright-expired books, self-published books), Wikipedia (encyclopedic knowledge), academic papers (arXiv, PubMed), code repositories (GitHub), news articles, and social media discussions. Raw data undergoes extensive cleaning: deduplication (removing near-duplicate documents), quality filtering (using classifiers to select high-quality text), toxicity filtering, PII removal, and language identification. The processed data is then tokenized and formatted into sequences for training. Modern models like LLaMA use mixtures: 67% web pages, 15% books, 4.5% GitHub, 4.5% Wikipedia, 4.5% academic, 4.5% other.

Dataset composition determines model capabilities and biases. Web data provides broad coverage of general knowledge and colloquial language. Books add narrative structure and deep exposition. Code enables reasoning and structured thinking. Wikipedia adds factual accuracy. Imbalances cause problems: overrepresentation of certain viewpoints, cultural biases, or knowledge gaps. Data quality matters as much as quantity - duplicated or low-quality text wastes training compute and can harm performance. Recent research shows careful data curation (deduplication, quality filtering) can match performance of models trained on 10× more uncleaned data. The emergence of model collapse also highlights risks of training on AI-generated content.

GPT-3's training mix: 60% filtered Common Crawl (410B tokens), 22% WebText2 (19B), 16% Books (67B), 3% Wikipedia (3B). This combination gives broad web knowledge, quality writing from books, factual information from Wikipedia. LLaMA used 1.4T tokens from 7 sources with careful deduplication, enabling strong performance despite smaller size than GPT-3. The RefinedWeb dataset showed that careful filtering of Common Crawl alone (5T tokens) can match performance of curated mixtures, reducing data collection complexity.

Data quality and quantity represent complementary factors in pretraining effectiveness. Quantity provides breadth and statistical power, while quality ensures the model learns accurate, coherent patterns rather than noise. Recent research shows optimal pretraining requires both, with quality often being the limiting factor in practical scenarios.

Data quantity scaling laws show model performance improves as a power law with dataset size, but only if data is sufficiently clean. Low-quality data introduces problems: duplicates waste capacity (model memorizes rather than generalizes), factual errors teach misinformation, boilerplate text teaches repetitive patterns, toxic content harms alignment. Quality interventions include: deduplication (removing near-duplicates at document, paragraph, and sentence level), filtering (using classifiers to select well-formed text), toxicity removal, and domain balancing. The Compute-Optimal scaling (Chinchilla) suggests optimal ratio of model size to training tokens, but this assumes high-quality data throughout.

Training on 10T tokens of web garbage is worse than training on 1T tokens of curated books. The LLaMA paper demonstrated that careful data curation (1.4T tokens) produced models competitive with GPT-3 (300B tokens of less curated data). The SlimPajama project showed 30% of typical web corpus is low-quality or duplicate - removing it speeds training and improves performance. For practitioners, this means data engineering (cleaning, deduplication, filtering) is as important as model architecture. Training on poorly curated data wastes compute (expensive) and produces models with harmful behaviors. The rise of synthetic data adds new quality dimensions - AI-generated text can cause model collapse if overused.

Comparing two 7B models trained on different data: Model A trained on 2T tokens of raw web crawl (minimal filtering). Model B trained on 1T tokens of carefully curated, deduplicated, filtered text from diverse sources. Model B consistently outperforms Model A on benchmarks despite half the tokens, because its training data has higher signal-to-noise ratio. Deduplication alone often improves performance by 5-10% by preventing memorization of repeated boilerplate. Quality filtering removes low-value content like SEO spam, allowing model to focus on meaningful patterns. This is why modern training pipelines invest heavily in data processing.

Curriculum learning is a training strategy where examples are presented to the model in a meaningful order, typically from easier to harder, rather than randomly shuffled. The intuition is that models learn better when they first master simpler patterns before tackling complex ones, similar to how humans learn. In LLM pretraining, this involves ordering training data by complexity metrics.

Implementing curriculum learning requires defining difficulty metrics for text: length (shorter sentences first), perplexity under a small model, vocabulary rarity, syntactic complexity, or domain specificity. During training, the model starts with easy examples (short, common words, simple syntax) and progressively introduces harder examples (long documents, rare words, complex reasoning). The curriculum schedule can be predefined (linear increase in difficulty) or adaptive (based on model performance). Some approaches use multi-stage training: train on general web data first, then add books, then code, then specialized domains.

Curriculum learning can improve training efficiency and final performance. Starting with simpler patterns helps model establish basic linguistic representations before tackling noise and complexity. This can reduce training time to reach target performance by 10-30% in some studies. It's particularly effective for multi-modal training and reinforcement learning. However, benefits for large-scale LLM pretraining are debated - many successful models (GPT-3, LLaMA) use random shuffling and still achieve state-of-art results. The massive scale and diversity of pretraining data may naturally provide a curriculum effect if sampled appropriately. Recent research suggests curriculum may matter more for smaller models or limited data regimes.

Training a model with curriculum: Phase 1 (first 10%): Short, clean sentences from Wikipedia and children's books. Model learns basic syntax, common words, simple facts. Phase 2 (next 30%): News articles, blog posts with moderate length and complexity. Model learns narrative structure, varied vocabulary. Phase 3 (final 60%): Full web corpus, books, academic papers, code. Model learns complex reasoning, specialized knowledge, long-range dependencies. Compare to random training: same total tokens but no ordering. Curriculum model might reach target perplexity 20% faster, though final performance may converge. Some argue random shuffling already approximates curriculum because simple patterns are more frequent and learned first naturally.

Chinchilla scaling laws, introduced by DeepMind in 2022, describe the optimal allocation of compute budget between model size (number of parameters) and training data size (number of tokens) for training transformer language models. They found that most large models were undertrained - too many parameters relative to training data - and proposed a different scaling relationship than previously believed.

The researchers trained over 400 models of varying sizes (70M to 16B parameters) on different amounts of data (5B to 500B tokens) and measured final loss. They derived power-law relationships showing that for compute-optimal training, model size and training tokens should scale roughly equally: N_opt ∝ C^0.5, D_opt ∝ C^0.5, where C is compute budget. This means when doubling compute budget, both model size and training tokens should increase by about 40% each. Previous scaling laws (Kaplan et al.) suggested model size should grow faster than data, leading to undertrained models like GPT-3 (175B params, 300B tokens) which, according to Chinchilla, should have been trained on 4.2T tokens for optimal performance.

Chinchilla laws fundamentally changed how practitioners allocate pretraining budgets. Following these recommendations, models can achieve same performance with less total compute by balancing parameters and data appropriately. For example, DeepMind's Chinchilla (70B params, 1.4T tokens) outperformed GPT-3 (175B params, 300B tokens) despite using less total compute, because it was trained to optimal ratio. This has major implications: many organizations were wasting compute on oversized, undertrained models. The laws guide decisions: given fixed compute budget, optimal performance comes from training a moderately sized model on proportionally more data. This sparked trend toward smaller models trained on more data (LLaMA 65B on 1.4T tokens, following Chinchilla recommendations).

Compute budget C = 1e23 FLOPs. Old scaling (Kaplan): optimal N ≈ 175B params, D ≈ 300B tokens. Chinchilla scaling: optimal N ≈ 70B params, D ≈ 1.4T tokens. The Chinchilla-optimal model trains 4.7× more tokens on 40% fewer parameters. In practice, Chinchilla (70B) outperforms GPT-3 (175B) on most benchmarks while using less training compute. For a practitioner with $10M training budget, Chinchilla says: don't build 175B model trained on 300B tokens; build 70B model trained on 1.4T tokens - better performance, same cost. This insight saves millions.

Pretraining is the initial training of a language model from random initialization on massive, diverse corpora to learn general language understanding. Continued pretraining (also called domain-adaptive pretraining) takes an already pretrained model and trains it further on additional data, often domain-specific, to adapt its knowledge without losing general capabilities.

Initial pretraining starts with randomly initialized weights and trains on hundreds of billions to trillions of tokens from diverse sources (web, books, code). This establishes fundamental language representations, world knowledge, and reasoning abilities. Continued pretraining takes the checkpoint from initial pretraining and continues training with the same objective (next-token prediction) but on a different corpus - typically domain-specific (medical papers, legal documents, customer support conversations). The learning rate is usually lower than initial pretraining to prevent catastrophic forgetting of general knowledge. Training continues for 10-100B additional tokens depending on domain size and desired adaptation.

Continued pretraining enables domain specialization without losing general capabilities. A general model like LLaMA knows medicine from Wikipedia but lacks deep medical knowledge from journals. Training from scratch on medical data alone would lose general knowledge and be impractical. Continued pretraining on PubMed and medical textbooks injects specialized knowledge while preserving general abilities. This is more effective than fine-tuning for knowledge acquisition because fine-tuning on QA pairs teaches task format but not deep domain knowledge. Continued pretraining is essential for high-performance domain-specific models (BioBERT, ClinicalBERT, LegalBERT) and for adapting models to new languages or data distributions.

Starting with LLaMA-2 (7B) pretrained on 2T general tokens. For medical application, continue pretraining on 50B tokens from PubMed, medical textbooks, clinical notes. After continued pretraining, model answers medical questions more accurately, understands medical terminology, and reasons about clinical scenarios better than base LLaMA. Crucially, it still answers general questions correctly because learning rate was low enough to avoid catastrophic forgetting. Compare to fine-tuning on 100k medical QA pairs: that teaches Q&A format but doesn't embed deep medical knowledge; continued pretraining injects knowledge at the language modeling level, improving all medical tasks simultaneously.

Training loss measures how well the model predicts the next token (or masked tokens) on the training data. Lower loss indicates better predictive accuracy, which generally correlates with model quality on downstream tasks. However, the relationship is nuanced - loss on held-out validation data is a better predictor of model capabilities than training loss, and different tasks may have different sensitivities to loss improvements.

During pretraining, cross-entropy loss is computed as the negative log probability of correct tokens. This loss decreases as training progresses, following a power law with compute. Validation loss (on unseen data) typically correlates strongly with downstream performance - models with lower validation loss generally perform better on benchmarks like MMLU, though the correlation weakens at very low loss. Loss can be decomposed: token prediction difficulty varies (rare words have higher loss), and different corpora have different baseline losses. Perplexity (exp(loss)) is often reported as a more interpretable metric - the average number of equally likely choices the model sees at each prediction step.

Training curves guide training decisions: when loss plateaus, it may indicate diminishing returns. Validation loss helps compare model architectures and data quality independent of downstream evaluation. Scaling laws use loss as the primary metric to predict performance of larger models. However, loss isn't everything - two models with identical loss may have different strengths (one better at reasoning, another at factual recall). Also, over-optimizing loss can lead to overfitting, where model memorizes training data rather than generalizing. In practice, practitioners monitor both loss and downstream benchmarks, using loss as a diagnostic tool and early indicator of training issues.

Training two 7B models: Model A achieves validation perplexity 8.5, Model B achieves 8.2. On average, Model B will score slightly higher on MMLU (maybe 65% vs 64%). But within this, Model B might be 2% better on science questions (where lower loss indicates better factual knowledge) but similar on reasoning tasks. If loss improvement comes from better modeling of common patterns rather than rare knowledge, downstream gain may vary. At extreme scale (GPT-4), loss is so low that further improvements yield diminishing returns on benchmarks - the model already knows most common patterns. This is why frontier models now focus on specific capabilities rather than just loss minimization.

Warmup is a training technique where the learning rate is gradually increased from a very small value to the target maximum learning rate over the first few thousand steps of training. This prevents instability in early training when model weights are randomly initialized and gradients can be extremely large or chaotic.

Standard warmup schedules start with learning rate near zero (e.g., 1e-7) and linearly increase to target rate (e.g., 3e-4) over N steps (typically 500-2000 for small models, 5000-10000 for large models). After warmup, learning rate may follow a decay schedule (cosine, linear) for remaining training. The intuition: early in training, random initialization causes gradients to be noisy and potentially very large. High learning rate with noisy gradients can cause divergence (loss explodes to infinity). Warmup allows model to find reasonable parameter region before applying full updates. Adaptive optimizers like Adam still benefit from warmup because their moment estimates are unreliable initially.

Warmup is essential for stable training of large models, especially transformers. Without warmup, many models diverge within first few hundred steps, wasting compute and requiring restarts. Even if training doesn't diverge, warmup often leads to better final performance by preventing early chaotic updates that send parameters into poor regions of loss landscape. The optimal warmup duration scales with model size - larger models need longer warmup because gradients are noisier and optimization landscape more complex. For GPT-3 scale (175B), warmup lasted 375 million tokens (about 3000 steps) - without this, training would be unstable.

Training a 13B model with target LR 3e-4. Without warmup (LR=3e-4 from step 0): loss starts ~11, drops to 9 after 100 steps, then suddenly spikes to 50 and continues increasing - training diverged, must restart. With 2000-step warmup: LR starts 1e-7, gradually increases. Loss decreases smoothly from 11 to 6 over warmup period without spikes. After warmup at step 2000, LR reaches 3e-4, loss continues decreasing stably. The model reaches target perplexity 5000 steps sooner than if warmup were shorter or absent. This stability is crucial for multi-million dollar training runs where divergence costs days and hundreds of thousands of dollars.

Gradient clipping is a technique that limits the magnitude of gradients during backpropagation by scaling them down if their norm exceeds a threshold. This prevents extremely large gradient updates that could destabilize training, causing loss spikes or divergence, especially in large language models with billions of parameters.

After computing gradients for all parameters (via backpropagation), the global gradient norm ||g|| is calculated (L2 norm of concatenated gradients). If ||g|| exceeds a predefined threshold C (typically 1.0), all gradients are scaled down by factor C/||g||. This preserves gradient direction while reducing magnitude. The scaled gradients are then used for parameter updates via optimizer (Adam, etc.). Clipping can be applied per-parameter (layer-wise) or globally (all parameters together), with global being more common. Some implementations use value clipping (capping each gradient element to [-C, C]) instead of norm clipping, but norm clipping preserves direction better.

In large-scale training, gradients can occasionally become extremely large due to numerical instability, rare tokens with huge errors, or chaotic loss landscape regions. Without clipping, these large updates can throw parameters into poor regions, causing loss to spike and potentially never recover. This is particularly important in FP16/FP16 mixed precision training where gradients exceeding representable range cause overflow to infinity. Clipping stabilizes training, allowing consistent progress. It also enables higher learning rates by providing safety margin against outliers. For models with billions of parameters, training runs lasting weeks would almost certainly diverge without clipping. The threshold is a hyperparameter: too low slows learning, too high doesn't prevent spikes.

Training 70B model, normal gradient norm ~0.5. At step 5000, due to a batch with rare tokens and unusual patterns, gradient norm spikes to 50 (100× normal). Without clipping: optimizer applies 100× larger update than usual, throwing parameters far from optimum. Loss jumps from 3.5 to 15 and may never recover. With clipping threshold 1.0: gradients scaled by 1/50 = 0.02, effective update magnitude remains ~0.5 (normal range). Parameters move slightly but not catastrophically. Next batch's gradients likely normal, training continues smoothly. Over weeks of training, clipping prevents hundreds of such spikes from derailing the run, enabling successful convergence.

Mixed precision training uses lower-precision floating point formats (FP16 or BF16) for most computations while maintaining a master copy of weights in higher precision (FP32) for accurate updates. This reduces memory usage and speeds up computation on modern GPUs with specialized tensor cores, enabling training of larger models within same hardware constraints.

In mixed precision training, forward and backward passes are computed in FP16 or BF16 (16-bit) which uses half the memory of FP32 and runs 2-8× faster on tensor cores. However, some operations (loss scaling, gradient accumulation) need FP32 precision to avoid underflow/overflow. The master weights are stored in FP32 and updated using FP32 gradients, then cast to FP16 for next forward pass. Loss scaling multiplies loss by factor before backward pass to prevent gradients from underflowing to zero in FP16, then unscales after. BF16 (bfloat16) is increasingly preferred as it has same exponent range as FP32, reducing underflow issues and often eliminating need for loss scaling.

Mixed precision is essential for training large models. A 175B parameter model in FP32 would require 700GB just for weights - impossible on any GPU. In FP16, weights are 350GB, still impossible but closer. With optimizer states (Adam needs 2× parameters for momentum and variance), FP32 master weights + FP16 gradients + FP32 optimizer states = massive memory. Mixed precision with sharding enables training such models across hundreds of GPUs. Training speed improves 2-3× with tensor cores, reducing training time from months to weeks. For practitioners, mixed precision is standard practice - frameworks like PyTorch's Automatic Mixed Precision (AMP) make it trivial to enable.

Training GPT-3 175B on A100 GPUs (80GB memory). Without mixed precision: each GPU stores ~3.5GB of model weights (sharded) but compute in FP32 would be slow, tensor cores unused, and memory bandwidth saturated. With mixed precision: tensor cores operate at 312 TFLOPS vs 19.5 TFLOPS for FP32 - 16× faster matrix multiplications. Training time drops from 180 days to 34 days. Memory savings allow larger batch sizes, further improving efficiency. For smaller models, mixed precision enables fitting larger batches on single GPU, accelerating research iteration. This is why every major model training run uses mixed precision.

Perplexity is a measurement of how well a probability model predicts a sample, calculated as the exponential of the cross-entropy loss. Intuitively, it represents the average number of equally likely choices the model sees at each prediction step - lower perplexity means the model is more certain and accurate in its predictions. It's the most common intrinsic evaluation metric for language models.

Mathematically, perplexity for a sequence of N tokens is exp(-(1/N)∑ log P(token_i | context)). This is the exponential of average negative log-likelihood. For a uniform distribution over V vocabulary items, perplexity = V. For a perfect model that assigns probability 1 to correct token, perplexity = 1. In practice, good models achieve perplexity 10-30 on held-out data, meaning on average they're as uncertain as choosing uniformly from 10-30 options. Perplexity is computed on test data the model hasn't seen during training, measuring generalization. It's tokenization-dependent - different tokenizers yield different perplexities for same text, so comparisons require identical tokenization.

Perplexity provides a quick, objective measure of model quality without requiring downstream task evaluation. It correlates reasonably with performance on many tasks, especially those relying on fluent language modeling. During training, decreasing perplexity indicates the model is learning. Scaling laws use perplexity to predict performance of larger models. However, perplexity has limitations: it doesn't measure factual accuracy, reasoning, or safety - a model can have low perplexity by memorizing training data (overfitting) or by being overly conservative. It's also domain-dependent - medical text perplexity differs from news. Despite limitations, it remains the standard for comparing model architectures and training progress.

Two models on Wikipedia test set: Model A perplexity 12.5, Model B perplexity 15.2. Model A is better at predicting next token, meaning it has learned language patterns more accurately. On a next-word prediction task, Model A would be correct more often. However, this doesn't guarantee Model A answers questions better - a model fine-tuned for QA might have higher perplexity but better task performance. For comparing base pretrained models, lower perplexity generally indicates stronger language understanding. GPT-3 achieved perplexity ~10 on some datasets, while smaller models might score 20+, reflecting its superior language modeling capability.

Data deduplication is the process of identifying and removing duplicate or near-duplicate examples from training corpora at multiple levels - document-level, paragraph-level, and sentence-level. It prevents the model from memorizing repeated content rather than learning generalizable patterns, improving training efficiency and model generalization.

Deduplication operates at multiple granularities. Document-level: using MinHash or similar techniques to find near-duplicate documents (e.g., same article on different websites) and removing all but one. Paragraph/sentence-level: removing repeated blocks within documents (boilerplate, navigation menus). URL-level: filtering known low-quality domains. Exact hash matching catches identical documents; fuzzy matching (MinHash, SimHash) catches near-duplicates. For web-scale corpora (billions of documents), deduplication uses distributed processing (Spark, MapReduce) and approximate algorithms for efficiency. The process can remove 10-30% of raw web crawl data.

Without deduplication, models waste capacity memorizing repetitions rather than learning general patterns. This causes several problems: 1) Training inefficiency - compute spent on repeated content provides diminishing returns. 2) Memorization over generalization - model learns to recite duplicates rather than understand. 3) Benchmark contamination - if test data duplicates training data, performance metrics are inflated. 4) Bias toward overrepresented content - repeated web templates dominate training signal. Studies show deduplication improves perplexity and downstream performance equivalent to training on 2× more undeduplicated data. For example, the C4 dataset deduplication removed 30% of tokens while improving model quality.

Common Crawl contains 50 copies of the same Wikipedia article from different mirrors, plus 100 copies of popular news articles across websites. Without deduplication, a 1.4T token corpus might actually contain only 1T unique tokens - 30% redundancy. Model trained on deduplicated data sees 1T unique tokens; model trained on raw sees same unique content but with repetitions. At same training compute, deduplicated model has seen 30% more unique content, leading to better generalization. In practice, C4 deduplication: 364M documents → 267M after URL dedup → 174M after line-by-line dedup → 146M after near-duplicate removal. Final corpus is 60% smaller but produces better models.

After initial pretraining, LLMs typically undergo several additional stages to make them useful, safe, and aligned with human preferences. The main stages are Supervised Fine-Tuning (SFT) to teach task following, and Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) to align outputs with human preferences for helpfulness, honesty, and harmlessness.

Stage 1 - SFT: The pretrained model is fine-tuned on high-quality instruction-response pairs (typically 10k-100k examples) collected from humans or distilled from stronger models. This teaches the model to follow instructions and respond in helpful formats. Stage 2 - Reward Modeling: Humans compare model outputs for the same prompt, ranking them by preference. A reward model is trained to predict human preferences. Stage 3 - RLHF: The policy (LLM) is further trained using reinforcement learning (PPO) to maximize reward model scores while minimizing KL divergence from SFT model to prevent drift. Alternative DPO: directly optimizes policy using preference data without separate reward model and RL loop. Additional stages may include safety filtering, refusal training, and red-teaming.

Pretrained models can generate text but don't follow instructions well and may produce harmful content. SFT alone improves instruction following but may not align with nuanced human preferences about helpfulness and safety. RLHF dramatically improves alignment, making models like ChatGPT helpful, harmless, and honest. The staged approach enables scaling: pretraining uses abundant unlabeled data, SFT uses modest labeled data, RLHF uses expensive preference data efficiently. This pipeline produces models that are both capable and aligned. Without these stages, models are raw text generators, not useful assistants.

GPT-3 (pretrained only) can complete text but if asked 'How do I make a bomb?' might provide information. After SFT on safety data, model learns to refuse harmful requests. After RLHF, model not only refuses but does so helpfully ('I cannot provide information on harmful substances...') and maintains helpful tone throughout conversation. The three stages progressively shape behavior: pretraining provides knowledge, SFT teaches format, RLHF aligns values. This is why ChatGPT feels so different from base GPT-3 - the post-training pipeline transforms raw capability into aligned assistance.

Context length during pretraining determines the maximum sequence length the model sees during training, which fundamentally shapes its ability to handle long-range dependencies, reason over long documents, and maintain coherence over extended generations. It's a critical design decision with implications for architecture, compute cost, and model capabilities.

During pretraining, sequences are truncated to the chosen context length (e.g., 512 for BERT, 2048 for GPT-3, 8192 for LLaMA, 128k for recent models). Attention computations scale quadratically with context length, so longer contexts require more compute and memory. Models learn to attend within this window; information beyond it is inaccessible. Positional encodings (like RoPE) may extrapolate to longer lengths at inference, but performance typically degrades beyond trained length. Some techniques (ALiBi, attention scaling) improve extrapolation, but training on target length is ideal.

Context length determines what tasks a model can handle. Short context (512-2048): sufficient for sentences, paragraphs, short articles. Long context (8192-32768): enables processing research papers, multi-turn conversations, code files. Very long context (128k-1M): allows book-length understanding, long video analysis, extended agent trajectories. Trade-offs: longer context increases pretraining cost (2× length = 4× attention compute) and may reduce quality on short tasks if capacity is diverted. Models trained on short context cannot handle long documents at inference; models trained on long context can handle short tasks but cost more to train. Recent research shows long-context pretraining improves performance even on short tasks by enabling better reasoning about relationships.

Model A trained on 2k context. User shares 10k token research paper and asks summary. Model must truncate to first 2k tokens, losing critical later findings. Summary will be incomplete and potentially wrong. Model B trained on 32k context processes entire paper, attends to conclusion while reading introduction, produces comprehensive summary. This enables applications like legal document analysis, book summarization, and long-context RAG where entire documents fit without chunking. The 16× context increase cost maybe 8× more compute, but enables entirely new use cases.

Tokenizer vocabulary size determines the granularity of text representation during pretraining, affecting both model architecture (embedding layer size) and data representation (sequence length). It's a fundamental design choice that impacts training efficiency, model capacity, and downstream performance through trade-offs between embedding parameters and sequence length.

Larger vocabulary (e.g., 256k) means more words become single tokens, reducing average sequence length for given text. This decreases attention compute (O(n²)) and allows more content within context window. However, embedding layer parameters = vocab_size × d_model, so larger vocabulary significantly increases model size (e.g., 256k × 4096 = 1.05B parameters just for embeddings). Smaller vocabulary (e.g., 32k) increases sequence length (more subwords per word) but reduces embedding parameters. The vocabulary is built via BPE/WordPiece/SentencePiece on training corpus, with merges continuing until target size reached. The tokenizer itself must be trained once, then frozen for all training.

Vocabulary size affects both training cost and model quality. Larger vocabularies reduce sequence length, directly reducing attention FLOPs - for a 1T token corpus, 20% sequence length reduction saves 20% compute. But embedding layer becomes parameter-heavy, potentially dominating model size. The optimal size depends on language: morphologically rich languages (Turkish, Finnish) benefit from larger vocabularies to capture word forms; character-based languages (Chinese) need larger vocabularies to cover characters. Scaling laws suggest optimal vocabulary grows with model size - larger models can afford bigger embeddings. The tokenizer also affects multilingual fairness - vocabulary must balance coverage across languages.

Two 7B models with different vocabularies: Model A vocab 32k, average English tokenization ratio 1.5 tokens/word, embedding layer 32k×4096=131M params. Model B vocab 128k, ratio 1.2 tokens/word, embedding layer 128k×4096=524M params (4× larger). For 1T training tokens: Model A processes 1.5T tokens (more compute), Model B processes 1.2T tokens (20% less compute). The extra 393M embedding params in Model B may improve handling of rare words and morphology. Which is better? Depends on total compute budget and language mix. For multilingual, Model B likely better; for English-only, trade-off may be neutral. This is why frontier models use different vocabularies (GPT-4 ~100k, LLaMA 32k, PaLM 256k).

Catastrophic forgetting is the tendency of neural networks to lose previously learned knowledge when trained on new data, particularly when the new data distribution differs from the original training distribution. In continual pretraining (training a model on new domains after initial pretraining), this can cause the model to forget general knowledge while acquiring domain expertise.

During gradient-based learning, weights are updated to minimize loss on current training data. These updates move weights away from regions that were optimal for previous data. If new data distribution differs significantly, weight changes that help on new data may harm performance on old tasks. In neural networks, knowledge is distributed across weights, and updates for new tasks can overwrite features useful for old tasks. The severity depends on learning rate, data similarity, and model capacity. High learning rates and dissimilar data cause more forgetting.

Catastrophic forgetting limits continual learning - you can't simply train a model sequentially on different domains without losing previous capabilities. For LLMs, this matters for continued pretraining on domain data (medical, legal). If you continue pretraining a general model on medical texts with standard learning rate, it may become excellent at medicine but forget how to write poetry or answer general questions. Mitigations include: lower learning rates (slower adaptation, less forgetting), replay (mixing in old data), regularization (penalizing weight changes), and elastic weight consolidation (identifying important weights for old tasks). Understanding forgetting guides how to adapt models to new domains while preserving general capabilities.

Base LLaMA trained on general corpus (web, books, code). Continue pretraining on 50B medical tokens with LR 1e-5 (standard for fine-tuning). After medical training, evaluate on general benchmarks (MMLU) and medical benchmarks (MedQA). Medical accuracy improved from 60% to 80%, but general MMLU dropped from 65% to 55% - catastrophic forgetting of general knowledge. To mitigate, use lower LR (1e-6) and mix 20% general data during medical training. Result: medical 75% (good), general 63% (minimal loss). This balanced approach preserves general capabilities while acquiring domain expertise. Without understanding forgetting, you'd ruin your general model.

Estimating pretraining compute cost in FLOPs (floating point operations) is essential for planning training runs, budgeting cloud resources, and applying scaling laws. The total FLOPs depend primarily on model size (parameters), training tokens, and architecture details, following well-established formulas derived from transformer mathematics.

For a transformer model, forward+backward FLOPs per token ≈ 6 × number of parameters (this is the key approximation). So total training FLOPs = 6 × N × D, where N is parameter count, D is number of training tokens. This accounts for both forward and backward passes. Additional factors: vocabulary size affects embedding layer FLOPs (2 × vocab_size × d_model per token), attention FLOPs scale as 2 × n_layers × seq_len × d_model (for QK^T and softmax), but for large models the 6ND approximation is accurate within 10-20%. For Chinchilla-optimal training, D ≈ 20N, so FLOPs ≈ 120N². More precise estimation uses: C = 6ND + 2N_vocab × d_model × D + 4 × n_layers × n_heads × d_head × seq_len × D.

Compute estimation enables cost forecasting (e.g., 6ND FLOPs on A100 at 312 TFLOPS → time). It guides model design - scaling laws show optimal N for given compute budget. For practitioners, it helps decide whether to train from scratch vs fine-tune. Cloud costs directly proportional to FLOP-seconds. For researchers, it enables comparing efficiency of architectures. Underestimating can lead to budget overruns; overestimating leaves resources idle. The 6ND formula is industry standard for quick estimation.

Planning to train 7B model on 2T tokens (Chinchilla-optimal for 7B is ~140B, so 2T is overkill). Compute = 6 × 7e9 × 2e12 = 8.4e22 FLOPs. On A100 GPUs at 312 TFLOPS (3.12e14 FLOPs/sec), total GPU-seconds = 8.4e22 / 3.12e14 = 2.69e8 seconds = 74,722 GPU-hours. With 1000 GPUs, training time = 74.7 hours (3.1 days). At $2/GPU-hour, cost = $149,444. More precise: add 10% for attention overhead → 82,000 GPU-hours, $164,000. This quick estimate lets you decide: is this worth the cost for your application? Should you use smaller model or less data? The formula turns abstract model sizes into concrete budgets.