AI Interview Questions

Fine-tuning is the process of further training a pre-trained language model on a specific dataset to adapt it for particular tasks or domains. Unlike prompt engineering which crafts inputs to guide the model without weight changes, fine-tuning modifies the model's parameters to optimize performance on targeted objectives, creating a specialized version of the base model.

Fine-tuning starts with a pre-trained model (e.g., LLaMA, GPT-3) and continues training on a curated dataset of examples relevant to your use case. The model's weights are updated via backpropagation, typically with a lower learning rate than pretraining to preserve general knowledge while adapting to new patterns. Training continues until performance on validation set plateaus. The resulting model retains general capabilities but excels at the target task. Prompt engineering, by contrast, uses techniques like few-shot examples, chain-of-thought, and system prompts to elicit desired behavior without any weight updates.

The choice between fine-tuning and prompt engineering involves trade-offs in cost, performance, and flexibility. Fine-tune when: task requires specialized knowledge not in base model (medical diagnosis, legal analysis), output format must be precise (structured data extraction), or you need consistent performance at scale. Prompt engineer when: task is general, you need quick iteration, or cost of fine-tuning outweighs benefits. Fine-tuning typically outperforms prompting on narrow tasks but loses flexibility - a fine-tuned model excels at its task but may struggle on others. Prompting keeps model general but may be less reliable. For production, many systems use both: prompt engineering for rapid prototyping, fine-tuning for deployment at scale.

Customer support automation for a bank. Prompt engineering with GPT-4 works for general queries but inconsistently handles specific policies (interest rates, loan terms). Fine-tuning on 10k support tickets with correct answers produces model that reliably cites correct policies, uses proper terminology, and maintains consistent tone. However, this fine-tuned model might perform worse on general chat. The bank deploys both: fine-tuned model for support, general model for other tasks. Cost-benefit: fine-tuning cost $500, saves 100 hours of agent time weekly - clear ROI.

Supervised fine-tuning is the process of training a pre-trained language model on labeled examples of inputs and desired outputs to teach it specific tasks or behaviors. Unlike pretraining which learns from unlabeled text via self-supervised objectives, SFT uses human-curated demonstrations to shape the model's responses for particular applications.

Pretraining trains on massive unlabeled corpora (trillions of tokens) using self-supervised objectives like next-token prediction. The model learns general language patterns, facts, and reasoning without explicit instruction. SFT takes this pretrained model and trains it on a dataset of (prompt, response) pairs, typically 10k-100k examples. The training objective is standard supervised learning - maximize probability of target responses given prompts. Learning rate is much lower than pretraining (typically 1e-5 vs 1e-4) to prevent catastrophic forgetting. The model updates weights to align with demonstration patterns while preserving general knowledge.

SFT bridges the gap between raw pretrained models and useful assistants. Pretrained models can generate text but don't follow instructions well - they complete text rather than answer questions. SFT on instruction-response pairs teaches the model to be helpful, follow formats, and handle diverse tasks. This is why ChatGPT feels different from base GPT-3 - SFT on high-quality conversations transformed raw generation into dialogue. SFT also enables domain adaptation: training on medical Q&A produces better doctor assistant. The quality of SFT data matters enormously - 10k excellent examples often outperform 100k noisy ones. SFT is the foundation upon which further alignment (RLHF) builds.

Base LLaMA pretrained on 2T tokens can complete text: prompt 'The capital of France is' → likely 'Paris'. But prompt 'What is the capital of France?' might get random continuation because it wasn't trained for Q&A format. After SFT on 50k instruction-response pairs (from human demonstrations or distilled from GPT-4), model learns pattern: given question, provide helpful answer. Now 'What is the capital of France?' → 'The capital of France is Paris.' The model hasn't learned new facts (it already knew Paris) but learned to apply knowledge in helpful format. This formatting knowledge transfers to other questions it already knew answers to.

LoRA is a parameter-efficient fine-tuning technique that freezes the pre-trained model weights and injects trainable low-rank matrices into each layer of the transformer architecture. Instead of updating all model parameters during fine-tuning, LoRA learns small rank decomposition matrices that adapt the model to new tasks, reducing trainable parameters by orders of magnitude.

For a pre-trained weight matrix W₀ of size d×k, LoRA adds a bypass: W₀ + ΔW = W₀ + BA, where B is d×r, A is r×k, and r << min(d,k) (typically r=1-64). During fine-tuning, W₀ is frozen (no gradient updates), while A and B are trained. The forward pass becomes h = W₀x + BAx. This approximates full fine-tuning because the adaptation ΔW is low-rank, assuming weight updates have low intrinsic rank. For multi-head attention, LoRA typically adapts query and value projection matrices. At inference, the adapted weights can be merged: W = W₀ + BA, adding no latency overhead.

LoRA dramatically reduces memory and storage requirements for fine-tuning. For a 7B model, full fine-tuning requires storing gradients and optimizer states for all 7B parameters - about 84GB memory with Adam (2× parameters for optimizer states). LoRA with r=8 on query/value matrices trains only ~4M parameters - 0.05% of original - requiring <1GB additional memory. This enables fine-tuning on consumer GPUs (24GB) that couldn't otherwise fit 7B model. Multiple LoRA adapters can be trained for different tasks and swapped at inference, enabling efficient multi-task serving. Storage of adapters is tiny (megabytes) vs full model copies (gigabytes). Performance matches full fine-tuning for many tasks, especially when r sufficient.

Fine-tuning LLaMA-7B for medical Q&A. Full fine-tuning: 7B parameters trained, requires 4× A100-80GB (distributed), takes 2 days, produces 14GB checkpoint. LoRA (r=8): 4.2M parameters trained (0.06% of model), runs on single 24GB GPU in 4 hours, produces 16MB adapter file. At inference, merge adapter with base model (once) or load separately. Performance on medical benchmark: full fine-tuning 82% accuracy, LoRA 81% - nearly identical. The 1000× reduction in trainable parameters makes fine-tuning accessible, faster, and cheaper. This democratizes model adaptation for researchers and practitioners with limited resources.

QLoRA (Quantized Low-Rank Adaptation) is a memory-efficient fine-tuning technique that combines 4-bit quantization of the base model with LoRA adapters. It enables fine-tuning of massive models (e.g., 65B parameters) on a single consumer GPU by drastically reducing memory footprint while preserving performance through novel quantization techniques.

QLoRA first quantizes the pre-trained model to 4-bit precision using NormalFloat (NF4) quantization, a information-theoretically optimal data type for normally distributed weights. The model is further compressed via double quantization - quantizing the quantization constants themselves. During fine-tuning, the 4-bit base model remains frozen, while LoRA adapters (in FP16/BF16) are trained on top. Forward pass dequantizes 4-bit weights on-the-fly to compute with LoRA adapters. Gradients flow only through LoRA parameters, not the quantized base. This reduces memory from 16-bit (2 bytes per parameter) to 4-bit (0.5 bytes) for base model - 4× reduction.

QLoRA democratizes fine-tuning of the largest models. A 65B model in 16-bit requires 130GB just for weights - impossible on consumer hardware. With 4-bit quantization, weights are 32.5GB, fitting on 48GB professional GPUs or multiple 24GB consumer GPUs with offloading. Including gradients and optimizer states for LoRA adapters (tiny), full fine-tuning fits on single 24GB GPU. This enabled the explosion of open-source fine-tuned models - anyone with a gaming PC can now fine-tune LLaMA-65B. Performance remains excellent because quantization preserves model capabilities and LoRA provides adaptation capacity. QLoRA-powered models like Guanaco achieved state-of-art on benchmarks, matching 16-bit performance.

Fine-tuning LLaMA-65B on consumer setup (RTX 4090 24GB). Without QLoRA: impossible - weights alone 130GB. With QLoRA: base model quantized to 4-bit (32.5GB) with page-to-host offloading (slightly slower). LoRA adapters (r=64) add 0.5GB. Total fits with memory optimization. Training takes 24 hours instead of 2 days on professional GPUs, costs $0 (existing hardware) instead of $1000 cloud bill. Final model achieves 98% of full fine-tuning performance. This is why QLoRA became the standard for open-source fine-tuning - it made 65B model adaptation accessible to individuals, not just large labs.

Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques that adapt large language models to downstream tasks by training only a small subset of parameters, freezing the majority of the pre-trained model. These methods drastically reduce computational and memory requirements while often matching full fine-tuning performance, enabling efficient multi-task serving and deployment on resource-constrained hardware.

PEFT methods modify or augment the pre-trained model in various ways while keeping most weights frozen. Major approaches include: 1) Adapter-based: Insert small trainable modules (adapters) between transformer layers (AdapterFusion, Series Adapters). 2) LoRA and variants: Inject low-rank matrices into attention layers. 3) Prefix-tuning: Optimize continuous prompts prepended to inputs. 4) Prompt-tuning: Learn soft prompts (trainable embeddings) while freezing model. 5) BitFit: Only train bias terms. 6) IA3: Learn vectors that rescale activations. Each method adds minimal parameters (0.1-2% of model) and can be trained independently for different tasks, enabling multi-task serving with single base model.

PEFT addresses the practical challenges of adapting ever-larger models. Full fine-tuning 175B model requires storing multiple 350GB copies for different tasks - prohibitively expensive. With PEFT, one base model plus tiny task-specific adapters (megabytes each) serves hundreds of tasks. Training costs drop from thousands of GPU-hours to hours on single GPU. Research iteration accelerates - try 10 PEFT configs in time of one full fine-tune. PEFT also enables on-device adaptation - download tiny adapter instead of full model. Performance often matches full fine-tuning, especially for tasks similar to pretraining. This is why PEFT has become standard practice for production LLM deployment.

Company needs 50 specialized models: customer support in 10 languages, 5 domains (medical, legal, technical), 5 tasks (classification, generation, summarization). Full fine-tuning: 50 copies of 7B model = 350GB storage, 5000 GPU-hours training, switching models slow. PEFT with LoRA: one base 7B model (14GB) + 50 LoRA adapters (16MB each = 0.8GB total). Training: 100 GPU-hours total (2 hours per adapter). Switching: load new adapter in milliseconds. Cost savings: storage 14.8GB vs 350GB, training 100 vs 5000 GPU-hours. This is why enterprises adopt PEFT - it makes specialization practical at scale.

Instruction tuning is a fine-tuning approach where language models are trained on a diverse set of tasks formatted as natural language instructions and corresponding responses. This teaches models to follow human instructions across many tasks without task-specific fine-tuning, dramatically improving zero-shot generalization and making models usable out-of-the-box for non-experts.

Researchers curate or generate thousands of tasks (translation, summarization, QA, reasoning, creative writing) and format each as an instruction-response pair. For example, 'Translate this sentence to French: Hello, how are you?' with response 'Bonjour, comment allez-vous?'. Models are fine-tuned on this mixture, learning to interpret instructions and produce appropriate responses. The key is diversity - covering many task types, formats, and domains teaches the model to generalize to unseen instructions. FLAN (Fine-tuned LAnguage Net) pioneered this, using 60+ NLP datasets. Modern versions use tens of thousands of tasks from public datasets and synthetic generation.

Before instruction tuning, using LLMs required careful prompt engineering - users had to know how to format tasks for the model. Instruction-tuned models like ChatGPT understand plain language commands: 'Summarize this article', 'Write a poem about dogs', 'Explain quantum physics simply'. This made LLMs accessible to everyone, not just ML engineers. Zero-shot performance on unseen tasks improved dramatically - instruction-tuned models match few-shot performance of base models. The technique also improves following complex instructions and reduces need for examples. Instruction tuning was the key innovation that transformed LLMs from research tools to consumer products.

Base GPT-3 given 'Translate to French: Hello' might complete with 'Hello in French is bonjour' (explanation) or continue with more English. Instruction-tuned FLAN given same prompt produces 'Bonjour' directly because it learned translation format. For novel task 'Explain this code in simple terms to a non-programmer' (not in training data), instruction-tuned model generalizes from similar 'explain' tasks and produces appropriate response. Base model might produce technical explanation or continue code. This generalization ability - following instructions it never saw - is why instruction tuning revolutionized LLM usability.

Catastrophic forgetting is the phenomenon where neural networks lose previously learned knowledge when trained on new tasks or data. In fine-tuning LLMs, this means the model may forget general knowledge or capabilities while adapting to a specific domain, degrading performance on tasks unrelated to the fine-tuning objective.

During fine-tuning, gradient updates modify weights to minimize loss on the target dataset. These updates move weights away from regions that were optimal for the original pretraining distribution. Since neural network representations are distributed and overlapping, changes that improve task A can damage features useful for task B. The severity depends on learning rate (higher = more forgetting), data similarity (dissimilar data causes more interference), and model capacity (larger models resist forgetting better). Forgetting manifests as degraded performance on general benchmarks, loss of language fluency, or inability to handle tasks outside fine-tuning domain.

Forgetting limits the practicality of fine-tuning - you can't create a specialist model without breaking general capabilities. A medical fine-tuned model that forgets how to answer basic questions is less useful. Several mitigation strategies exist: 1) Lower learning rates (1e-5 vs 1e-4) slow adaptation, reducing forgetting. 2) Mix in general data during fine-tuning (10-20% of batch from original distribution). 3) Elastic Weight Consolidation (EWC) - identify important weights for general tasks and penalize their changes. 4) Multi-task fine-tuning - train on target and general tasks simultaneously. 5) PEFT methods like LoRA inherently reduce forgetting by limiting parameter updates. Understanding and mitigating forgetting is essential for successful fine-tuning.

Fine-tune LLaMA on 100k medical Q&A pairs with LR 2e-5. Before: MMLU general knowledge score 65%, medical QA 60%. After: medical QA 85% (great), MMLU drops to 45% - catastrophic forgetting. Model forgot general facts while learning medicine. Mitigation: reduce LR to 5e-6, mix 20% general data in each batch. After: medical QA 82% (slightly lower), MMLU 62% (preserved). The trade-off - slight loss in medical performance for preserved generality - is usually worth it. For production, you want specialist that remains useful for general queries too.

Preparing a high-quality fine-tuning dataset involves collecting, cleaning, formatting, and validating examples that represent the desired task or behavior. The dataset quality directly determines fine-tuning success - 10k excellent examples often outperform 100k noisy ones. Systematic preparation ensures the model learns correct patterns without picking up artifacts or biases.

Process typically includes: 1) Data collection - gather from existing sources (support tickets, human demonstrations) or generate (using stronger models, human annotation). 2) Cleaning - remove duplicates, filter low-quality examples, fix formatting errors, handle PII redaction. 3) Format standardization - ensure consistent prompt structure, output format, special tokens. 4) Diversity analysis - check coverage of input types, edge cases, difficulty levels. 5) Quality validation - sample review by humans, test on held-out set, check for contradictions. 6) Balancing - ensure representation across categories, prevent majority class bias. 7) Splitting - train/validation/test sets with careful stratification.

Garbage in, garbage out applies strongly to LLMs. Noisy data teaches wrong patterns - model may learn to hallucinate, ignore instructions, or produce low-quality outputs. Duplicates cause overfitting and memorization. Format inconsistencies confuse the model about expected structure. Missing edge cases mean model fails in production on those scenarios. Bias in data propagates to model behavior. Investing in dataset quality pays off in model performance, reducing need for multiple fine-tuning iterations. For many applications, data preparation is 80% of the work, fine-tuning 20%.

Building customer support fine-tuning dataset. Raw data: 50k support tickets with agent responses. Issues: 30% duplicates, 10% contain customer PII, responses vary in quality (some agents better), formats inconsistent (some with ticket numbers, some without). Preparation: deduplicate (35k remain), redact PII, filter to only high-rated agent responses (20k), standardize format to remove ticket numbers, add system prompt prefix. Validate 500 samples manually, finding 5% still problematic - fix those patterns. Create train/val/test splits stratified by issue type. Resulting 19k high-quality examples produce model that outperforms one trained on raw 50k, despite fewer examples. The cleaned data taught correct patterns without noise.

Full fine-tuning updates all parameters of a pre-trained model during adaptation, creating a completely new version of the model specialized for the target task. Adapter-based fine-tuning (including LoRA, prefix tuning, etc.) keeps the original model frozen and trains only a small set of additional parameters inserted into the architecture, creating lightweight task-specific modules that can be swapped or combined.

Full fine-tuning initializes from pre-trained weights and continues training on target data with gradients flowing through all parameters. All 7B (or 175B) weights are updated, producing a new model checkpoint of same size. Adapter methods freeze base model and introduce trainable parameters: LoRA adds low-rank matrices to attention layers, adapters insert small MLP modules between transformer layers, prefix tuning optimizes continuous prompts. Only these added parameters (0.1-2% of total) are updated. During inference, adapters can be merged (LoRA) or used as separate branches (adders).

The choice involves trade-offs in performance, storage, flexibility, and compute. Full fine-tuning can achieve slightly better performance on very specialized tasks, especially when target distribution differs significantly from pretraining. But it requires: storing separate model copies per task (e.g., 14GB per 7B model), expensive training (multiple GPUs), and risks catastrophic forgetting. Adapter methods excel at multi-task serving: one base model + tiny adapters (MB each) serves hundreds of tasks. Training is cheap (single GPU, hours), enabling rapid iteration. Performance often matches full fine-tuning, especially for tasks similar to pretraining. For most practical applications, adapter methods are preferred due to efficiency.

Company needs models for 10 languages and 5 domains (50 total). Full fine-tuning: 50 copies of 7B model = 700GB storage, 5000 GPU-hours training, switching models slow. Adapter-based (LoRA): one base 7B model (14GB) + 50 LoRA adapters (16MB each = 0.8GB total). Training: 200 GPU-hours (4 hours each). Performance comparison: on domain-specific tests, full fine-tuning averages 85% accuracy, LoRA 84% - essentially identical. For 2% performance gain, full fine-tuning costs 25× more storage and 25× more training compute - not worth it. Adapter methods win for most production scenarios.

Fine-tuning hyperparameters control the optimization process and significantly impact model quality, training stability, and computational cost. Key hyperparameters include learning rate, batch size, number of epochs, optimizer settings, weight decay, and warmup steps. Their careful tuning is essential for achieving optimal performance without overfitting or catastrophic forgetting.

Learning rate (LR) determines step size during optimization - too high causes instability and forgetting, too low leads to slow convergence or underfitting. Typical range 1e-6 to 5e-5. Batch size affects gradient estimate quality and memory usage - larger batches provide stable gradients but require more memory. Epochs control how many passes over training data - too few underfits, too many overfits (especially with small datasets). Optimizer (usually AdamW) has its own hyperparameters (β1, β2, ε). Weight decay adds regularization (typically 0.01-0.1). Warmup steps (0-10% of total) gradually increase LR to prevent early instability. Learning rate schedule (cosine, linear) shapes LR decay.

Poor hyperparameter choices waste compute and produce suboptimal models. LR too high: loss spikes, model forgets general knowledge, may diverge. LR too low: training takes longer, may plateau at poor performance. Too many epochs: model memorizes training data (overfitting), fails on validation. Too few: underfitting, doesn't learn task. Batch size affects both convergence and hardware utilization. Finding optimal values typically requires systematic experimentation (grid search, Bayesian optimization) but heuristics help: start with LR 2e-5, batch size adapted to GPU memory, early stopping based on validation loss. The right hyperparameters often improve performance by 5-10% over default choices.

Fine-tuning 7B model on 10k examples with different LR: LR 1e-4: loss spikes, validation accuracy 60%. LR 2e-5: smooth training, validation accuracy 85%. LR 1e-6: training very slow, validation accuracy 75% after same steps. Epochs: at 1 epoch, accuracy 80%; 3 epochs, 85%; 10 epochs, 86% but validation loss increases (overfitting). Optimal: 3 epochs, LR 2e-5 with cosine decay, batch size 32 (fits GPU), 100 warmup steps. This tuning improved performance 5% over initial guess, worth the experiment cost. In production, systematic hyperparameter optimization pays for itself through better model quality.

Prefix tuning is a parameter-efficient fine-tuning method that prepends a small number of trainable continuous vectors (prefix tokens) to each transformer layer's input. Unlike LoRA which modifies weights via low-rank matrices, prefix tuning keeps all model weights frozen and instead optimizes these virtual tokens that influence attention computations across all layers.

For each transformer layer, prefix tuning adds k trainable prefix vectors to the keys and values in the attention mechanism. These vectors are prepended to the actual input representations, so attention can attend to both real tokens and learned prefixes. The prefixes are continuous embeddings (not actual tokens) that are optimized during fine-tuning while the base model remains frozen. Typical prefix length is 5-100 vectors per layer, adding 0.1-2% of model parameters. During inference, prefixes are concatenated with actual input for each forward pass. The prefixes learn task-specific attention patterns that steer model behavior.

Prefix tuning offers different trade-offs than LoRA. It adds no inference latency (prefixes are part of input) and can be more expressive than low-rank updates for some tasks. It's particularly effective for conditional generation tasks where prefixes can learn task-specific control codes. However, prefixes consume part of the context window (k tokens per layer effectively), reducing available space for actual input. Training can be less stable than LoRA. Performance generally comparable to LoRA, with some tasks favoring one over the other. The choice depends on whether you prefer weight-based adaptation (LoRA) or activation-based steering (prefix tuning).

Fine-tuning on summarization task. LoRA adapts attention weights to focus on important sentences. Prefix tuning learns 50 prefixes per layer that, when prepended, guide attention to salient parts of input. Both achieve similar ROUGE scores. But prefix tuning uses 50×12 layers × 2 (K,V) = 1200 tokens of context window for prefixes - significant for long documents. LoRA adds no context overhead. Conversely, prefix tuning can be more interpretable - analyzing learned prefixes reveals what patterns model learned to attend to. In practice, LoRA is more widely adopted due to simplicity and no context overhead, but prefix tuning remains valuable for specific applications.

Fine-tuning and RAG represent different approaches to incorporating knowledge into LLM applications. Fine-tuning embeds knowledge directly into model weights through continued training on domain data. RAG keeps the model static and retrieves relevant information from external databases at inference time to augment prompts. The choice depends on knowledge characteristics, update frequency, and operational requirements.

Fine-tuning modifies model weights to internalize domain knowledge, making it part of the model's parametric memory. Once trained, the model answers queries using this internalized knowledge without external retrieval. RAG maintains a separate knowledge base (vector database) and at inference time: 1) retrieves relevant documents for query, 2) adds them to prompt context, 3) generates answer grounded in retrieved information. Knowledge updates require re-fine-tuning vs simply updating database.

Choose fine-tuning when: knowledge is stable (won't change frequently), you need fast inference (no retrieval latency), domain has subtle patterns best learned through weight updates, or you're working with small, well-defined knowledge sets. Choose RAG when: knowledge changes frequently (news, products), you need to cite sources, handling very large knowledge bases (millions of documents), or different users need different knowledge access. RAG also enables updating knowledge without model retraining and reduces hallucination by grounding in retrieved data. Many production systems combine both: fine-tune for style/task mastery, RAG for factual knowledge.

Customer support for software product with version 2.0 launching in 3 months. RAG: knowledge base updated with new docs at launch, no model retraining needed. Fine-tuning: would require new training run for each version, slow and costly. Conversely, medical diagnosis system using stable textbooks and journals: fine-tuning internalizes this stable knowledge, providing faster inference and better reasoning patterns. RAG would need retrieval for every query, adding latency. The trade-off: fine-tuning gives speed and deep integration, RAG gives flexibility and updatability. Choose based on your knowledge dynamics and latency requirements.

Evaluating a fine-tuned model requires measuring its performance on the target task, assessing generalization to held-out examples, and ensuring it hasn't lost general capabilities through catastrophic forgetting. A comprehensive evaluation combines task-specific metrics, human evaluation, and benchmark testing to validate that the model meets production requirements.

Process typically includes: 1) Hold-out test set - evaluate on examples never seen during training using task-appropriate metrics (accuracy for classification, ROUGE for summarization, BLEU for translation). 2) Human evaluation - sample outputs rated by humans for quality, helpfulness, correctness. 3) Side-by-side comparison - against base model and alternatives. 4) General capability testing - run on standard benchmarks (MMLU, HellaSwag) to detect catastrophic forgetting. 5) Edge case testing - evaluate on difficult examples, out-of-distribution inputs. 6) A/B testing in production - compare against existing system on live traffic with business metrics.

Metric improvements alone don't guarantee production success. A model might achieve high accuracy on test set but fail on real user queries (distribution shift), produce fluent but incorrect answers (hallucination), or lose general knowledge needed for mixed queries. Human evaluation catches nuances metrics miss. Benchmark testing ensures model remains useful for other tasks. Production A/B testing measures actual business impact (conversion, satisfaction). Without comprehensive evaluation, you risk deploying a model that looks good on paper but fails in practice, eroding user trust and wasting investment.

Fine-tuned customer support model achieves 92% accuracy on test set (vs base 85%). Human evaluation reveals 10% of responses are technically correct but tone is abrupt - metric missed this. MMLU score dropped from 65% to 55% - significant forgetting. Production A/B test shows 5% faster resolution but 2% lower customer satisfaction due to tone. Evaluation reveals need for: 1) more training examples with good tone, 2) mixing general data to prevent forgetting, 3) toning adjustments. Without comprehensive evaluation, would have deployed model with hidden problems. The full picture guides further iteration, leading to 94% accuracy, 68% MMLU, and +3% satisfaction in next version.

Multi-task fine-tuning trains a single model simultaneously on multiple different tasks using a mixture of datasets, rather than fine-tuning separate models for each task. The model learns shared representations that benefit all tasks, improving efficiency and often performance through positive transfer between related tasks.

Training data combines examples from multiple tasks (e.g., summarization, translation, QA, classification), each formatted consistently (often with task prefixes). During training, batches sample from all tasks, and loss is computed per example regardless of task. The model must learn to perform each task correctly while sharing representations across tasks. Task balancing ensures no single task dominates training - typically via proportional sampling or loss weighting. The resulting model can handle multiple tasks through task-specific prompts or prefixes at inference.

Multi-task fine-tuning offers several benefits over single-task models. First, efficiency - one model serves many tasks, reducing storage and serving costs. Second, positive transfer - learning related tasks improves performance on each (e.g., summarization benefits from translation). Third, improved generalization - model learns more robust features by seeing diverse tasks. Fourth, enables zero-shot generalization to new tasks by composing learned skills. Fifth, reduces overfitting by providing more varied training signal. Instruction tuning is a form of multi-task fine-tuning at massive scale, which is why instruction-tuned models generalize so well.

Building models for legal tech: tasks include contract classification (good/bad clause), question answering on case law, summarization of rulings, and entity extraction. Single-task: four separate 7B models = 56GB storage, separate maintenance. Multi-task: one 7B model trained on mixture of all four tasks = 14GB storage. Performance: on individual tasks, multi-task model matches or exceeds single-task models due to transfer learning (contract classification improves via entity extraction knowledge). Training cost: 100 GPU-hours vs 4×25 = same, but inference cheaper and maintenance simpler. New task added later (legal translation) can leverage existing representations, requiring minimal additional training. Multi-task is almost always preferred when tasks are related.

Domain adaptation fine-tuning is the process of further training a general language model on text from a specific domain (e.g., medical, legal, technical) to improve its understanding of domain terminology, writing conventions, and knowledge. Unlike task-specific fine-tuning which teaches a particular output format, domain adaptation improves the model's fundamental comprehension of domain language.

Domain adaptation continues pretraining (not supervised fine-tuning) on large corpora of unlabeled domain text - millions to billions of tokens from medical papers, legal documents, technical manuals. The training objective remains next-token prediction (or masked LM), same as pretraining, but learning rate is lower to prevent forgetting. This exposes model to domain terminology, writing styles, and knowledge embedded in text. After adaptation, the model better represents domain concepts, which benefits all downstream tasks in that domain. Task-specific fine-tuning can then be applied on top.

General models know medicine from Wikipedia but lack depth from journals. Domain adaptation injects this depth, improving performance on all domain tasks simultaneously. For medical applications, a domain-adapted model (like BioBERT, ClinicalBERT) outperforms general models on entity recognition, QA, and classification without task-specific training. The adaptation is data-efficient - unlabeled domain text is abundant, unlike labeled task data. For specialized domains (nuclear engineering, patent law), general models perform poorly; domain adaptation is essential. The cost is modest relative to pretraining, making it highly cost-effective for domain-focused applications.

Building medical QA system. Base LLaMA scores 60% on MedQA. Domain adaptation on 50B tokens from PubMed and medical textbooks (cost $5k compute) improves to 75% on MedQA. Then task-specific fine-tuning on 10k medical QA pairs (cost $500) improves to 85%. Total improvement 25% from domain adaptation + 10% from fine-tuning. Without domain adaptation, fine-tuning alone might reach 70%. The domain adaptation gave 15% boost for 10× the compute cost of fine-tuning but 1/1000 the cost of pretraining from scratch. For high-stakes medical applications, this boost is well worth it. Domain adaptation is the secret weapon for specialized domains.

Fine-tuning LLMs can fail in several predictable ways, producing models that underperform, behave unexpectedly, or degrade in production. Understanding these failure modes helps practitioners design experiments, monitor training, and validate results before deployment, saving time and resources.

Common failures include: 1) Catastrophic forgetting - model loses general knowledge while learning task. 2) Overfitting - model memorizes training examples, fails on new data. 3) Format overfitting - model learns to mimic training format rigidly, failing on slight variations. 4) Hallucination increase - fine-tuning on noisy data teaches model to make things up. 5) Safety degradation - fine-tuning on unsafe data removes alignment. 6) Data contamination - test examples accidentally in training, inflating metrics. 7) Distribution shift - training data doesn't match deployment conditions. 8) Hyperparameter mismatch - poor choices lead to suboptimal convergence. 9) Catastrophic divergence - loss explodes, model unusable.

Each failure mode wastes resources and can damage user trust if deployed. Overfit model fails on real queries. Forgetting makes model less useful for general questions. Hallucination increases, spreading misinformation. Safety degradation can cause harmful outputs. Detection requires systematic evaluation: validation on held-out data (overfitting), benchmark testing (forgetting), human evaluation (hallucination), red-teaming (safety). Prevention strategies include: regularization (dropout, weight decay), early stopping, mixing general data, careful data cleaning, lower learning rates, validation throughout training. Understanding these modes guides quality assurance.

Fine-tuning customer support model on 5k tickets. After training, validation accuracy 95% - looks great. Deployment reveals: 1) For novel queries slightly different from training, accuracy drops to 60% (overfitting). 2) Model now responds with 'I don't know' to general questions it previously answered (forgetting). 3) Hallucinates product features not in training data (learned from noisy examples). 4) Occasionally responds rudely to frustrated customers (safety degradation from mimicking agent shortcuts). Root causes: training data lacked diversity, had some low-quality responses, no general data mixed in. Mitigation: collect more diverse data, clean aggressively, mix 20% general data, lower LR, add validation on general benchmarks. Next version fixes all issues. Without identifying failure modes, would blame model and restart from scratch.

The rank parameter (r) in LoRA determines the dimensionality of the low-rank matrices used to approximate weight updates. It controls the capacity of the adapter - higher rank allows more expressive adaptations but increases parameters and training cost. Choosing appropriate rank balances adaptation power against efficiency and overfitting risk.

LoRA decomposes the weight update ΔW into product of two matrices: B (d×r) and A (r×k), where original weight shape is d×k. Rank r << min(d,k) limits the expressivity - ΔW can only represent updates that lie in an r-dimensional subspace. With higher r, more of the full fine-tuning update space can be captured. Typical values: r=1 for very simple tasks, r=8-16 for most tasks, r=64-128 for complex tasks or when matching full fine-tuning. Parameters added = r×(d+k) per adapted layer. For LLaMA-7B (d=4096, k=4096), r=8 adds ~65k parameters per layer, total ~2M for all layers.

Rank selection involves trade-offs. Too low (r=1): may lack capacity to learn complex adaptations, underfitting the target task. Too high (r=128): more parameters, risk overfitting small datasets, slower training, minimal quality gain. Research shows most tasks need surprisingly low rank - r=8 often matches full fine-tuning performance, suggesting weight updates have low intrinsic dimensionality. Rank affects both final quality and training dynamics. For new tasks, start with r=8, evaluate on validation set. If underfitting, increase rank; if overfitting (validation loss increases after some steps), decrease rank or increase regularization. Some tasks benefit from different ranks per layer type.

Fine-tuning on legal document classification with varying r. Dataset: 10k examples. Results: r=1 accuracy 82%, r=4 87%, r=8 90%, r=16 91%, r=32 91% (plateau), r=64 91% but training 2× slower. Optimal r=8-16 gives 90-91% with efficient training. For smaller dataset (1k examples): r=1 75%, r=4 80%, r=8 82%, r=16 82% then overfitting (validation loss increases). Here optimal r=4-8. For very large dataset (100k examples): r=32 reaches 93% where r=8 stuck at 91% - more capacity needed. Rule: larger datasets and more complex tasks need higher rank; start with r=8, adjust based on validation curves.

Overfitting occurs when a model learns to memorize training examples rather than generalizing to new data, causing excellent training performance but poor validation/test results. Small datasets (hundreds to few thousand examples) are particularly susceptible, requiring careful regularization strategies to ensure the model learns task patterns rather than memorizing specifics.

Several techniques combat overfitting in fine-tuning: 1) Lower learning rate (1e-5 to 5e-6) slows adaptation, reducing capacity to memorize. 2) Early stopping - monitor validation loss, stop when it starts increasing. 3) Weight decay (0.01-0.1) penalizes large weights, encouraging simpler solutions. 4) Dropout (0.1-0.3) randomly masks neurons during training, preventing co-adaptation. 5) Data augmentation - create variants of training examples (paraphrasing, back-translation). 6) PEFT methods (LoRA with low rank) inherently limit capacity. 7) Mixing general data (10-20%) provides regularization. 8) Reduced training epochs (2-5 instead of 10+). 9) Gradient clipping prevents extreme updates.

Small datasets are common in practice - specialized domains with limited labeled data, new tasks without existing datasets. Overfitting renders fine-tuning useless - model fails on real examples despite good training metrics. Preventing overfitting is often the difference between successful adaptation and wasted effort. The right combination of techniques can enable effective fine-tuning with as few as 100 examples. Without them, even 1000 examples may produce overfit model. Understanding these techniques democratizes fine-tuning for resource-constrained scenarios.

Fine-tuning on 200 examples for sentiment classification in legal domain. Initial attempt: LR 2e-5, 10 epochs, no regularization. Training accuracy 98%, validation accuracy 65% - severe overfitting. Apply techniques: LR 5e-6, LoRA r=4 (limits capacity), weight decay 0.05, early stopping at epoch 4 when validation loss minimal, dropout 0.2, mix 20% general sentiment data. Results: training accuracy 85%, validation accuracy 82% - good generalization. The model learned true sentiment patterns, not memorized specifics. For production, this 82% model is useful; the 65% model is useless. The regularization turned failure into success with same 200 examples.

Direct Preference Optimization (DPO) is a fine-tuning method that aligns language models with human preferences without requiring reinforcement learning. Unlike RLHF which trains a separate reward model then uses PPO for optimization, DPO directly optimizes the policy using a binary cross-entropy objective on preference data, simplifying the pipeline while achieving similar or better results.

DPO starts with preference dataset of (prompt, chosen response, rejected response) triples. It derives a loss function that implicitly optimizes the same objective as RLHF but in closed form: L = -log σ(β log(π_θ(chosen)/π_ref(chosen)) - β log(π_θ(rejected)/π_ref(rejected))), where π_θ is policy, π_ref is reference model (usually SFT model), σ is sigmoid, β controls deviation from reference. This loss increases probability of chosen responses relative to rejected ones while staying close to reference model via KL constraint. Training uses standard supervised learning - no reward model, no RL loop, no PPO complexity.

DPO simplifies alignment dramatically. RLHF requires: 1) collect preference data, 2) train reward model, 3) run PPO (unstable, many hyperparameters, compute-intensive). DPO requires only step 1 and then direct fine-tuning - 2-3× simpler, more stable, and faster. Performance often matches or exceeds RLHF because DPO optimizes the same objective exactly while RLHF approximates it. This has democratized alignment research - teams without RL expertise can now align models. DPO also works better with smaller datasets and is less prone to reward hacking. It's become the default alignment method for many open-source models.

Aligning 7B model for helpfulness. RLHF approach: collect 50k preferences, train reward model (1 day), run PPO (3 days, many tuning attempts) → final model with 70% win rate vs SFT. DPO approach: same 50k preferences, direct fine-tuning (1 day) → final model with 72% win rate. DPO achieved better results in 1/4 the time with less complexity. For practitioner with limited RL expertise, DPO is far more accessible. This is why models like Zephyr, Tulu, and many others use DPO for alignment - it just works better and faster. The simplicity also enables rapid iteration on preference data quality.

The ROI of fine-tuning represents the business value gained from a customized model relative to the costs of development and deployment. For stakeholders, this translates to concrete metrics: cost savings, revenue increases, efficiency gains, or quality improvements that justify the investment in model adaptation over using generic models or existing solutions.

ROI calculation compares fine-tuned model against alternatives (generic LLM, existing system, human labor). Costs include: data collection/curation, compute for training, ongoing inference, and maintenance. Benefits include: reduced human labor (automation), improved accuracy (fewer errors), faster response times (better customer experience), increased conversion (better recommendations), or enabled new capabilities (previously impossible tasks). Break-even analysis shows when cumulative benefits exceed costs. Ongoing value continues as long as model deployed.

Business stakeholders approve investments based on expected returns. Technical metrics (accuracy 92%) don't translate to business value without context. A 5% accuracy improvement might save $1M annually in support costs. Faster inference might increase user engagement by 10%. New capabilities might open million-dollar revenue streams. Without ROI framing, fine-tuning appears as cost center; with ROI, it's value creation. Practitioners must learn to translate technical improvements into business language.

Customer support automation for e-commerce site processing 100k tickets/month. Current: human agents cost $20/ticket = $2M/month. Generic GPT-4 can handle 60% automatically with 85% satisfaction, saving $1.2M/month. Fine-tuning on 10k historical tickets costs $5k data + $2k compute. Fine-tuned model handles 80% automatically with 92% satisfaction, saving $1.6M/month. Additional $400k/month savings vs generic. ROI: monthly gain $400k, one-time cost $7k, payback period <1 day. Over year, $4.8M additional savings. Stakeholder sees: invest $7k, save $4.8M - clear yes. Without ROI framing: 'Fine-tuning improves accuracy 7%' - less compelling. The business case drives decision, not technical metrics.