AI Interview Questions

A context window is the maximum number of tokens a language model can process at once, determining how much prior information the model can consider when generating new tokens. It defines the model's working memory - the amount of conversation history, document text, or reasoning context available for each prediction.

The context window is a fixed architectural constraint determined during pretraining. All input tokens (user prompt, system message, conversation history, retrieved documents) plus generated output tokens must fit within this limit. The model's attention mechanism can only access tokens within the window - information beyond is lost. For example, with 8k context, you can process about 6000 words (English). Models use positional encodings (like RoPE) that may allow some extrapolation beyond trained length, but performance degrades.

Context window size fundamentally determines what tasks a model can handle. Small windows (2k-4k) limit applications to short conversations and documents. Large windows (32k-128k) enable processing research papers, legal contracts, and extended multi-turn conversations. Very large windows (1M+) allow book-length understanding and complex agent trajectories. Window size affects both capability and cost - larger windows require more compute (O(n²) attention) and memory. The trend toward larger windows (GPT-4 128k, Claude 200k, Gemini 1M) reflects demand for handling real-world complexity.

User asks model to summarize a 50-page research paper (≈15k tokens). Model with 8k context must truncate to first 8k tokens, missing critical findings from later sections. Summary will be incomplete and potentially wrong. Model with 32k context processes entire paper, attends to conclusion while reading introduction, produces comprehensive summary. This capability difference determines whether model is useful for professional applications. Context window isn't just a spec - it's a capability boundary.

Extending context window length faces fundamental computational challenges due to the quadratic scaling of attention (O(n²) in sequence length) and linear growth of KV cache memory. Doubling context length quadruples compute and doubles memory, making long contexts exponentially more expensive.

The challenges are multi-faceted: 1) Attention compute - full attention matrix of size n×n requires O(n²) FLOPs. For n=128k, this is 16B operations per layer - 100× more than 4k context. 2) Memory for attention scores - storing full matrix requires n² × precision bytes, e.g., 128k² × 2 bytes = 32GB per layer - impossible on current GPUs. 3) KV cache growth - linear in n, for 128k context, 7B model KV cache ≈ 8GB per request, limiting batch size. 4) Training instability - longer sequences harder to optimize. 5) Extrapolation - positional encodings may not generalize beyond trained lengths.

These challenges explain why long-context models are rare and expensive. Without innovations like FlashAttention (reducing memory), sparse attention (reducing compute), and optimized kernels, 1M context would be impossible. The challenges create trade-offs: you can have long context OR high throughput, not both at current hardware. This drives research into efficient attention mechanisms and means practitioners must right-size context to actual needs.

Comparing 4k vs 128k context for 7B model. 4k: attention FLOPs 16M per layer, KV cache 256MB per request, batch size 32 possible. 128k: attention FLOPs 16B per layer (1000× more), KV cache 8GB per request, batch size 1 only. Same model, 32× more context but 1000× more compute and 32× less throughput. This is why production systems often use retrieval (RAG) rather than raw long context - it's more efficient to retrieve relevant chunks than process everything.

RoPE (Rotary Position Embedding) is a position encoding method that represents token positions by rotating the query and key vectors in complex space based on the position index. Unlike absolute position embeddings, RoPE's relative position formulation and its mathematical properties enable better extrapolation to longer sequences than seen during training.

RoPE applies a rotation matrix to query and key vectors that depends on position. For position m, the transformation rotates vector by mθ for each dimension, where θ is a learned or fixed frequency. The dot product between query at position m and key at position n then depends only on relative position (m-n) through the rotation angle difference. This relative position bias is built into the architecture. During training on length L, the model learns attention patterns for relative distances up to L. At inference on longer sequences, the same relative position mechanism continues working for distances beyond L because the rotation formula extrapolates naturally, though performance may degrade.

RoPE enables better length extrapolation than absolute position embeddings. Models using RoPE (GPT-NeoX, PaLM, LLaMA) can often handle 2-4× longer contexts at inference than trained on, with graceful degradation. This is critical for deploying models on longer sequences than pretraining budget allowed. RoPE also provides better theoretical properties for capturing relative position information. Combined with techniques like position interpolation, RoPE-based models can be extended to much longer contexts with minimal fine-tuning.

LLaMA trained on 2k context with RoPE. At inference, prompted with 8k document. Attention scores for pairs 6k apart are computed using same rotation formula as for 2k - the model has never seen such distances but the mechanism still works. Quality degrades but remains usable (perplexity increases from 5 to 7). Without RoPE (absolute embeddings), model would fail completely beyond 2k. This extrapolation capability is why RoPE is standard in modern models - it provides flexibility for longer contexts.

The 'lost in the middle' problem refers to the observation that LLMs tend to disproportionately focus on information at the beginning and end of long contexts, while information in the middle is often ignored or underutilized. This phenomenon significantly degrades performance on tasks requiring access to all parts of the context equally.

Research (Liu et al., 2023) systematically tested models by placing relevant information at different positions in long contexts. Results consistently showed: when relevant info was at start (primacy effect) or end (recency effect), models performed well (80%+ accuracy). When relevant info was in middle third, performance dropped dramatically (50% or less). This occurs because attention patterns are biased - early tokens attend to many positions, late tokens receive strong attention due to recency, middle tokens get diluted. The effect worsens with longer contexts and persists across model families.

The 'lost in the middle' problem undermines the promise of long contexts. If models ignore middle content, you can't trust them with documents requiring holistic understanding. For RAG systems retrieving multiple documents, the middle ones may be ignored even if relevant. For multi-turn conversations, middle turns may be forgotten. Mitigations include: re-ranking retrieved documents to put most relevant at ends, using structured prompting to emphasize middle content, and architectural changes (sliding windows, attention mechanisms) that reduce positional bias. Understanding this phenomenon is crucial for designing reliable long-context applications.

Legal document review with 20 cases. Relevant precedent is case #10 (middle). Model with 80% accuracy when relevant at position 2 or 19 drops to 40% accuracy at position 10. Lawyer relying on model would miss critical precedent 60% of time. This is unacceptable for professional use. Solution: reorder cases by relevance before feeding to model, putting most important at start and end. Or use multi-query approaches that explicitly focus on each section. Without addressing lost-in-middle, long-context models are unreliable for many tasks.

ALiBi (Attention with Linear Biases) is a position encoding method that adds a bias term to attention scores based on the distance between tokens, rather than adding position information to embeddings. This simple approach enables strong extrapolation to much longer sequences than seen during training and improves length generalization.

Instead of adding position embeddings to token embeddings, ALiBi modifies the attention computation directly. For each head i, it adds a bias term -m·|pos_j - pos_k| to the attention score between positions j and k, where m is a head-specific slope. The slopes form a geometric sequence (e.g., 1/2, 1/4, 1/8, ...) giving different heads different sensitivity to distance. This bias penalizes attention between distant tokens, encouraging locality while allowing long-range when needed. During training on length L, the model sees distances up to L. At inference on length L' > L, the bias continues working for larger distances because it's purely distance-based.

ALiBi provides excellent length extrapolation - models can generalize to 10× longer sequences than trained on with minimal degradation. This is far better than absolute embeddings and often better than RoPE. It's also simpler - no position embeddings, no interpolation needed. Models trained with ALiBi (like some GPT variants) can be deployed on much longer contexts than training budget allowed. The bias also naturally encourages locality, which can improve efficiency. However, some tasks requiring precise long-range dependencies may need different handling.

Model trained on 512 token sequences with ALiBi. At inference, given 8k token document (16× longer), performance degrades gracefully. Perplexity increases from 4.5 to 5.2, but model remains usable - it can attend to relevant information across the entire document. Equivalent model with absolute embeddings would fail completely beyond 512. This extrapolation capability enables deployment on longer tasks without retraining, making ALiBi attractive for production where context needs vary.

Sliding window attention restricts each token to attend only to a fixed-size window of nearby tokens (e.g., 512 neighbors on each side), rather than all tokens in the sequence. This reduces attention complexity from O(n²) to O(n×w) where w is window size, enabling efficient processing of arbitrarily long documents while maintaining local context.

For sequence length n and window size w, each token attends to tokens within distance w/2 in both directions. The attention matrix becomes banded (non-zero only near diagonal). Computation and memory scale as O(n×w) instead of O(n²). Information propagates long-range through stacked layers - after L layers, each token's receptive field is L×w. Models like Longformer, BigBird, and Mistral use variants of sliding window attention. Some combine with global tokens (like [CLS]) that attend to everything, enabling both local and global context.

Sliding window attention makes processing very long documents (100k+ tokens) practical on limited hardware. For w=2048, n=100k, complexity 200M vs full attention 10B - 50× reduction. This enables applications like book summarization, long document QA, and genomic sequence analysis that would be impossible with full attention. The trade-off is potential loss of long-range dependencies beyond window size, though stacking layers mitigates this. Models like Mistral use sliding window successfully, achieving strong performance on long-context tasks while maintaining efficiency.

Processing 500-page book (500k tokens) with Mistral's sliding window (w=4096). Each token attends to 4k neighbors, not all 500k. Complexity 500k×4k = 2B per layer vs full attention 250B - 125× reduction. After 32 layers, each token's receptive field covers 128k tokens - enough for book-level understanding. Book summarization works well despite never doing full attention. This efficiency enables deployment on consumer hardware. Without sliding window, same task would require supercomputer.

Context window extension refers to techniques that enable models trained on shorter sequences to handle longer contexts at inference time or through additional training. These methods modify position representations or attention mechanisms to overcome the length limitations of the original training, expanding model capability without full retraining.

Several approaches exist: 1) Position interpolation (PI) - scales position indices to fit within trained range. For RoPE, if trained on L and want to extend to L', map positions i to i×L/L'. This keeps relative distances within trained distribution. 2) NTK-aware scaling - adjusts RoPE frequencies based on neural tangent kernel theory to preserve high-frequency information. 3) YaRN (Yet another RoPE extensioN) - combines PI with temperature tuning and attention scaling. 4) Continued pretraining - further train on long sequences with lower learning rate. 5) Architectural modifications - sparse attention, sliding windows. Each technique offers different trade-offs between extension factor, quality retention, and compute cost.

Context window extension democratizes long-context capabilities. Instead of expensive full retraining on long sequences, models can be adapted with minimal compute (hours vs weeks). LLaMA originally trained on 2k context has been extended to 100k+ using these techniques, enabling long-document tasks. Quality typically degrades gracefully - 32k extension might retain 95% of original performance. This allows practitioners to customize context length to their needs without training from scratch. As models grow, extension techniques become essential for keeping them useful for evolving applications.

LLaMA-2 7B trained on 4k context. Using YaRN with 1000 steps of fine-tuning on 32k sequences (cost $100), model extended to 32k. Passkey retrieval accuracy: 99% at 4k, 95% at 16k, 90% at 32k - usable for many applications. Without extension, fails at 5k. This enables deployment on long documents without retraining full model. For production need of 100k context, could extend further with more fine-tuning. Extension techniques are why open-source models quickly gain long-context variants after release.

Ultra-long context windows (128k-1M tokens) fundamentally change RAG system design by potentially eliminating the need for retrieval in many applications. Instead of chunking documents and retrieving relevant pieces, entire documents or knowledge bases can be placed directly in context, simplifying architecture and potentially improving answer quality.

Traditional RAG: documents chunked (500-1000 tokens), embedded, vector search retrieves top-k chunks, chunks added to context. With ultra-long context: entire documents (50k tokens) or small knowledge bases (100k tokens) fit directly. System design simplifies: no chunking, no vector database, no retrieval logic. Just put relevant documents in context. For larger corpora, hybrid approaches emerge: use retrieval to select top documents, then put full documents in context rather than chunks. This preserves document integrity and cross-chunk reasoning.

Ultra-long context transforms RAG trade-offs. Advantages: no information loss from chunking, better cross-reference reasoning (information spread across document), simpler architecture, lower latency (one pass vs retrieval+generation). Disadvantages: context window becomes bottleneck (limits knowledge base size), quadratic attention cost may exceed retrieval cost, may still need retrieval for very large corpora. For many enterprise applications with document sets <1M tokens (e.g., product manuals, policy documents), retrieval becomes unnecessary. This simplifies deployment and improves quality.

Customer support for software product with 500-page manual (400k tokens). Traditional RAG: chunk into 1000 token pieces (400 chunks), embed, store in vector DB. For each query, retrieve 5 chunks (5000 tokens) + query in context. Complex pipeline, potential missing cross-chunk information. With 1M context model: put entire manual in context once (cached KV), for each query just add question. Model answers using full manual, reasoning across sections. Latency: initial prefill 10s (one-time), then 0.5s per query vs RAG 1s per query. Quality improves (no missed connections). This is why ultra-long context is revolutionary - it makes retrieval optional for many use cases.

Positional interpolation and extrapolation are two approaches for handling sequences longer than training context. Extrapolation uses position encodings beyond trained range directly, relying on model generalization. Interpolation compresses longer sequences to fit within trained range by scaling position indices, keeping relative distances within the distribution seen during training.

Extrapolation: with RoPE or ALiBi, position encodings for indices > L are computed using same formulas as training. The model sees distances it never encountered during training. Some methods (RoPE, ALiBi) extrapolate better than others (absolute embeddings). Performance typically degrades with distance. Interpolation: to extend from L to L', map each position i in L' to i×L/L' in original index space. For example, with L=4k, L'=32k, position 16k maps to 2k in original. All relative distances are scaled down, remaining within trained range. The model sees only distances it knows, but at scaled values. Interpolation usually requires fine-tuning to adapt to scaled distances.

Choice affects extension success. Pure extrapolation works for some methods (ALiBi) but quality degrades. Interpolation with fine-tuning (position interpolation, YaRN) achieves better quality for large extension factors (8-16×). Without fine-tuning, interpolation can confuse model because distances are compressed. With fine-tuning, model adapts to new scale. Understanding difference guides approach: for small extensions (2×), extrapolation may suffice; for large extensions, interpolation+fine-tuning is necessary. Research shows interpolation generally outperforms extrapolation for large factors.

Extending 4k model to 32k (8×). Extrapolation with RoPE: perplexity increases from 5 to 9 at 32k, retrieval accuracy drops from 95% to 60%. Interpolation (position scaling) without fine-tuning: perplexity 12 (worse). Interpolation with 1000 steps fine-tuning: perplexity 5.5, accuracy 92% - much better. The fine-tuning teaches model that distances are now scaled. Pure extrapolation insufficient for large extension; interpolation+fine-tuning preserves quality. This is why YaRN and PI with fine-tuning are standard for major extensions.

The choice between long-context LLMs and RAG depends on document collection size, query patterns, latency requirements, and cost constraints. Long-context models excel when documents fit entirely in context and reasoning requires holistic understanding. RAG scales to arbitrarily large collections but introduces retrieval complexity and potential information loss.

Decision framework: 1) Collection size: if total documents < context window (e.g., 500-page manual), long-context viable. If larger (thousands of documents), RAG required. 2) Query diversity: if queries similar (same document set), caching long-context KV amortizes cost. If queries diverse (different subsets each time), RAG may be more efficient. 3) Reasoning type: cross-document synthesis needs RAG; deep document understanding needs full context. 4) Latency: long-context prefill expensive but per-query cheap; RAG has consistent latency. 5) Cost: break-even analysis based on query volume.

Wrong choice wastes money or degrades quality. Using RAG when documents fit in context adds unnecessary complexity and may miss cross-references. Using long-context for massive collections impossible. Hybrid approaches often optimal: retrieve top documents, then use long-context for full document analysis. As context windows grow (1M+), boundary shifts - more tasks become pure long-context. Understanding trade-offs enables cost-effective architecture decisions.

Three scenarios. A) Company policy manual (200k tokens), 1000 queries/day. Long-context: prefill once (10s), cache KV, each query 0.5s, cost $0.001/query. RAG: chunk, embed, retrieve, generate - 1s per query, cost $0.002/query, plus vector DB maintenance. Long-context wins. B) Legal document repository (10M tokens), 100 queries/day. RAG necessary - doesn't fit context. C) Research papers (50k each), 1000 papers, queries about specific papers. RAG retrieves paper, long-context processes it. Hybrid: retrieval + long-context optimal. Each scenario demands different solution.

Increasing context length has superlinear effects on inference cost and latency due to attention's O(n²) complexity in prefill and O(n) per-token in decode. Longer contexts dramatically increase computational requirements, memory usage, and response times, fundamentally changing deployment economics.

Prefill phase: compute scales as O(n²) with context length n. Doubling context quadruples prefill FLOPs and memory. For 128k context, prefill can take seconds even on fast hardware. Decode phase: each generated token attends to all n cached tokens, so per-token time scales O(n). Generating with 128k context is 32× slower per token than 4k context. KV cache grows linearly with n, reducing batch size proportionally. For 128k, KV cache per request ~8GB for 7B model, limiting concurrency.

These scaling laws constrain practical deployment. A model with 1M context might take minutes to prefill and seconds per token - unusable for interactive applications. Cost per request increases dramatically: 128k context can cost 50-100× more than 4k context. This is why many applications still use RAG - retrieving 5k tokens costs fraction of processing 100k. Understanding these costs guides feature decisions: adding long-context capability increases infrastructure requirements even if rarely used.

Comparing 4k vs 128k for 7B model. 4k: prefill 10ms, generation 20ms per token, batch size 32, $0.001/request. 128k: prefill 320ms, generation 640ms per token, batch size 1, $0.05/request. Same model, 50× cost difference. For application where 95% of queries need 4k but 5% need 128k, must provision for worst case or accept degraded service for long queries. This economic reality shapes product decisions - many services limit context or charge premium for long contexts.

Needle-in-a-haystack evaluation tests a model's ability to retrieve and use a specific piece of information (the needle) buried within a large amount of irrelevant text (the haystack). It's the standard benchmark for long-context understanding, measuring whether models can truly access information anywhere in their context window, not just at the beginning or end.

The test constructs a long document (e.g., 100k tokens) of filler text (e.g., essays, Wikipedia articles). A single factual statement (the needle) like 'The special magic number is 42' is inserted at a specific position. The model is prompted with a question about that fact (e.g., 'What is the special magic number?'). Success requires finding and using the needle despite overwhelming irrelevant context. Testing varies: needle position (early, middle, late), context length, question types. Results reveal position bias (lost-in-middle), effective context length, and retrieval capability.

Needle test exposes whether long context claims are real. A model advertising 1M context that fails to find needles in the middle isn't truly using that context. Results show dramatic variation: some models maintain >95% accuracy across full claimed length; others drop to 50% at half length. This correlates with real tasks like document QA. The test has become industry standard for validating long-context capabilities, used by Anthropic, OpenAI, and Google to demonstrate their models. For practitioners, needle scores predict whether model can handle your long-document tasks.

Testing two 100k-context models. Model A: 98% accuracy at all positions up to 100k. Model B: 98% at start/end, 45% in middle third. Model A truly uses full context; Model B suffers lost-in-middle and is unreliable for documents where information may be anywhere. For legal document review, only Model A acceptable. This is why needle test matters - it separates marketing from capability. When Claude 3 demonstrated 99% accuracy on 200k needle test, it proved real long-context ability.

Despite advances in long-context models, practical limitations remain including computational cost, positional bias, attention dilution, and the fact that many tasks don't actually need all that context. These limitations mean very long contexts (1M+) are not a universal solution and come with significant trade-offs.

Key limitations: 1) Cost - processing 1M tokens costs $50-100 per query at current rates, prohibitive for most applications. 2) Latency - minutes to prefill, seconds per token, unusable for interactive use. 3) Lost-in-middle - models still struggle with mid-context information even at 1M scales. 4) Attention dilution - with millions of tokens, attention spread thin, making it hard to focus on relevant information. 5) Retrieval still needed for larger collections - 1M tokens is only 3-4 books, not enterprise scale. 6) Diminishing returns - many tasks need only recent/relevant context, not entire history.

These limitations mean long context isn't magic bullet. For most applications, RAG with 10k context may outperform raw 1M context at fraction of cost. The optimal solution often combines both: retrieve relevant documents, then use long context for full-document reasoning. Very long context is valuable for specific use cases (book analysis, long videos, scientific papers) but not general replacement for RAG. Practitioners must evaluate whether their tasks truly need all that context or if targeted retrieval suffices.

Customer service with 1-year chat history (500k tokens). Option A: put all history in context - cost $50/query, latency 5 minutes. Option B: retrieve last 10 relevant conversations (10k tokens) using vector search - cost $0.50/query, latency 2 seconds. Option B provides 99% of value at 1% cost. The long-context solution is overkill. This is why production systems don't just throw everything in context - they intelligently select what's needed. Very long context is a tool, not a replacement for good system design.

When documents exceed a model's context window, several strategies exist to process them effectively: truncation (losing information), chunking with summarization, hierarchical processing, retrieval-augmented generation, and sliding window approaches. The choice depends on the task and the importance of complete information.

Common strategies: 1) Truncation - simply take first/last N tokens. Fast but loses information. 2) Chunk and summarize - split document into chunks, summarize each, then process summaries. 3) Hierarchical - chunk, extract key information (entities, facts), process extracted information. 4) RAG - chunk, embed, retrieve relevant chunks for each query. 5) Sliding window - process overlapping windows, aggregate results. 6) Map-reduce - process chunks independently, combine results with another model pass. Each has trade-offs between completeness, accuracy, and cost.

Most real-world documents exceed context windows - books, legal contracts, research papers. Handling them correctly determines application success. Poor handling (simple truncation) can miss critical information. Overly complex handling may be expensive and slow. The right strategy depends on task: summarization needs different approach than QA. For question answering, RAG with chunking works well. For comprehensive analysis, hierarchical processing may be needed. Understanding options enables robust document processing systems.

300-page legal contract (200k tokens) with 32k context model. For specific clause lookup: chunk into 30 chunks, embed, retrieve relevant chunks (RAG) - works well, low cost. For contract summary: chunk and summarize each section (10k tokens summaries), then summarize summaries - hierarchical approach captures full document. For compliance checking requiring full document analysis: sliding window with overlap, aggregate findings. Each task demands different strategy. The art is matching approach to requirements, not forcing one-size-fits-all.

Hierarchical summarization is a technique for processing extremely long documents by recursively summarizing chunks at multiple levels of abstraction. It builds a tree of summaries: leaf nodes are chunk summaries, internal nodes summarize groups of chunks, root provides overall summary. This enables comprehensive document understanding within context constraints.

Process: 1) Split document into chunks that fit context window (e.g., 2k tokens each). 2) Summarize each chunk independently, producing leaf summaries. 3) Group leaf summaries into batches that fit context (e.g., 10 summaries per batch). 4) Summarize each batch, producing level-2 summaries. 5) Repeat until single root summary emerges. Each level preserves key information while compressing. For question answering, can search summary tree: start at root, expand relevant branches. For analysis, can retrieve at appropriate granularity.

Hierarchical summarization enables processing documents millions of tokens long that would never fit in context. Books, technical manuals, legal archives become accessible. The hierarchical structure preserves both high-level themes and specific details through the tree. Compared to single-pass summarization, it maintains more information and enables granular access. It's particularly useful for: comprehensive document analysis, creating searchable knowledge bases from long docs, and situations where different users need different detail levels.

1000-page technical manual (800k tokens) too long for any context window. Hierarchical summarization: chunk into 500 chunks (1600 tokens each), summarize each (200 tokens) = 100k tokens level-1. Batch 10 summaries (2000 tokens) into 50 batches, summarize each (200 tokens) = 10k tokens level-2. Batch 10 level-2 summaries = 1k tokens level-3. Root summary 500 tokens. Now can answer: user asks high-level question → use root summary; needs specific procedure → navigate down tree to relevant section. This makes massive document usable where raw document would be impossible to process directly.

Memory-augmented LLMs incorporate external memory mechanisms that can store and retrieve information beyond the immediate context window, enabling theoretically unlimited context through selective recall. Unlike long-context models that process all tokens uniformly, memory-augmented models maintain a separate memory store and learn to read from it when relevant.

Several architectures exist: 1) Transformer-XL - caches hidden states from previous segments, extending context through recurrence. 2) Memorizing Transformer - adds kNN lookup to attend to similar past keys. 3) RAG itself is a form of memory augmentation using vector databases. 4) Compressive Transformer - compresses old memories into compact representations. 5) MemGPT - manages hierarchical memory with different tiers (working memory, episodic memory, semantic memory). These systems decide what to store, how to index, and when to retrieve, mimicking human memory rather than brute-force attention.

Memory augmentation offers different trade-offs than long context. Long context processes everything but at O(n²) cost. Memory augmentation can handle infinite history at O(1) retrieval cost but may miss information if retrieval fails. It's more scalable for truly long-term interactions (years of conversation) where even 1M context insufficient. Memory systems can also organize information hierarchically, preserving important details while compressing routine information. This approach may be necessary for agents that need persistent memory across sessions.

Personal AI assistant with year-long conversation history (10M tokens). Long-context impossible (would cost $1M/query). Memory-augmented: system stores conversations in vector DB, retrieves relevant past interactions for each query, maintains compressed summaries of long-term patterns. User asks about vacation planning mentioned 6 months ago - system retrieves that memory, provides continuity. Long-context model would have lost it. Memory augmentation enables persistent, evolving AI that remembers you across time, which pure long-context cannot achieve practically.

Document chunking is the process of splitting long documents into smaller segments for embedding and retrieval in RAG systems. Optimal chunking balances competing goals: creating chunks small enough for precise retrieval while preserving enough context to answer questions accurately and maintaining semantic coherence to avoid splitting related information across chunks.

Several chunking strategies exist with different trade-offs. Fixed-size chunking splits documents into equal-length segments (e.g., 500 tokens) regardless of content boundaries - simple but may cut sentences or ideas in half. Recursive chunking uses separators (paragraph breaks, section headers, sentence boundaries) to create semantically coherent chunks while respecting size limits. Semantic chunking uses embeddings or models to detect topic boundaries, splitting when content shifts. Overlapping chunks (10-20% overlap) ensures that information near boundaries isn't lost. Document structure awareness preserves section hierarchies (e.g., keeping sections together, maintaining header context). Chunk size selection involves testing different sizes (300-1500 tokens) on your specific data to find the sweet spot between precision and context preservation.

Chunking quality directly determines RAG success. Poor chunking leads to: 1) Lost context - related information split across chunks, neither chunk contains full answer. 2) Boundary problems - answers cut off mid-sentence. 3) Irrelevant chunks - too-large chunks contain distracting information. 4) Missed retrievals - relevant chunks not found because they're too small to match query semantics. 5) Context window waste - too many small chunks consume limited space. Research shows optimal chunking can improve retrieval accuracy by 10-20% over naive approaches. For production RAG, chunking strategy is as important as embedding model choice.

Processing a research paper with two chunking strategies. Naive fixed-size 500-token chunks: splits the methods section across 3 chunks, losing methodology coherence. When user asks about experimental setup, retrieval may find only one chunk, missing critical details. Semantic chunking with structure awareness: keeps the entire methods section as one chunk (800 tokens). Retrieval finds complete methodology, enabling accurate answers. Overlap 10% ensures transitions between sections preserve context. Chunk size tuned to 800 tokens based on testing shows optimal precision/recall balance for academic content. The result: 92% answer accuracy vs 78% with naive chunking. This is why thoughtful chunking is essential for RAG performance.

Attention sparsity reduces computational cost by limiting each token to attend to only a subset of positions, based on patterns that approximate full attention while exploiting the observation that most attention weights are near-zero. This transforms O(n²) complexity to O(n log n) or O(n√n), making long contexts feasible.

Various sparsity patterns: 1) Local sparsity - attend only to nearby tokens (sliding window). 2) Strided sparsity - attend to every k tokens for long-range coverage. 3) Dilated sparsity - increasing gaps with distance. 4) Block sparsity - attend within and between fixed-size blocks. 5) Learnable sparsity - model learns which tokens to attend to (Reformer, Sinkhorn attention). 6) Combination patterns (BigBird, Longformer) mix local, global, and random attention. These patterns are implemented via block-sparse matrix multiplication kernels that skip zero computations.

Sparsity is essential for scaling to ultra-long contexts. Full attention for 1M tokens requires 1T operations - impossible. With 99% sparsity (attending to 10k tokens each), complexity drops to 10B - feasible on modern hardware. Models like BigBird achieve 95% of full attention quality on long-document tasks while using 90% less compute. Sparsity enables processing book-length documents that would otherwise require supercomputers. As context lengths grow to millions, sparsity becomes not optional but mandatory.

Processing 1M token book with full attention: 1e12 FLOPs per layer, >1TB memory - requires HPC cluster. With sliding window (w=4096): 4e9 FLOPs per layer - 250× reduction, fits on single GPU. With BigBird's sparse pattern (local+global+random): 5e9 FLOPs, similar reduction. Quality within 2-3% of full attention on book QA. This makes long-document processing practical for production. Without sparsity, long-context models would remain research curiosities, not deployable systems.

Designing a system for 500-page documents requires choosing among strategies based on task requirements: comprehensive understanding, question answering, summarization, or fact extraction. The optimal approach combines chunking, hierarchical processing, retrieval, and multiple model passes to handle the scale within context constraints.

Step 1: Document parsing - extract text, tables, structure (headings). Step 2: Chunking - split into 2k token chunks with overlap, preserving section boundaries. Step 3: For question answering - embed chunks, build search index. For each query, retrieve top-k chunks, put in context. Step 4: For summarization - hierarchical approach: summarize chunks, then summaries of summaries. Step 5: For comprehensive analysis - extract entities, relationships, key claims from each chunk, build knowledge graph. Step 6: For specific tasks (compliance check) - create query-specific retrieval, maybe multiple passes. Step 7: Result synthesis - combine partial results with final model pass.

A 500-page document (≈1M tokens) exceeds any model's practical context for interactive use. Even with 1M context models, cost and latency prohibitive. A well-designed system achieves 90%+ of ideal quality at 1-10% cost by intelligent processing. The design must match task: summarization needs different approach than fact-checking. Poor design may miss critical information or waste resources. Understanding document structure (chapters, sections) enables smarter chunking.

500-page technical manual for compliance checking. System: 1) Parse into 500 chunks by section. 2) Extract all regulatory requirements mentioned (using model per chunk) → 2000 requirements. 3) Build requirement database. 4) For compliance query about specific regulation, search requirements, retrieve relevant sections with context. 5) Generate answer based on retrieved sections. This processes entire document once (extraction) then answers queries cheaply. Alternative: for each query, search chunks (RAG) - also works but misses cross-chapter patterns. The extraction approach enables deeper analysis at slightly higher upfront cost. Choice depends on query patterns.

Chunk size in RAG involves fundamental trade-offs between retrieval precision, context utilization, and cross-chunk reasoning. Larger chunks preserve more context but may contain irrelevant information diluting relevance; smaller chunks are more precise but may split related content and increase retrieval complexity.

Trade-off dimensions: 1) Retrieval precision: small chunks (200 tokens) highly focused, high precision. Large chunks (2000 tokens) may contain relevant info but also noise, lowering precision. 2) Context utilization: small chunks need more retrieved to cover topic, consuming context window. Large chunks use context efficiently. 3) Cross-chunk reasoning: related information split across small chunks may be missed. Large chunks preserve relationships. 4) Embedding quality: small chunks may lack context for good embeddings. 5) Storage/index size: more chunks = larger index, slower retrieval. 6) Latency: retrieving many small chunks takes more time.

Chunk size choice can swing RAG performance by 10-20%. Too small: miss relationships, retrieve too many chunks, context overflow. Too large: low precision, irrelevant info distracts model. Optimal size depends on content type (narrative vs factual), query patterns (specific vs broad), and model context window. Testing different sizes on your data is essential. Common sizes range 500-1500 tokens, with overlap 10-20% to preserve boundaries.

Legal contract RAG with two chunk sizes. 300-token chunks: retrieval precision 0.85, recall 0.70, need 8 chunks (2400 tokens) per query, answers accurate 80%. 1000-token chunks: precision 0.70, recall 0.85, need 3 chunks (3000 tokens), answers accurate 82%. Larger chunks slightly better due to cross-clause relationships preserved. For technical documentation with discrete facts: 300-token chunks achieve 90% accuracy vs 85% for large chunks - smaller better. There's no universal best - must test on your data. This is why chunk size experimentation is RAG best practice.