AI Interview Questions

A document ingestion pipeline for RAG transforms raw documents (PDFs, Word files, web pages) into searchable chunks with metadata, ready for embedding and indexing. It's the foundation of any RAG system, and its quality directly determines what information can be retrieved.

Main steps: 1) Document loading - extract text from various formats using appropriate parsers (PyMuPDF for PDFs, python-docx for Word, BeautifulSoup for HTML). 2) Text extraction - clean extracted text, remove headers/footers, handle columns. 3) Document splitting/chunking - divide into manageable pieces (500-1500 tokens) with overlap, respecting document structure. 4) Metadata extraction - capture source, date, author, section headings, page numbers. 5) Embedding - generate vector representations for each chunk. 6) Indexing - store vectors and metadata in vector database with appropriate index (HNSW, IVF). 7) Quality checks - validate chunk quality, remove near-duplicates. 8) Versioning - track document versions for updates.

Garbage in, garbage out. Poor ingestion leads to retrieval failures: text extraction errors lose information, bad chunking splits related content, missing metadata prevents filtering. A well-designed pipeline ensures all document information is accessible to retrieval. For production systems, ingestion must be reliable, scalable, and maintainable.

Ingesting 10,000 PDF research papers. Pipeline: PyMuPDF extracts text (handles complex layouts). Recursive chunking by section (preserves structure). Extracts metadata: title, authors, year, journal. Embeds with scientific embedding model. Indexes in Qdrant with metadata filters. Result: queries can search by year, author, and concept. Without proper extraction, tables and figures might be missed; without metadata, cannot filter by year. Good pipeline makes all information available.

PDF parsing tools extract text and structure from PDF files, each with different strengths: PyMuPDF (fitz) is fast and handles many PDFs well, pdfplumber excels at table extraction, and Unstructured.io provides a comprehensive pipeline for complex documents with layout analysis.

PyMuPDF: Python bindings for MuPDF, extracts text, images, and metadata. Fast, handles most PDFs, good for text-heavy documents. pdfplumber: Built on pdfminer, focuses on detailed layout analysis, table extraction, and character-level information. Slower but better for complex layouts. Unstructured.io: Comprehensive toolkit that partitions documents, detects elements (titles, lists, tables), and can use OCR for scanned PDFs. Provides a pipeline from raw PDF to cleaned chunks. Also offers API service. Choice depends on document complexity: simple text PDFs → PyMuPDF; complex tables → pdfplumber; mixed, scanned, or highly formatted → Unstructured.

PDF is the most common document format but notoriously difficult to parse. Wrong tool leads to: missing text (especially in columns), garbled tables, lost formatting, or failed extraction entirely. For RAG, this means information is unavailable. Choosing right tool(s) and combining them (e.g., PyMuPDF for text, pdfplumber for tables) ensures maximum extraction quality.

Processing annual report PDF with financial tables. PyMuPDF extracts narrative text well but tables become garbled. pdfplumber extracts tables accurately as structured data. Unstructured.io pipeline: uses OCR for scanned pages, detects table regions, extracts both text and tables, outputs in markdown format. For this document, best approach: use Unstructured.io for complete extraction, or combine: PyMuPDF for text, pdfplumber for tables, then merge. Result: all information available for retrieval, including table data that would otherwise be lost.

OCR (Optical Character Recognition) is technology that converts images of text (scanned documents, screenshots, photos) into machine-readable text. It's needed when documents are not born-digital but are images, such as scanned PDFs, photographed documents, or faxes.

OCR process: 1) Image preprocessing - deskew, denoise, binarize to enhance text. 2) Layout analysis - detect text regions, columns, tables. 3) Character recognition - identify individual characters using ML models (Tesseract, Google Cloud Vision, Azure OCR). 4) Post-processing - correct common errors using dictionaries, language models. 5) Output - generate text with position information (bounding boxes). Modern OCR engines can handle multiple languages, handwriting (limited), and complex layouts. Accuracy varies from 95%+ for clean scans to much lower for poor quality.

Many important documents exist only as scans: historical records, handwritten notes, old books, physical documents. Without OCR, this information is inaccessible to RAG systems. For enterprise, contracts, invoices, and forms often arrive as scans. OCR unlocks this data. However, OCR errors propagate to retrieval - if a word is misread, it won't be found. Quality matters: for critical documents, consider human verification or higher-quality OCR services.

Law firm digitizing 10,000 old contracts (scanned PDFs). Without OCR, no text searchable. With Tesseract OCR, accuracy ~90% - most contracts searchable but some terms may be misread. For high-stakes litigation, they use Google Cloud OCR (99% accuracy) at higher cost, ensuring critical clauses are correctly extracted. The processed text goes into RAG system; now lawyers can search across all contracts for specific clauses. OCR transformed dead documents into searchable knowledge.

Handling tables in PDFs for RAG requires extracting structured data while preserving relationships between rows, columns, and headers. Simply extracting raw text loses tabular structure, making table information inaccessible for semantic search and question answering.

Approaches: 1) Table extraction tools - pdfplumber, camelot, tabula extract tables as structured data (DataFrames, JSON). 2) Convert to text representation - serialize tables to text: 'Table: Product Sales. Row1: iPhone, 2023, $100M. Row2: iPad, 2023, $50M.' 3) Markdown/HTML - represent as markdown tables for embedding. 4) Summarization - use LLM to generate natural language summary of table. 5) Hybrid - store both structured representation and text summary. For retrieval, table chunks need embedding; serialized text works with standard embedding models. For Q&A, may need to retrieve table then use with LLM that can understand structured data.

Tables contain dense, structured information that text extraction loses. A query like 'What were iPhone sales in 2023?' requires understanding row/column relationships. If table becomes raw text 'iPhone 2023 $100M', retrieval may find it but can't associate 'iPhone' with sales. Proper table handling preserves these relationships, enabling accurate answers. In many documents, key information lives in tables.

Financial report with table: | Product | Year | Revenue | | iPhone | 2023 | $100M | | iPad | 2023 | $50M |. Good handling: serialize as 'Table: Product Revenue. iPhone 2023 revenue $100M. iPad 2023 revenue $50M.' Embed this text. Query 'iPhone 2023 revenue' matches directly. Better: store as structured JSON, retrieve, and use with LLM that can reason over tables. Without table handling, extraction might yield 'iPhone 2023 $100M' without context, or miss table entirely. User query fails.

Preserving document structure during ingestion means capturing hierarchical information like headings, sections, lists, and their relationships. This structure provides context for chunks and enables more intelligent retrieval (e.g., retrieving entire sections when needed).

Methods: 1) Layout analysis - tools like Unstructured.io, Adobe Extract, or LayoutLM detect document elements and their roles (heading, paragraph, list). 2) Font/size analysis - larger/bold text often indicates headings. 3) Markdown conversion - convert to markdown with # for headings, preserving hierarchy. 4) Metadata tagging - store element type and level with each chunk. 5) Parent-child relationships - link chunks to their section headers. 6) Hierarchical chunking - create chunks that respect section boundaries, optionally with overlapping hierarchies.

Structure provides context. A chunk about 'warranty' from a 'Product Specifications' section is different from same text in 'Return Policy' section. Without structure, this context is lost. For retrieval, knowing section helps disambiguate. For generation, providing section header with chunk improves answer accuracy. Structure also enables section-level retrieval: for overview questions, retrieve entire section rather than fragments.

User manual with sections: '1. Safety Instructions', '2. Installation', '3. Troubleshooting'. Query about 'safety during installation' needs information from both sections. With structure preserved, chunks from section 2 include header 'Installation', and we know section 1 exists. Retrieval can find both. Without structure, chunks are just text; might find installation steps but miss safety context. Generation might give incomplete answer. Structure enables better retrieval and synthesis.

Unstructured.io is an open-source library and platform designed to preprocess raw, unstructured documents (PDFs, HTML, Word, images) into structured formats suitable for LLM applications. It solves the 'messy document' problem by providing a unified pipeline for parsing, cleaning, partitioning, and chunking diverse document types.

Unstructured provides: 1) Partitioning - auto-detects document type and applies appropriate parser. 2) Element detection - identifies document elements (titles, narrative text, lists, tables, figures) using layout models. 3) Cleaning - removes boilerplate, headers/footers, page numbers. 4) Chunking - intelligent chunking that respects element boundaries. 5) Serialization - outputs to formats ready for embedding (JSON, text). Supports OCR for scanned docs. Can run locally or use hosted API. Handles over 20 file types with consistent interface.

Building document pipelines manually is tedious and error-prone. Each file type needs different parsers, each has quirks. Unstructured abstracts this complexity, providing a single API that works across formats. It's become the standard for RAG ingestion because it handles the chaos of real-world documents. The library is used by major RAG frameworks (LlamaIndex, LangChain) as the default document loader.

Ingestion pipeline for 50,000 mixed documents: PDFs (some scanned), Word docs, PowerPoints, emails. Without Unstructured, need separate tools for each, custom code to unify output. With Unstructured: one function partition processes all, returning consistent elements with metadata. Chunk with chunk_by_title preserving structure. Output ready for embedding. Development time reduced from weeks to days. This is why Unstructured is ubiquitous in RAG systems.

Multi-modal documents contain both text and images (diagrams, charts, photos). Handling them for RAG requires extracting information from images to make it searchable, either via captioning (convert image to text) or using multi-modal embeddings that can directly compare images and text.

Approaches: 1) Image captioning - use vision-language models (BLIP, GPT-4V) to generate text descriptions of images. Store captions as text chunks. 2) OCR for text-in-images - extract text from screenshots, scanned documents, or images containing text. 3) Multi-modal embeddings - use models like CLIP that embed images and text in shared space. Store image embeddings; queries can retrieve images directly. 4) Figure/table extraction - specialized tools for extracting data from charts. 5) Hybrid - store both image (for potential multi-modal generation) and caption (for text retrieval).

Many documents convey critical information visually: architecture diagrams, scientific figures, product photos, infographics. Text-only RAG misses this information entirely. A query about 'system architecture' can't find a diagram if only captioning used. Multi-modal handling ensures all document content is accessible. As multi-modal LLMs advance, storing images themselves enables richer answers.

Technical manual with diagram of product assembly. Text-only: misses diagram. Captioning: 'Diagram showing step-by-step assembly of product X with parts labeled A, B, C.' Now retrievable for queries about assembly. User asks 'How do I assemble product X?' retrieves caption, system can answer. Even better: store image embedding; user could upload photo of their disassembled product, find similar diagrams. Multi-modal handling turns images from silent to searchable.

Document layout analysis is the process of identifying and classifying different regions in a document image (text blocks, headings, tables, figures, lists) and understanding their reading order. It's essential for converting scanned or image-based documents into structured text suitable for RAG.

Layout analysis models: 1) Traditional - rule-based using heuristics about text blocks. 2) LayoutLM family (LayoutLMv3) - transformer models pre-trained on document images that understand both text and layout. 3) Detectron2 - Facebook's object detection framework can be trained for layout detection. 4) YOLO-based - object detection models fine-tuned on document layout datasets (PubLayNet, DocLayNet). 5) Unstructured.io - uses combination of models for layout detection. Process: detect regions, classify them (title, text, table, figure), determine reading order (usually top-left to bottom-right, but can be complex with columns).

Without layout analysis, scanned documents become a stream of text in wrong order (e.g., columns read across). Multi-column documents become garbled. Headings lose their role. Tables become unreadable. Layout analysis restores structure, enabling proper chunking and retrieval. For RAG on scanned documents, it's essential.

Two-column research paper scanned. Without layout analysis, text extracted reads across columns: line1 colA + line1 colB, etc., producing nonsense. With layout analysis, detects two columns, reads colA top to bottom, then colB, preserving meaning. Also detects title, headings, abstract as separate elements. Now chunks can respect section boundaries. Retrieval works because text is coherent. Layout analysis turns unusable scan into structured document.

Processing large batches of documents (millions) efficiently requires parallelization, incremental processing, and careful resource management. The pipeline must scale horizontally while maintaining reliability and handling failures gracefully.

Strategies: 1) Distributed processing - use Spark, Ray, or Dask to parallelize across many workers. Split documents into partitions, process in parallel. 2) Batch processing - process documents in batches, not one-by-one, to amortize overhead. 3) Incremental processing - only process new/changed documents, not full corpus each time. 4) Queue-based - use message queues (SQS, RabbitMQ) to distribute work and handle backpressure. 5) Checkpointing - save progress to resume after failures. 6) Resource optimization - use spot instances for cost, monitor memory usage (PDF parsing can be memory-intensive). 7) Caching - cache parsed results to avoid reprocessing.

At scale, sequential processing becomes impossible. Processing 10M documents sequentially at 1 second each takes 115 days. With 100 parallel workers, 1 day. Efficient batch processing is essential for keeping knowledge bases current. It also reduces cost through better resource utilization and spot instances.

Legal firm processing 5M documents. Sequential: 5M × 2 seconds = 10M seconds ≈ 115 days - impossible. Implement Spark pipeline with 200 cores: 115 days / 200 = 0.6 days - feasible. Use incremental processing: daily updates of 10k new docs take minutes. Queue-based with SQS ensures reliability; failed documents retried. Cost: spot instances reduce cloud bill 70%. This architecture makes large-scale ingestion practical.

Document versioning in RAG tracks changes to documents over time and updates the vector index accordingly. When documents are updated or new versions released, the system must reflect these changes while maintaining consistency and avoiding stale information.

Approaches: 1) Replace strategy - on update, delete old chunks, embed new version, insert. Simple but may cause temporary unavailability. 2) Versioned chunks - store version metadata; retrieval can filter to latest version. 3) Incremental updates - process only changed documents, not full corpus. 4) Soft deletion - mark old chunks as inactive, filter them out during search. 5) Point-in-time queries - support querying as of specific version (for audit). 6) Change detection - monitor document sources for changes (file modification time, database triggers). 7) Batch updates - run nightly updates to refresh changed documents.

Stale information is dangerous. Policy documents change, product specs update, knowledge evolves. If RAG uses old versions, answers become wrong, eroding trust. Versioning ensures users get current information. For regulated industries, auditability of which version was used is required. Proper update handling maintains freshness without downtime.

HR policy document updated. Old version: 'vacation days: 15'. New: 'vacation days: 20'. Without update, RAG answers 15 - wrong. With versioning: system detects change, reprocesses document, updates index. Now answers 20. For audit, can store version IDs with each query log. For compliance, point-in-time queries can reconstruct what information was available when. Versioning turns static knowledge base into living system.

Metadata is structured information about document chunks that enables filtering, provenance, and context. Good metadata dramatically improves retrieval by allowing queries to restrict by source, date, author, or type, and provides essential context for generation (e.g., citing sources).

Key metadata fields: 1) Document-level: title, author, date, source (file path/URL), document type (policy, manual, FAQ), version, access permissions. 2) Chunk-level: chunk index, parent section, headings hierarchy, page numbers, character offsets. 3) Provenance: document ID, chunk ID, embedding model version, ingestion timestamp. 4) Domain-specific: for legal: case number, jurisdiction; for medical: patient ID, study ID; for products: SKU, category. 5) Security: access control lists, sensitivity labels.

Metadata enables precise retrieval: 'Find documents about return policy from 2024' requires date filter. 'Show me product specs for iPhone' needs category filter. For security, access control prevents unauthorized document access. For generation, metadata provides citations: 'According to the 2024 policy document...' Without metadata, retrieval is just semantic similarity; with metadata, it's targeted and trustworthy.

Customer support RAG with metadata per chunk: source='policy_2024.pdf', section='Returns', date='2024-01-15', product_category='electronics', access_level='public'. Query: 'What's the return policy for electronics?' retrieves chunks with product_category='electronics'. Filter by date='2024' ensures latest policy. Generation cites 'According to the 2024 Returns Policy'. User can verify. Without metadata, might retrieve 2023 policy or non-electronics info, causing wrong answer. Metadata transforms vague retrieval into precise, trustworthy system.

Duplicate and near-duplicate documents (same content, different versions, or slightly modified copies) waste storage, increase retrieval noise, and can bias results. Detecting and handling them is essential for clean knowledge bases and fair retrieval.

Approaches: 1) Exact deduplication - hash document content, remove identical copies. 2) Near-duplicate detection - use MinHash or SimHash to identify documents with high similarity (e.g., >90% Jaccard similarity). 3) Semantic deduplication - use embeddings to detect semantically similar documents; may be too aggressive for some use cases. 4) Version tracking - if duplicates are versions, keep newest, archive old. 5) Document-level deduplication before chunking - most efficient. 6) Chunk-level deduplication - if same text appears in multiple documents (e.g., boilerplate), can remove duplicate chunks. 7) De-duplication strategy - decide whether to keep one copy or keep all with weights.

Duplicates waste storage (2× cost) and skew retrieval. If the same document appears 10 times, it will dominate search results, pushing out diverse content. For RAG, this means answers may over-represent duplicated information. Near-duplicates (e.g., slightly different versions) cause similar issues. Deduplication ensures fair representation and efficient storage.

Company knowledge base has 100 copies of the same FAQ page from different years (slightly updated). Without deduplication, retrieval for common queries returns all 100, swamping results. With near-duplicate detection, identifies these as versions, keeps only latest, removes others. Storage reduced by 99%, retrieval now returns diverse documents from different topics. Query about return policy gets policy docs, not 100 copies of same FAQ. Deduplication improves both efficiency and quality.

LlamaParse is a managed document parsing service from LlamaIndex designed specifically for RAG applications. It handles complex documents (PDFs with tables, images, complex layouts) and outputs markdown-formatted text ready for chunking, with superior accuracy on tables and layout preservation.

LlamaParse uses a combination of OCR, layout analysis, and LLM-based post-processing. Features: 1) High-accuracy table extraction - preserves tabular structure in markdown. 2) Layout preservation - maintains headings, lists, reading order. 3) Multi-modal support - extracts text from images within documents. 4) API-based - simple integration. 5) Custom instructions - can specify how to handle certain elements. 6) Output formats - markdown, JSON, text. 7) Handles complex PDFs - scanned, multi-column, mixed layouts. Pricing is usage-based.

Many RAG implementations struggle with complex PDFs: tables become garbled, multi-column text reads across, images ignored. LlamaParse solves these with production-grade parsing. It's particularly valuable for enterprise documents (financial reports, contracts, research papers) where accuracy matters. The markdown output works seamlessly with chunking and embedding.

Processing 10-K financial filing (complex document with tables, multi-column, fine print). Open-source parsers produce messy output with tables as raw text, losing structure. LlamaParse outputs clean markdown with tables preserved as markdown tables, headings with #, proper reading order. Chunking now preserves table data. Query 'What were R&D expenses in 2023?' finds the table cell, answers correctly. Without LlamaParse, would miss or misread table data. The improved parsing directly impacts answer quality.

Poorly scanned documents (faded, skewed, low resolution, with noise) require enhanced preprocessing before OCR and extraction. Without treatment, text quality suffers, leading to missing or incorrect information in RAG.

Techniques: 1) Image preprocessing - deskew (correct rotation), denoise (remove speckles), binarize (convert to black/white), enhance contrast. 2) Advanced OCR engines - Google Cloud Vision, Azure OCR handle poor quality better than open-source Tesseract. 3) Multiple OCR passes - combine results from different engines. 4) LLM-based correction - use LLM to fix obvious OCR errors based on context. 5) Confidence thresholds - discard low-confidence OCR results. 6) Human verification - for critical documents, manual review. 7) Fallback to original image - if text can't be extracted, store image and use multi-modal model for Q&A.

Many valuable historical documents, old records, or poorly scanned materials are low quality. If OCR fails, information is lost. In enterprise, acquisition documents, legacy contracts, and physical records often fall in this category. Proper handling can recover usable text, unlocking previously inaccessible knowledge.

Historical company records from 1950s, microfilm scans, low quality. Raw OCR yields 60% accuracy - many words wrong, sentences garbled. Pipeline: deskew, denoise, use Google Cloud Vision (better for poor quality), then run LLM correction: 'Fix OCR errors in this text based on context.' Accuracy improves to 90%. Now searchable for historical research. Without treatment, documents remain unusable. For critical contracts, also have human verify key clauses. Good handling turns garbage into gold.

Rule-based parsing uses hand-crafted rules (regex, pattern matching, layout heuristics) to extract information, while ML-based parsing uses models trained on labeled examples. Each has strengths and weaknesses in accuracy, adaptability, and development cost.

Rule-based: define rules for headers (e.g., all caps, font size >14), tables (detect consistent spacing), extraction patterns. Fast, interpretable, works well for consistent formats. But brittle - breaks when format varies. ML-based: train models (LayoutLM, YOLO) on labeled document datasets. Can generalize across formats, handle variation. Requires labeled data and compute for training. Slower inference but more robust. Hybrid approaches common: ML for layout detection, rules for specific fields.

Choice depends on document variety. For fixed-format documents (same template every time), rule-based is simpler, faster, cheaper. For diverse documents (many layouts, sources), ML-based essential. Many real-world collections are diverse - annual reports from different companies, each with unique formatting. ML handles this variation; rules would require endless exceptions.

Processing invoices from 10 different vendors. Each has different layout: vendor A has date top-right, vendor B date bottom-left. Rule-based: need 10 different rule sets, maintenance nightmare. ML-based: train model on 1000 labeled invoices from all vendors, learns to find date regardless of position. Works for new vendor without new rules. Trade-off: initial investment in labeling (1000 invoices) vs ongoing maintenance of 10+ rule sets. ML wins for diversity. For single-template forms, rules win.

Monitoring document ingestion is critical because failures mean documents aren't available for retrieval. A robust monitoring system tracks success rates, detects errors, and alerts when ingestion pipelines fail or degrade.

Components: 1) Success metrics - track documents processed successfully, chunks created, embedding jobs completed. 2) Error tracking - categorize errors (parse failures, OCR failures, timeouts, embedding failures). 3) Alerting - set thresholds: if success rate <95% over 1 hour, alert. 4) Dead letter queue - store failed documents for retry/inspection. 5) Dashboard - visualize ingestion throughput, error rates, latency. 6) Logging - detailed logs for debugging. 7) Health checks - regular test documents to ensure pipeline working. 8) Version tracking - know which documents are in which index version.

Undetected ingestion failures lead to missing documents and stale information. Users may query for documents that failed to index, get no results, and assume information doesn't exist. For critical systems, this is unacceptable. Monitoring ensures issues are caught quickly, often before users notice. For regulated industries, audit trails of ingestion are required.

📋 Example: Daily ingestion of 10,000 new support articles. Monitoring dashboard shows success rate 98% - good. Alert triggers when rate drops to 85%: investigation reveals new PDF format causing parse failures. Team fixes parser, reprocesses failed documents. Without monitoring, would have missing articles for days, users frustrated. Also track per-source errors: notices that vendor X's documents fail 20% of time - escalate to vendor. Monitoring turns firefighting into proactive improvement.

Certain file formats present unique challenges for extraction: scanned PDFs (need OCR), image-heavy documents (multi-modal), complex layouts (multi-column, tables), encrypted files, and proprietary formats. Each requires specialized handling to extract usable text.

Challenging formats: 1) Scanned PDFs - no text layer, require OCR which can introduce errors. 2) Image-heavy documents - brochures, magazines, slides - need layout analysis and potentially multi-modal handling. 3) Complex PDFs with forms, annotations, or digital signatures - extraction may miss filled-in fields. 4) Encrypted/protected files - need decryption, may have restrictions. 5) Proprietary formats - old WordPerfect, Lotus Notes, etc. - need format converters. 6) Emails with attachments - need to extract both email body and attachments, preserve threading. 7) Web pages with dynamic content - need JavaScript rendering.

Real-world document collections contain these challenging formats. Ignoring them means missing information. Underestimating difficulty leads to poor extraction and failed retrieval. Knowing which formats are problematic helps allocate engineering effort appropriately and set expectations about extraction quality.

Corporate archive includes 30-year-old WordPerfect files, scanned contracts, and emails with attachments. WordPerfect: need converter, may lose formatting. Scanned contracts: OCR accuracy 95%, good enough. Emails: need to extract body and attachments as separate documents with relationship preserved. Without handling, all this information would be lost. With proper pipeline, becomes searchable. The effort for challenging formats is higher but unlocks valuable data.

Handling access control and PII (Personally Identifiable Information) during ingestion ensures that sensitive information is either redacted before indexing or properly secured so that only authorized users can access it. This is critical for compliance and data privacy.

Access control: 1) Metadata-based - store access control lists (ACLs) with each chunk. 2) Pre-filtering - during retrieval, apply user's permissions as metadata filters. 3) Separate indexes - different sensitivity levels in different indexes. PII redaction: 1) Detection - use NER models or regex to identify PII (names, SSNs, emails, addresses). 2) Redaction - replace with placeholders ([REDACTED]) or mask (e.g., 'John' → '[NAME]'). 3) Encryption - encrypt sensitive fields, decrypt only for authorized users. 4) Exclusion - for highly sensitive documents, don't index at all. 5) Audit logging - track access to sensitive documents.

Data privacy regulations (GDPR, CCPA, HIPAA) require protecting personal information. Breaches can lead to fines and reputational damage. Access control prevents unauthorized document access. For RAG systems handling customer data, employee records, or sensitive business information, these are not optional - they're legal requirements.

Healthcare RAG with patient records. Pipeline: detects PII (patient names, medical record numbers) and redacts them before embedding, storing original mapping separately. Access control metadata: each chunk tagged with patient ID and clinician access list. Query from Dr. Smith: metadata filter ensures only her patients' documents retrieved. Audit log tracks all access. Complies with HIPAA. Without these, system would be illegal to deploy. Proper handling enables safe use of sensitive data.

Designing for 10M documents requires a distributed, fault-tolerant architecture that can process at scale while maintaining reliability and cost efficiency. The pipeline must handle varied document types, manage resources, and support incremental updates.

Architecture components: 1) Document storage - cloud object storage (S3, Blob) for raw files. 2) Queue-based ingestion - new documents trigger messages in queue (SQS). 3) Worker pool - auto-scaling workers (AWS Lambda, Kubernetes) pull from queue, process documents. 4) Distributed processing - Spark or Ray for large batch jobs. 5) Checkpointing - store processing state to resume after failures. 6) Metadata database - track document status, versions. 7) Vector database - scalable (Qdrant, Pinecone) with sharding. 8) Monitoring - CloudWatch, Prometheus for metrics. 9) Cost optimization - spot instances, right-sized workers.

At 10M scale, naive approaches fail. Sequential processing takes months. Single-node processing runs out of memory. Costs spiral without optimization. A well-designed pipeline processes 10M documents in days, not months, at reasonable cost, and scales to handle growth. It's the difference between a toy and a production system.

10M documents, average 10 chunks each = 100M vectors. Architecture: S3 for storage, SQS for work queue, 100 Kubernetes workers (auto-scaling). Each worker processes 1000 docs/hour → 100k docs/hour total → 100 hours for full corpus. Daily updates of 50k docs take 30 minutes. Vector DB sharded across 10 nodes for fast query. Cost: $5000 for initial processing, $500/month ongoing. Monitoring alerts on failures. This scales to 100M documents with more nodes. Without this design, impossible.

A document processing SLA (Service Level Agreement) defines commitments for how quickly and reliably documents will be ingested and available for querying. It includes metrics for freshness, success rate, and latency, with associated consequences for breaches.

Typical SLA metrics: 1) Time-to-index - 95% of documents indexed within 24 hours of receipt. 2) Success rate - 99% of documents processed successfully. 3) Freshness - indexed documents reflect source within 1 hour for critical docs. 4) Accuracy - extraction accuracy >= 95% on sampled documents. Tracking: 1) Timestamp each document from receipt to indexing completion. 2) Log all failures with reasons. 3) Sample processed documents for quality checks. 4) Dashboard showing SLA compliance over time. 5) Alerts when approaching breach.

SLAs set expectations with stakeholders and drive operational discipline. If business users expect documents available within hours, pipeline must deliver. Without SLAs, priorities unclear: should you optimize for speed or accuracy? Breaches trigger investigation and improvement. For regulated industries, SLAs may be contractual.

Legal document processing SLA: '99% of documents indexed within 4 hours, 99.5% success rate, extraction accuracy 98% for key fields.' Dashboard shows: last week 99.2% within 4 hours (met), success rate 99.6% (met), accuracy 97.5% (miss). Investigation reveals new document type causing lower accuracy. Team adds specialized parser. Accuracy improves to 98.2%. SLA tracking caught degradation before client complaints. Without SLA, might not have noticed until renewal time.