AI Interview Questions

LLMOps (Large Language Model Operations) is the set of practices, tools, and processes for managing the lifecycle of LLM applications in production. While building on traditional MLOps, LLMOps addresses unique challenges: prompt management, versioning, cost optimization, latency, and the non-deterministic nature of LLMs.

Traditional MLOps focuses on model training, deployment, and monitoring for predictive models. LLMOps extends this with: 1) Prompt management - versioning, testing, and optimizing prompts across model versions. 2) Cost tracking - LLM API costs can be significant and need monitoring per user/feature. 3) Latency optimization - techniques like caching, streaming, and model selection. 4) Evaluation - LLM outputs are non-deterministic, requiring different evaluation approaches (LLM-as-judge, human feedback). 5) Safety and security - prompt injection, output validation. 6) RAG infrastructure - managing vector databases, retrieval pipelines.

LLM applications have different operational characteristics than traditional ML. They're API-first, have unpredictable costs, and require continuous prompt engineering. Without LLMOps practices, teams struggle with reliability, cost overruns, and slow iteration. As LLM adoption grows, LLMOps becomes essential for production success.

Team deploys chatbot with traditional MLOps: they monitor model accuracy but not token costs. After a month, they get a $50,000 bill due to unexpected usage. LLMOps would have included cost monitoring, per-user limits, and caching to control expenses. Another difference: when model version updates, prompts may need adjustment - LLMOps includes prompt regression testing.

The LLM application lifecycle encompasses all stages from ideation to production and ongoing maintenance. It includes prompt development, integration, testing, deployment, monitoring, and continuous improvement.

Key stages: 1) Ideation and prototyping - define use case, experiment with prompts, choose model. 2) Development - build application logic, integrate with tools/APIs, implement RAG if needed. 3) Prompt engineering - iterate on prompts, version them, create test cases. 4) Evaluation - test on golden dataset, human evaluation, safety checks. 5) Deployment - choose deployment strategy (API, self-hosted), set up infrastructure, implement caching and rate limiting. 6) Monitoring - track costs, latency, user feedback, error rates. 7) Continuous improvement - analyze failures, update prompts, fine-tune if needed, A/B test changes. 8) Retirement - deprecate old versions, migrate users.

Understanding the lifecycle helps teams plan resources and avoid common pitfalls. Each stage has specific tools and practices. Skipping evaluation leads to poor quality; skipping monitoring leads to cost surprises. A structured lifecycle ensures reliable, maintainable LLM applications.

Travel agent app lifecycle: Prototyping (test with 50 queries), Development (integrate flight/hotel APIs), Prompt engineering (version 1.0 prompts), Evaluation (90% success on test set), Deployment (to staging, then canary to 5% users), Monitoring (track cost per booking, success rate), Improvement (add new prompts for edge cases). This systematic approach catches issues early and ensures quality.

Prompt versioning is the practice of tracking changes to prompts over time, treating them as code artifacts with versions, commit messages, and rollback capabilities. It's essential because prompt changes directly impact application behavior and need management like any code change.

Implementation: 1) Store prompts in version control (Git) alongside code. 2) Use semantic versioning for prompts (e.g., v1.0.0). 3) Link prompts to specific model versions (prompt-v1.2 works with GPT-4, not GPT-3.5). 4) Include metadata: author, date, purpose, test results. 5) Use prompt management tools (LangSmith, PromptHub) for visualization and collaboration. 6) Automate testing of prompt versions against golden dataset. 7) Support canary deployments of new prompts.

Prompts are code. They determine application behavior. Without versioning, you can't: 1) Roll back if a prompt change causes issues. 2) Know which prompt version was used for a given user interaction (debugging). 3) Collaborate effectively across team members. 4) A/B test prompt variations. 5) Reproduce results. Prompt versioning brings software engineering best practices to prompt engineering.

Team updates prompt to improve booking accuracy. After deployment, user complaints increase. With versioning, they quickly roll back to previous prompt version, restoring service. Without versioning, they'd have to guess what changed or redeploy old code. Later, they analyze the failed prompt, fix, and deploy as v1.3 with confidence. Versioning saved the day.

LLM observability tools provide visibility into LLM application performance, costs, latency, and quality. They capture traces, metrics, and feedback, helping teams debug issues, optimize costs, and improve quality.

LangSmith: comprehensive platform from LangChain for tracing, debugging, evaluating, and monitoring LLM applications. Provides run tracing, feedback collection, dataset management, and A/B testing. Langfuse: open-source observability with tracing, cost tracking, and prompt versioning. Good for self-hosted. Helicone: focused on cost and usage analytics, caching, and rate limiting. Lightweight, easy to integrate. Arize: ML observability platform with LLM-specific features for performance monitoring, drift detection, and hallucination tracking. Each provides: 1) Trace logs of LLM calls with timestamps, tokens, latency. 2) Cost aggregation per user/model/feature. 3) Quality metrics via feedback or automated evaluation. 4) Alerting on anomalies.

LLM applications are complex, with many moving parts. When something goes wrong (high latency, errors, bad answers), you need to know why. Observability tools provide the data. They also help optimize costs by identifying expensive patterns. For production, observability is essential.

Travel agent sees cost spike. Langfuse dashboard shows one user making 1000 calls with long prompts - possible extraction attack. Team implements rate limiting. Another example: LangSmith trace shows a step taking 5 seconds due to slow API call - optimize with timeout and fallback. Without observability, blind to these issues.

Production LLM metrics span multiple dimensions: quality, cost, latency, safety, and user experience. Tracking these provides visibility into application health and guides optimization.

Key metrics: 1) Quality metrics - success rate (task completion), answer correctness (via golden set), faithfulness (grounding in context), user satisfaction (ratings). 2) Cost metrics - cost per request, per user, per feature; token usage (input vs output); cost by model. 3) Latency metrics - end-to-end latency, time-to-first-token, per-step latency, p95/p99. 4) Reliability metrics - error rate (API failures, timeouts), availability. 5) Safety metrics - policy violation rate, toxicity score, PII leakage incidents. 6) Business metrics - conversion rate, support ticket deflection, user retention. 7) Operational metrics - cache hit rate, rate limit hits, queue length.

Without metrics, you're flying blind. A sudden cost spike goes unnoticed until bill arrives. Quality degradation goes unaddressed until users complain. Metrics enable proactive management and data-driven optimization. For production, define SLOs for key metrics and alert when breached.

Customer support agent dashboard shows: success rate 92% (SLO 90% met), cost per session $0.15 (budget $0.20), p95 latency 2.1s (SLO 2.5s). All green. But safety metrics show 0.5% policy violations - investigate. Turns out agent occasionally gives wrong refund info. Fix prompt. Metrics caught issue before major impact.

Model drift in LLM applications refers to degradation in model performance over time due to changes in the underlying model, user behavior, or knowledge base. Unlike traditional ML drift (data distribution shift), LLM drift can be subtle and harder to detect.

Types of drift: 1) Model version drift - provider updates model (e.g., GPT-4 to GPT-4-turbo) with different behavior. Same prompt may produce different results. 2) Concept drift - user queries change over time (new products, new terminology). 3) Knowledge drift - model's knowledge becomes outdated (e.g., events after training cutoff). 4) Performance drift - model may become slower or more expensive. 5) Safety drift - safety alignment may degrade with updates. Detection requires continuous evaluation on golden datasets and monitoring of user feedback.

Drift can silently degrade user experience. A model that worked perfectly at launch may become unreliable months later. Without detection, you won't know why users are complaining. Mitigations: regular evaluation on golden set, A/B testing new model versions, and maintaining fallback options.

📋 Example: Travel agent using GPT-4 works well. OpenAI releases GPT-4-turbo with different prompt sensitivity. Same prompts now produce more hallucinated flight details. Golden dataset evaluation shows accuracy drop from 94% to 87%. Drift detected. Team adjusts prompts for new model or sticks with old version temporarily. Without monitoring, would have deployed worse experience unknowingly.

A/B testing for LLMs involves comparing two versions (prompts, models, or configurations) by splitting traffic and measuring key metrics to determine which performs better. It enables data-driven decisions for improvements.

Process: 1) Define hypothesis - e.g., 'New prompt will increase booking success rate by 5%'. 2) Select metrics - success rate, cost, latency, user satisfaction. 3) Split traffic - randomly assign users or requests to control (A) and treatment (B). 4) Run test with sufficient sample size (power analysis). 5) Collect data - ensure consistent measurement across variants. 6) Statistical analysis - compute significance, effect size. 7) Check guardrails - ensure new version doesn't harm safety or increase cost unacceptably. 8) Decision - deploy if wins, iterate if not. Tools: LangSmith supports A/B testing, custom implementations with feature flags.

Intuition about prompt improvements is often wrong. A/B testing provides empirical evidence, preventing deployment of changes that degrade quality or increase cost. It also quantifies business impact, justifying investment.

Travel agent A/B test: Control (current prompt) vs Treatment (new prompt with more detailed instructions). 10,000 users each, run for 1 week. Results: Treatment success rate 94% vs Control 92% (p<0.01), cost per booking same. Treatment wins, deploy to all. Without A/B test, might have assumed new prompt better or worse based on intuition. Data proves it.

Shadow deployment runs a new model version in parallel with the production model, serving the same traffic but without showing results to users. It allows evaluation of the new model's performance on real data without risk.

Implementation: 1) Production model handles user requests normally. 2) Shadow model receives copy of each request, processes it, but responses are discarded or logged for analysis. 3) Compare shadow outputs with production outputs on metrics: quality (via LLM-as-judge), cost, latency. 4) Analyze differences, identify edge cases where shadow performs better or worse. 5) Use insights to improve shadow before deployment. Shadow deployment is safe because users never see shadow responses. It's especially useful when evaluating models with different pricing or latency characteristics.

Shadow deployment reveals how a new model will perform on your actual traffic without risking user experience. It catches issues that offline testing might miss: unexpected behavior on long-tail queries, cost surprises, latency impacts. It's a key step before canary deployment.

Team considers switching from GPT-4 to GPT-4-mini for cost savings. Shadow deploy for 1 week: mini processes all traffic in background. Analysis shows mini 95% as good as GPT-4 on success rate, but 70% cheaper. However, on complex queries, mini fails more often. Team decides to use mini for simple queries, GPT-4 for complex (hybrid). Shadow provided data for this decision without risking users.

Managing LLM costs is critical because API expenses can scale rapidly with usage. Effective cost management combines monitoring, optimization strategies, and architectural choices to balance quality and expense.

Strategies: 1) Caching - store frequent query results to avoid repeated API calls. 2) Model selection - use smaller/cheaper models for simple tasks, larger only when needed. 3) Prompt optimization - shorten prompts, use fewer examples to reduce token count. 4) Context management - limit retrieved chunks, use compression to reduce tokens. 5) Rate limiting - cap usage per user to prevent abuse. 6) Batching - combine multiple requests where possible. 7) Fallback models - try cheaper model first, escalate if needed. 8) Cost monitoring - track cost per user, per feature, alert on spikes. 9) Budget alerts - notify when approaching limits.

Unmanaged LLM costs can bankrupt a startup or blow through enterprise budgets. A single misconfigured agent could cost thousands per day. Proactive cost management ensures predictable expenses and maximizes ROI. It's as important as quality management.

Customer support app with 1M queries/month. Without optimization: GPT-4 only, average 1000 tokens/query = $20,000/month. With optimization: cache 30% queries (saves $6k), use GPT-3.5 for 50% of queries (saves $7k), prompt compression reduces tokens 20% (saves $4k). Total $3k/month, 85% savings. Optimization made app viable.

Token budget management involves setting limits on token usage per user, session, or time period to control costs and prevent abuse. It's essential for production LLM applications where costs scale with token consumption.

Implementation: 1) Per-user budgets - e.g., 100K tokens per day per user. Track usage in database. When exceeded, block or throttle. 2) Per-request limits - max tokens per request to prevent excessively long generations. 3) Tiered budgets - free tier (limited), paid tier (higher). 4) Real-time tracking - increment counter as tokens used, check before each API call. 5) Alerting - notify user when approaching limit, admin on abuse. 6) Cost allocation - attribute costs to users/features for internal billing. 7) Budget recovery - reset daily/monthly. Implement via middleware that counts tokens before/after API calls.

Without token budgets, a single user could consume unlimited resources, causing cost spikes. A malicious user could exhaust your budget with automated queries. Budgets ensure fair usage and cost predictability. For SaaS products, they enable usage-based pricing.

SaaS product offers 10K free tokens/month per user. User signs up, uses 5K tokens in first week. System tracks, shows remaining. If user exceeds, blocks further requests until next month or prompts to upgrade. Another user attempts to script 100K queries - budget blocks after 10K, preventing $500 cost. Token budget management essential.

Model deprecation occurs when an LLM provider announces that a model version will be discontinued, requiring applications to migrate to newer versions. This can cause unexpected behavior changes and requires careful management.

Process: 1) Monitor provider announcements - subscribe to updates, track deprecation schedules. 2) Assess impact - test new model on golden dataset, compare performance. 3) Prompt adaptation - new models may need prompt adjustments. 4) Shadow deployment - run new model in parallel to validate on real traffic. 5) Gradual migration - canary deployment, monitor metrics. 6) Update documentation and prompts. 7) Communicate to users if behavior changes significantly. 8) Have fallback plan - if new model fails, ability to roll back to old (if still available) or alternative provider.

Unexpected model deprecation can break production applications. Without preparation, you might have hours to migrate, risking downtime. Proactive management ensures smooth transitions and maintains service quality.

📋 Example: OpenAI announces GPT-3.5-turbo deprecation in 3 months. Team immediately tests new recommended model on golden dataset: accuracy drops 5%. They spend 2 months optimizing prompts, regain accuracy. Shadow deploy for 2 weeks, confirm. Migrate smoothly before deadline. Without preparation, would have scrambled at last minute, possibly degrading user experience.

A fallback strategy ensures application availability when the primary LLM provider experiences outages or rate limiting. It may involve secondary providers, cached responses, degraded mode, or graceful error messages.

Strategies: 1) Multiple providers - configure secondary provider (e.g., Anthropic as backup for OpenAI). On failure, route to backup. 2) Cached responses - for common queries, serve cached results. 3) Degraded mode - use simpler model or rule-based system. 4) Queue requests - store requests and retry later. 5) Graceful error - inform user of temporary issue, offer email notification when resolved. 6) Local fallback - for critical functions, have a small local model that can handle basics. Implement with circuit breaker pattern: detect failures, open circuit, use fallback, periodically test primary.

LLM providers can and do have outages. Without fallback, your application goes down too, frustrating users and potentially losing revenue. Fallback strategies maintain availability, building trust and resilience.

Customer support app using OpenAI. OpenAI has 30-minute outage. With fallback: traffic automatically routed to Anthropic Claude. Users unaffected. Without fallback: all requests fail, users see errors, support tickets flood. Fallback saved the day. For non-critical apps, graceful error message might suffice, but for critical apps, multi-provider fallback essential.

CI/CD for LLM applications adapts traditional software engineering practices to the unique needs of LLMs: testing prompts, evaluating model outputs, and managing prompt versions as code.

Key practices: 1) Version control for prompts - store prompts in Git with code. 2) Automated testing - run prompts against golden dataset on each PR, measure success rate. 3) Evaluation gates - PR cannot merge if accuracy drops below threshold. 4) Integration tests - test full application flow with mocked LLM responses. 5) Canary deployments - gradually roll out new prompts/models to small user segment. 6) Monitoring integration - deploy with observability, track metrics. 7) Rollback automation - ability to revert prompt changes quickly. 8) Environment parity - test in staging with same model versions as production.

LLM applications are software. They deserve same engineering rigor. CI/CD catches prompt regressions before they reach production, ensures reliability, and enables confident, frequent updates.

Developer submits PR changing prompt for travel agent. CI runs golden dataset: success rate drops from 92% to 88% (below 90% threshold). PR blocked, developer fixes. Without CI, bad prompt would deploy, degrading user experience. CI/CD prevents this.

Logging and tracing for LLM calls capture detailed information about each request: prompt, response, tokens, latency, timestamps, and metadata. This data is essential for debugging, optimization, and compliance.

Implementation: 1) Structured logging - log each LLM call as JSON with fields: timestamp, user_id, session_id, model, prompt, response, tokens (prompt, completion, total), latency, error (if any). 2) Tracing - capture chain of calls for multi-step agents (thought, action, observation). Tools like LangSmith, Langfuse provide automatic tracing. 3) Store in central system - Elasticsearch, database, or observability platform. 4) Retention policy - balance debugging needs with storage costs (e.g., 30 days online, longer in cold storage). 5) Privacy - redact PII before logging. 6) Sampling - for high volume, log only a percentage or errors.

When something goes wrong, logs are your first resort. They tell you exactly what happened: what prompt caused error, how long it took, how many tokens used. Without logs, debugging is guesswork. Tracing is essential for complex agent flows.

User reports bad answer. Developer searches logs for that user's session, sees exact prompt and response. Notices model hallucinated due to ambiguous query. Fixes prompt. Without logs, couldn't reproduce or fix. Another: cost spike traced to user with unusually long prompts - logs reveal extraction attempt, implement rate limiting.

Caching stores responses for frequent queries so they can be reused without calling the LLM API. Since many user queries are similar or identical, caching can dramatically reduce costs and latency.

Implementation: 1) Cache key - typically the prompt (or normalized version) plus model and parameters. 2) Cache store - Redis, Memcached, or database with TTL. 3) Cache hit - if prompt found, return stored response immediately. 4) Cache miss - call LLM, store response with TTL. 5) Considerations: semantic caching - group similar prompts (not just identical) using embeddings; cache invalidation - when knowledge changes, clear relevant cache; privacy - don't cache sensitive queries. Cache hit rates of 30-50% common, saving significant costs.

LLM APIs cost per token. Caching eliminates costs for repeated queries. It also reduces latency (cache hit is milliseconds vs seconds). For high-traffic applications, caching can save millions annually.

Customer support FAQ: many users ask same questions. Without cache: 10,000 identical queries/day × 500 tokens × $0.01/1K = $50/day. With cache: first query costs $0.05, cached, subsequent 9,999 cost $0. Total daily cost $0.05 vs $50. 1000x savings. Even with semantic caching for similar but not identical questions, savings substantial. Caching is first line of cost optimization.

Handling PII (Personally Identifiable Information) in LLM systems requires measures to prevent data leakage, ensure compliance (GDPR, HIPAA), and protect user privacy. This spans data ingestion, processing, and output.

Strategies: 1) Data minimization - only collect necessary data, don't log full conversations unnecessarily. 2) PII detection - use NER models or regex to identify PII in inputs and outputs. 3) Redaction - replace PII with placeholders ([NAME], [EMAIL]) before sending to LLM. 4) Encryption - encrypt stored data, use TLS for transmission. 5) Access controls - limit who can view logs containing PII. 6) Retention limits - delete logs after necessary period. 7) Model selection - use on-premise or private deployments for sensitive data. 8) Auditing - track access to sensitive data. 9) User consent - obtain permission before storing personal data.

PII leaks can cause legal liability, regulatory fines, and reputational damage. In healthcare, HIPAA violations costly. In finance, customer data protection required. For global apps, GDPR compliance mandatory. Proper PII handling is not optional.

Healthcare chatbot receives patient message: 'I'm John Smith, DOB 1/1/1980, with diabetes.' System detects PII, redacts before sending to LLM: 'I'm [NAME], DOB [DATE], with [CONDITION].' LLM responds, system reinserts placeholder for personalized response but doesn't log raw PII. Logs stored encrypted with access limited. Compliant with HIPAA.

An LLM gateway is a middleware layer that sits between your application and LLM providers, handling common concerns: routing, caching, rate limiting, authentication, logging, and failover. It centralizes LLM management.

Gateway features: 1) Provider abstraction - single API for multiple providers (OpenAI, Anthropic, etc.). 2) Routing - direct requests to appropriate model based on rules. 3) Caching - store responses, serve cache hits. 4) Rate limiting - per-user, per-API key limits. 5) Authentication - validate API keys, manage usage. 6) Logging and monitoring - unified logs across providers. 7) Cost tracking - aggregate costs. 8) Failover - if primary provider fails, route to backup. 9) Prompt management - version prompts centrally. Implemented via open-source (LiteLLM, Gateway) or cloud services.

Direct integration with multiple providers leads to duplicated code, inconsistent handling, and management overhead. A gateway centralizes concerns, reducing complexity and improving reliability. It also enables vendor independence - switch providers with config change, not code change.

Company uses OpenAI, Anthropic, and self-hosted models. Without gateway: each integration custom-coded, different error handling, separate logging. With gateway: all requests go through common interface. When OpenAI has outage, gateway automatically routes to Anthropic. Caching saves costs. Central dashboard shows costs across all providers. Gateway simplifies everything.

Managing API keys and secrets for multiple LLM providers involves secure storage, rotation, and access control. Poor key management can lead to security breaches and unexpected costs.

Best practices: 1) Never hardcode keys in code or config files. 2) Use secrets management services (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault). 3) Environment variables for local development, but not for production. 4) Key rotation - regularly rotate keys, automate if possible. 5) Least privilege - use keys with minimal permissions (e.g., read-only if possible). 6) Per-service keys - separate keys for different applications/users for cost tracking and revocation. 7) Monitoring - alert on unusual key usage. 8) Access control - restrict who can view keys. 9) Audit logging - track key access.

Leaked keys can lead to unauthorized usage, costing thousands. A single exposed key in GitHub could be mined by bots within minutes. Proper management prevents this. Also, when an employee leaves, you can revoke their key access without affecting others.

Developer accidentally commits OpenAI key to public GitHub. Within hours, bot discovers, uses key for crypto mining prompts. Company gets $10,000 bill. With proper secrets management, key never in code, breach prevented. Also, with separate keys per developer, can revoke individual key without impacting production.

SLOs (Service Level Objectives) and SLAs (Service Level Agreements) for LLM features define expected performance: uptime, latency, quality, and accuracy. They set user expectations and guide engineering priorities.

Typical SLOs: 1) Availability - 99.9% uptime for API endpoints. 2) Latency - p95 response time < 2 seconds for simple queries, < 5 seconds for complex. 3) Success rate - 95% of requests complete without error. 4) Quality - 90% of responses rated acceptable by users or automated evaluation. 5) Cost - cost per request under budget. SLAs may include financial penalties for breaches. Important: LLM quality is harder to guarantee than traditional metrics - responses may be correct but not helpful. Define quality SLOs carefully, often using human evaluation or LLM-as-judge.

SLOs align team on what matters. Without them, you might optimize for latency while quality suffers. For customers, SLAs provide confidence. Setting realistic SLOs acknowledges LLM limitations while ensuring acceptable experience.

Travel agent SLOs: Uptime 99.5%, p95 latency 3s, booking success rate 92%, user satisfaction 4.5/5. Dashboard tracks these. When latency exceeds 3s for 2 hours, alert triggers. Team investigates and fixes. Quarterly review shows success rate 93% - exceeding SLO. SLOs drive improvement and set boundaries.

Setting up a production LLM system for high traffic requires scalable architecture, cost controls, monitoring, and reliability mechanisms. It must handle millions of requests while maintaining performance and budget.

Architecture components: 1) Gateway layer - routes requests, handles rate limiting, caching, failover. 2) Caching layer - Redis for frequent query caching, semantic caching for similar queries. 3) Model routing - tiered models: small/fast for simple queries, large for complex. 4) Asynchronous processing - for non-real-time tasks, use queues. 5) Observability - comprehensive logging, metrics, tracing (LangSmith, Datadog). 6) Cost controls - per-user budgets, alerts. 7) Scaling - auto-scaling based on queue depth. 8) Fallbacks - multiple providers, degraded mode. 9) Security - input sanitization, output validation, PII redaction. 10) CI/CD - automated testing, canary deployments.

High-traffic applications amplify any weakness. A small percentage of errors becomes thousands of frustrated users. Cost overruns become millions. Robust architecture ensures reliability, cost control, and scalability.

Consumer travel app with 1M daily users. Architecture: Edge gateway with rate limiting (100 requests/user/day). Redis cache for popular destinations (60% hit rate). Model tier: 70% simple queries go to fast 7B model, 30% complex to 70B model. Queue for batch processing (itinerary generation). Multi-provider fallback (OpenAI primary, Anthropic backup). Monitoring alerts if cost per user exceeds $0.01. This scales to millions while controlling costs and ensuring reliability.