How would you reduce AI latency without hurting quality?

### Signal to interviewer

I can optimize latency with architectural strategy while preserving quality through risk-aware serving decisions.

### Clarify

I would clarify latency targets, workload mix, acceptable quality variance, and cost constraints.

### Approach

Use adaptive depth serving: classify requests, route simple ones to fast paths, and reserve deep pipelines for high-complexity or high-risk cases.

### Metrics & instrumentation

Primary metric: first-token latency percentile. Secondary metrics: end-to-end response time, cache contribution, and route distribution. Guardrails: task success drop, confidence miscalibration, and quality complaints.

### Tradeoffs

Uniform fast paths reduce latency but can underperform on complex requests. Rich pipelines improve quality but increase delay and cost.

### Risks & mitigations

Risk: route misclassification; mitigate with fallback checks. Risk: cache staleness; mitigate with invalidation policies. Risk: hidden quality drift in fast paths; mitigate with cohort-level monitoring.

### Example

In a help assistant, FAQ queries use cached fast responses while troubleshooting requests invoke deeper retrieval and validation pipelines.

### 90-second version

Cut latency with adaptive routing, not blanket simplification. Protect quality by reserving deep inference where needed and tracking latency-quality balance continuously.