Design AI evaluation infrastructure.

### Signal to interviewer

I can design evaluation infrastructure that turns model quality into a continuous operational signal, not a one-time benchmark report.

### Clarify

I would clarify quality dimensions, release cadence, domain risk, and who consumes evaluation outputs for decisions.

### Approach

Build an eval fabric with reusable test suites, baseline management, failure taxonomy, and online sampling loops tied to incident workflows.

### Metrics & instrumentation

Primary metric: critical-use-case coverage by automated evaluations. Secondary metrics: eval runtime, detected-regression precision, and human-review agreement. Guardrails: escaped critical failures, stale benchmark usage, and unresolved severe buckets.

### Tradeoffs

Deeper evaluation increases confidence but adds release latency. Lightweight checks are fast but may miss edge-case regressions.

### Risks & mitigations

Risk: benchmark overfitting; mitigate with rotating hidden sets. Risk: evaluation blind spots; mitigate with production sampling. Risk: inconsistent scoring across teams; mitigate with centralized metric definitions.

### Example

For a customer support assistant, every release is scored on policy adherence, hallucination resilience, and escalation correctness before production exposure.

### 90-second version

Create an eval fabric that combines fast gating and deep validation. Use shared metrics, live sampling, and failure taxonomies so release quality stays measurable and actionable.