Architecture: How to build an eval framework for Vitto quality regression ...

Documentation for Sessions

🏗️ Architecture: How to build an eval framework for Vitto quality regression ...

Date: 2026-03-18
Time: 00:15:54 UTC
Session: gspace1-231.8
Agent: backend
Confidence: 91%
Status: ✅ Documented

📋 Decision Context

Question:

How to build an eval framework for Vitto quality regression detection without external tools like Opik?

Task: S231.2 Phase 3.4 — Eval framework (GAP-02)

🎯 Decision Made

Chosen:

Built eval framework inside backend-vitto-api: EvalModule with GoldenDataset schema (30 Q&A pairs across 5 categories), EvalResult schema, EvalRunnerService with @Cron('0 2 * * *') nightly run, Anthropic Haiku claude-haiku-4-5-20251001 as LLM-as-judge scoring accuracy/helpfulness/safety 1-5, 7-day rolling average regression alerts at >10% drop. Manual trigger via POST /eval/run. CRUD endpoints for dataset management.

Options Considered:

Opik (Comet) integration (rejected: external dependency)
Custom eval with GPT-4o judge (rejected: more expensive, Haiku sufficient for scoring)
Manual QA only (rejected: doesn't scale, can't catch regressions overnight)
A/B testing framework (rejected: over-engineering for first eval system)

💭 Reasoning

GAP-02 was critical: no way to detect quality regressions on prompt/routing/memory changes. Every change was a coin flip. The eval framework uses the existing conversational pipeline (engine.startConversationalMessage + getConversationalStream) to test real end-to-end behavior. Haiku was chosen as judge for cost efficiency (~$0.01 per 30-question run). Golden dataset seeded idempotently on startup (seedIfEmpty). Rolling average comparison excludes current run to avoid self-comparison. Fallback scores (3/3/5) when ANTHROPIC_API_KEY not set prevents crashes in dev environments. 90s timeout per question prevents hung evals from blocking the nightly cron.

📁 Files Involved

APIs/backend-vitto-api/src/app/eval/eval-runner.service.ts
APIs/backend-vitto-api/src/app/eval/eval.controller.ts
APIs/backend-vitto-api/src/app/eval/eval.module.ts
APIs/backend-vitto-api/src/app/eval/schemas/golden-dataset.schema.ts
APIs/backend-vitto-api/src/app/eval/schemas/eval-result.schema.ts
APIs/backend-vitto-api/src/app/eval/golden-dataset.seed.ts

🔗 Related

Decision ID: dev_1773792954487_c5q6muc
Session: 231
Workspace: gspace1

Auto-generated by MCP Smart Consolidation