Architecture: How should skills improve over time based on real outcomes i...

Documentation for Sessions

🏗️ Architecture: How should skills improve over time based on real outcomes i...

Date: 2026-03-19
Time: 14:21:00 UTC
Session: gspace1-238.9
Agent: angular
Confidence: 92%
Status: ✅ Documented

📋 Decision Context

Question:

How should skills improve over time based on real outcomes instead of accumulating endlessly?

Task: Not specified

🎯 Decision Made

Chosen:

3-tier confidence system (experimental → validated → trusted) with outcome-based graduation, warm-start protection for new skills, and weighted retrieval. Skills graduate at >=80% success after 10 uses (validated), >=90% after 25 (trusted), demote at <30% after 5. New skills get 14-day warm start where they're weighted equally with validated to prevent cold-start starvation.

Options Considered:

  1. No confidence system (status quo)
  2. Binary success/fail without tiers
  3. 3-tier with warm start and weighted retrieval (chosen)
  4. ML-based adaptive scoring (over-engineering)

💭 Reasoning

Root cause: Skills accumulated with no quality signal — Vitto didn't know which skills actually helped. The outcome loop existed for tasks (OutcomeFeedbackService) but not for skills. The existing recordOutcome() always received success=true, masking failures. The 3-tier system is deliberately simple — no ML, no complex scoring. Graduation thresholds are conservative (10+ uses for validated, 25+ for trusted) to avoid premature promotion. The cold-start problem was identified during design review: new skills would stay experimental, get deprioritized, never used, never graduate. The 14-day warm start breaks this cycle by giving new skills equal weighting during their trial period. Weighted retrieval (trusted=1.0, validated=0.7, experimental=0.4) ensures battle-tested skills get priority.

📁 Files Involved

🔗 Related

Auto-generated by MCP Smart Consolidation