caio-review

# /cs:caio-review — CAIO Forcing Questions **Command:** `/cs:caio-review <plan>` The eval-demanding CAIO pressure-tests any plan that involves AI. Six questions before any AI feature ships, any multi-year vendor commitment, or any AI team expansion. ## When to Run - Before shipping any new AI-powered feature - Before signing a multi-year AI vendor contract (API or self-hosted infra) - Before EU launch of any AI feature - Before a major AI team hire (especially ML engineer or research scientist) - Before a fine-tuning project commitment - Before adopting AI in a regulated domain (employment, credit, healthcare, education, etc.) - When the founder uses the word "AI" near "competitive advantage" or "moat" ## The Six CAIO Questions ### 1. What does this AI need to be good at, and how would you measure it? **No eval set = no ship.** Before any AI feature deploys, define the eval criteria. - 50-100 representative inputs minimum - Expected outputs OR rubric for grading - Edge cases: ambiguous, adversarial, format-edge - If you can't write down what "good" looks like, you don't have a feature; you have a vibe. ### 2. What's the SLO on hallucination / error rate, and what's the fallback? **Every AI feature has a failure mode. Plan for it.** - Quantified SLO: "<5% hallucination on factual queries" - Detection mechanism: monitoring, sampling, customer feedback loop - Fallback: human-in-loop review, lower-risk default response, refuse-to-answer - Blast radius if SLO breached: how ...