triage-calibration

Solid

Daily triage accuracy calibration — use during scheduled calibration runs to verify triage classification accuracy against few-shot examples and adjust confidence thresholds

AI & Automation 96 stars 22 forks Updated today MIT

Install

View on GitHub

Quality Score: 82/100

Stars 20%
66
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
86
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Triage Calibration ## Purpose Review recent triage decisions against actual outcomes. Adjust signal weights and depth thresholds to improve classification accuracy over time. ## When to Use - Scheduled daily (end of day or low-activity period). - After a significant misclassification is identified. - After a new signal collector is added or modified. ## Workflow 1. **Collect recent triage results** — Pull the last 24h of awareness ticks with their depth classifications and signal readings. 2. **Match against outcomes** — For each tick, determine what actually happened: - Did the classified depth match the actual work required? - Were any signals misleadingly high or low? 3. **Compute accuracy metrics** — - Classification accuracy (correct depth / total ticks) - Over-triage rate (classified higher than needed) - Under-triage rate (classified lower than needed) 4. **Identify systematic errors** — Are certain signal types consistently miscalibrated? Are certain time periods problematic? 5. **Propose adjustments** — Suggest weight or threshold changes. Do NOT auto-apply in V3 (fixed weights). Log recommendations for user review. 6. **Write calibration report**. ## Output Format ```yaml date: <YYYY-MM-DD> period: <start> to <end> total_ticks: <n> accuracy: <percentage> over_triage_rate: <percentage> under_triage_rate: <percentage> systematic_errors: - signal: <signal name> bias: over | under magnitude: <how far off> recommendations: - ...

Details

Author
WingedGuardian
Repository
WingedGuardian/GENesis-AGI
Created
5 months ago
Last Updated
today
Language
Python
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

triage-dailies

Drive the dailies morning triage ritual over an overnight batch of AI-generated video takes. Check judge health, open with the failure dossier and cost, review audited regen takes first, look at multi-frame evidence behind every doubtful verdict, record the human's rulings as gold labels, surface judge disagreement patterns, recalibrate when the gold set can carry it, and close with a plan for tonight. Use when the user says /morning-triage, "morning triage", "triage the overnight batch", "review last night's takes", "what happened overnight", or "run dailies on the batch".

0 Updated 2 weeks ago
zhang-liz
AI & Automation Listed

feedback-triage

Triage a tool's accumulated dogfooding feedback reports into a leverage-ordered improvement backlog — reconcile what already shipped, cluster findings by underlying cause rather than symptom, assign each cluster a disposition (attack this tool, route out to the tool that owns it, or decline), apply a promotion gate (reinforced across reports, specific, actionable), and emit a triage document with a status-tracked promotion table. Use on "triage the feedback backlog", "cluster the feedback reports", "what should this tool fix next", "promote the recurring feedback", or "/feedback-triage". Explicitly invoked maintenance — never run proactively; it reads a whole corpus. If the tool's binding registers its own triage template (e.g. keel's reflection-triage), follow that template. Not for consolidating journal entries into guidance (that is consolidate-knowledge), not for a corpus of one report with no prior triage (nothing to cluster yet — a 1-report delta over an existing baseline IS a valid later pass), not for

2 Updated 5 days ago
grimaldost
AI & Automation Listed

triage-followups

After a run completes, walk the GitHub issues that run auto-filed (review non-blocking findings + improvement ideas from the evidence-out seam, AND any issue a task filed for work it deliberately cut) ONE AT A TIME with the human — explaining each from its source finding/rationale and the code it points at — and decide keep / close / promote / edit. The human triage gate the auto-filing seams deliberately do not have. Runs after any completed run (batch or single-task); re-runnable later.

1 Updated 6 days ago
cperler