calibration-trainer

# Calibration Trainer ## Overview The Calibration Trainer skill provides capabilities for assessing and improving forecaster calibration. It helps decision-makers align their confidence levels with actual accuracy, reducing overconfidence and improving the quality of probabilistic judgments. ## Capabilities - Calibration quiz generation - Confidence interval elicitation - Brier score calculation - Calibration curve plotting - Overconfidence/underconfidence diagnosis - Training exercise management - Progress tracking over time - Benchmark comparison ## Used By Processes - Cognitive Bias Debiasing Process - Decision Quality Assessment - Predictive Analytics Implementation ## Usage ### Calibration Quiz ```python # Generate calibration quiz quiz_config = { "type": "general_knowledge", "format": "confidence_interval", "questions": 20, "confidence_levels": [50, 80, 90], # percentiles to elicit "difficulty": "medium", "domains": ["business", "economics", "technology", "geography"] } # Example question quiz_question = { "id": "Q001", "question": "In what year was Amazon founded?", "actual_answer": 1994, "format": "numeric_interval", "required_responses": [ {"confidence": 50, "prompt": "Give your best estimate"}, {"confidence": 80, "prompt": "Give a range you're 80% confident contains the answer"}, {"confidence": 90, "prompt": "Give a range you're 90% confident contains the answer"} ] } ``` ### Respons...