clickhouse-incident-runbook

# ClickHouse Incident Runbook ## Overview Step-by-step procedures for triaging and resolving ClickHouse incidents using built-in system tables and SQL commands. ## Severity Levels | Level | Definition | Response | Examples | |-------|------------|----------|----------| | P1 | ClickHouse unreachable / all queries failing | < 15 min | Server down, OOM, disk full | | P2 | Degraded performance / partial failures | < 1 hour | Slow queries, merge backlog | | P3 | Minor impact / non-critical errors | < 4 hours | Single table issue, warnings | | P4 | No user impact | Next business day | Monitoring gaps, optimization | ## Quick Triage (Run First) ```bash # 1. Is ClickHouse alive? curl -sf 'http://localhost:8123/ping' && echo "UP" || echo "DOWN" # 2. Can it answer a query? curl -sf 'http://localhost:8123/?query=SELECT+1' && echo "OK" || echo "QUERY FAILED" # 3. Check ClickHouse Cloud status curl -sf 'https://status.clickhouse.cloud' | head -5 ``` ```sql -- 4. Server health snapshot (run if server responds) SELECT version() AS version, formatReadableTimeDelta(uptime()) AS uptime, (SELECT count() FROM system.processes) AS running_queries, (SELECT value FROM system.metrics WHERE metric = 'MemoryTracking') AS memory_bytes, (SELECT count() FROM system.merges) AS active_merges; -- 5. Recent errors SELECT event_time, exception_code, exception, substring(query, 1, 200) AS q FROM system.query_log WHERE type = 'ExceptionWhileProcessi...