chaos-and-resilience

Solid

Chaos engineering, resilience patterns, failure recovery, and fault tolerance for the {{PROJECT_NAME}}. Covers circuit breaker patterns (HTTP, WS, per-venue), reconnect gate design, graceful shutdown protocol, backpressure strategies (PubSub 1MB/4MB, persistence queues), worker lifecycle management, IPC channel health detection, failure scenario matrix, bounded queue design, and event loop stall recovery. Use when reviewing or writing any code that touches error handling, retry logic, reconnection, circuit breakers, worker management, IPC channels, queue management, backpressure, graceful shutdown, health checks, or any failure-adjacent code path.

AI & Automation 3 stars 0 forks Updated today MIT

Install

View on GitHub

Quality Score: 82/100

Stars 20%
20
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Chaos and Resilience ## Fail-Fast Rule If a system component enters an unrecoverable state, **fail fast and loud**. Never silently swallow errors in resilience-critical paths. Never assume a retry will fix a structural problem. Never disable safety mechanisms (watchdogs, circuit breakers) without a replacement. ## Current Resilience Posture (Audit 2026-03-23: 6.5/10) ### What Works - Circuit breakers: 3 layers (HTTP, WS, per-venue) - Reconnect gate: max 3 connections per 10s with jitter backoff - Graceful shutdown: SIGINT/SIGTERM with 10s hard kill - Adapter process auto-restart: 10x with exponential backoff - PubSub backpressure: 1MB skip, 4MB force-close, slow-client detection - EL monitoring: stall detector, iteration tracker, per-operation profiler ### What Does NOT Work | Issue | Impact | File | |-------|--------|------| | IPC batch buffers UNBOUNDED | OOM then restart cascade then permanent death | adapter-process.ts:330-351 | | BP watchdog DISABLED | Worker can hang indefinitely | adapter-process.ts:71 | | Max restarts without recovery | After 10 crashes, dies permanently | gateway-wiring.ts | | Supabase without circuit breaker | Flush workers waste CPU when Supabase is down | supabase-persistence.ts | | unshift bypass in backpressure | Queues grow beyond cap | supabase-persistence.ts:877 | | unhandledRejection not fatal | System in inconsistent state continues running | index.ts:2557 | ## Top 5 Failure Scenarios ### 1. IPC Silent Failure (CRITICAL) ``` IPC...

Details

Author
Canhada-Labs
Repository
Canhada-Labs/ceo-orchestration
Created
4 weeks ago
Last Updated
today
Language
Python
License
MIT

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

chaos-engineering

Failure injection patterns, blast radius control, steady state hypothesis, and gameday planning for resilience testing.

519 Updated 1 weeks ago
vibeeval
Code & Development Listed

principle-resiliency

Resiliency principles — fault tolerance, resilience, partial failure, blast radius, failure domains, bulkheads, resource isolation, graceful degradation, fail-fast vs fail-soft, health checks, liveness vs readiness probes, cascading failure, gray failure, fault isolation, idempotency, idempotency keys, safe retry, deduplication, exactly-once, rate limiting, throttling, quotas, backpressure, token bucket, leaky bucket, jitter, thundering herd, synchronized retries, 429. Auto-load when designing for partial failure, isolating dependencies via bulkheads, planning graceful degradation, choosing fail-fast vs fail-soft, configuring health/readiness/liveness probes, evaluating cascading failure risk, designing fallback paths, reviewing system-level fault tolerance, making operations safely retryable with idempotency keys, implementing dedup stores, designing rate limiters or token buckets, handling 429 responses and backpressure, adding jitter to avoid thundering herd retry storms and synchronized retry spikes, or p

3 Updated today
lugassawan
Testing & QA Listed

chaos-test

Analyze codebase for resilience gaps — missing retry logic, absent timeouts, circuit-breaker opportunities, single points of failure, and degradation-mode blind spots

1 Updated 1 weeks ago
manastalukdar