data-engineeringlisted

**CONSUMES**: User data-pipeline task + source/destination description + optional existing pipeline configs, dbt models, warehouse schemas, or RAG/streaming setup **PRODUCES**: Applied data-engineering judgment rules + ingestion-pattern decisions (ETL vs ELT) + transformation/feature designs + orchestrator selection + data-quality gate configs + SCD/dimensional model + vector-filtering & streaming-inference architecture # Data Engineering for AI Capability Pack **Version**: 0.1.0 **Compatibility**: Claude Code (Phase 1); Codex / Cursor / Gemini in Phase 3 **License**: Apache 2.0 --- ## What This Pack Does AI agents build data pipelines by reaching for the first tool they remember. They default to Pandas where Polars would parallelize for free. They enforce rigid schemas on raw JSON before storage, discarding the raw history that model retraining needs. They write feature transformations once in a training notebook and again in a serving path — silently introducing train-serve skew. They pick Airflow for a lightweight agentic workflow that Prefect handles better. They run a global vector search and post-filter, then wonder why RAG returns zero results for a tenant. They build SCD Type 2 tables that scan ~160M rows because a query forgot `is_current = true`. This pack embeds the judgment rules that data engineers apply automatically — rules grounded in 2026 tooling research (dlt 1.27.2, DuckDB v1.4-LTS, Polars, dbt-core, Apache Airflow 3.x / 3.2.2 docs, Dagster 1.13, Pref