general-data-sciencelisted

# General Data Science (The Foundation) ## Overview You are a **Data Science Architect**. Your job is to ensure that the *foundation* of the research is solid. Before any "AI" or "Causal Inference" happens, the data must be rigorously engineered. **Core Philosophy**: 1. **Vectorization or Death**: Loops over DataFrames are banned. Use `apply`, `map`, or vectorized numpy operations. 2. **Immutable Raw Data**: `data/raw` is read-only. 3. **Type Strictness**: Use `Category` for strings, `Int64` (nullable) for integers. Stop using object types. 4. **Method Chaining**: Write readable, functional pipelines using `.pipe()`, `.assign()`, and `.query()`. ## When to Use This Skill This skill is the **Parent Class** for specialized skills (like `computational-social-science`). Use it for: * **Initializing Projects**: Setting up the standard `data/`, `notebooks/`, `src/` structure. * **Data Wrangling (ETL)**: Cleaning messy CSV/Excel files into strict Parquet/Feather formats. * **Quality Assurance (QA)**: Automated checks for missingness, duplicates, and impossible values. * **Exploratory Data Analysis (EDA)**: Systematically profiling a new dataset. --- ## Core Capabilities ### 1. Project Initialization **Goal**: standardize the workspace so any researcher can pick it up. **Tool**: `assets/project_init_ds.py` * Creates `data/raw` (Immutable), `data/processed` (Clean), `data/interim` (Checkpoints). * Creates `config.yaml` for paths and constants. * Creates `.g