data-scientist

# Data Scientist: Hybrid-Engine Data Processing Answer data questions through the cheapest engine and surface that can prove the answer, and decide where the computation should live before touching the data. ## Execution surfaces: resident kernel first A persistent REPL/eval kernel (many harnesses expose one for JavaScript and Python) is the default surface. Reason: each one-shot process pays roughly a second of spawn-plus-import overhead and re-scans the input file, while a resident connection amortizes both — after a one-time load, repeat queries return in milliseconds. Exploration is repeat queries, so this difference dominates the session. 1. **JavaScript kernel (Bun)**: run `scripts/ensure-js-deps.sh` once; it prints the absolute import path for `@duckdb/node-api`. Dynamic-import it, connect once, query across cells. 2. **Python kernel**: the default surface for Python work. duckdb/numpy/matplotlib are typically resident; Polars and pyarrow come from `scripts/ensure-py-deps.sh`, which installs them once into a user cache keyed to the kernel's interpreter — `sys.path.insert` the printed directory and import. The interpreter itself is never mutated. 3. **uv lane** (`uv run --with ...`): isolation for a heavy or crash-prone one-shot that should not take the kernel down. 4. **No kernel** (plain-shell harness): the same engines as one-shots — `bun -e` for DuckDB-js, `uv run python -c` for the Python stack — batching several questions per process. ...