developing-incremental-models

Solid

Develops and troubleshoots dbt incremental models. Use when working with incremental materialization for: (1) Creating new incremental models (choosing strategy, unique_key, partition) (2) Task mentions "incremental", "append", "merge", "upsert", or "late arriving data" (3) Troubleshooting incremental failures (merge errors, partition pruning, schema drift) (4) Optimizing incremental performance or deciding table vs incremental Guides through strategy selection, handles common incremental gotchas.

AI & Automation 113 stars 8 forks Updated 1 weeks ago MIT

Install

View on GitHub

Quality Score: 85/100

Stars 20%
68
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# dbt Incremental Model Development **Choose the right strategy. Design the unique_key carefully. Handle edge cases.** ## When to Use Incremental | Scenario | Recommendation | |----------|----------------| | Source data < 10M rows | Use `table` (simpler, full refresh is fast) | | Source data > 10M rows | Consider `incremental` | | Source data updated in place | Use `incremental` with `merge` strategy | | Append-only source (logs, events) | Use `incremental` with `append` strategy | | Partitioned warehouse data | Use `insert_overwrite` if supported | **Default to `table` unless you have a clear performance reason for incremental.** ## Critical Rules 1. **ALWAYS test with `--full-refresh` first** before relying on incremental logic 2. **ALWAYS verify unique_key is truly unique** in both source and target 3. **If merge fails 3+ times**, check unique_key for duplicates 4. **Run full refresh periodically** to prevent data drift ## Workflow ### 1. Confirm Incremental is Needed ```bash # Check source table size dbt show --inline "select count(*) from {{ source('schema', 'table') }}" ``` If count < 10 million, consider using `table` instead. Incremental adds complexity. ### 2. Understand the Source Data Pattern Before choosing a strategy, answer: - **Is data append-only?** (new rows added, never updated) - **Are existing rows updated?** (need merge/upsert) - **Is there a reliable timestamp?** (for filtering new data) - **What's the unique identifier?** (for merge matching...

Details

Author
AltimateAI
Repository
AltimateAI/data-engineering-skills
Created
6 months ago
Last Updated
1 weeks ago
Language
N/A
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

creating-dbt-models

Creates dbt models following project conventions. Use when working with dbt models for: (1) Creating new models (any layer - discovers project's naming conventions first) (2) Task mentions "create", "build", "add", "write", "new", or "implement" with model, table, or SQL (3) Modifying existing model logic, columns, joins, or transformations (4) Implementing a model from schema.yml specs or expected output requirements Discovers project conventions before writing. Runs dbt build (not just compile) to verify.

113 Updated 1 weeks ago
AltimateAI
AI & Automation Solid

refactoring-dbt-models

Safely refactors dbt models with downstream impact analysis. Use when restructuring dbt models for: (1) Task mentions "refactor", "restructure", "extract", "split", "break into", or "reorganize" (2) Extracting CTEs to intermediate models or creating macros (3) Modifying model logic that has downstream consumers (4) Renaming columns, changing types, or reorganizing model dependencies Analyzes all downstream dependencies BEFORE making changes.

113 Updated 1 weeks ago
AltimateAI
Data & Documents Listed

dbt-patterns

dbt model design, ref chains, sources, tests, macros, incremental strategies, materializations, and documentation best practices. Use this skill whenever the user is writing or reviewing dbt models, configuring dbt tests, designing model layers (staging/intermediate/marts), asking about incremental models, choosing materializations, writing macros, setting up sources.yml, or troubleshooting dbt run/test failures. Also trigger when the user mentions dbt refs, lineage graphs, model dependencies, dbt Cloud, the dbt CLI, or when they want to transform data already in the warehouse using SQL. If the project uses dbt at all, this skill should be active for any transformation questions.

1 Updated 1 weeks ago
Methasit-Pun