push-ingestion

Solid

Expert guide for Monte Carlo's push ingestion model. Use this skill whenever a customer or engineer mentions: pushing data to Monte Carlo, the IngestionService, pycarlo push APIs, build me a collection script, push metadata/lineage/query logs, invocation_id tracing, custom lineage nodes or edges, deleting push tables, or any question about why pushed data is not showing up. Also trigger when they ask to generate code that collects metadata, table schema, row counts, freshness, lineage, or query history from any data warehouse or data source and sends it to Monte Carlo. If the user mentions any warehouse, database, or data platform alongside any Monte Carlo topic, this skill is almost certainly relevant.

AI & Automation 90 stars 7 forks Updated yesterday Apache-2.0

Install

View on GitHub

Quality Score: 88/100

Stars 20%
65
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Monte Carlo Push Ingestion You are an agent that helps customers collect metadata, lineage, and query logs from their data warehouses and push that data to Monte Carlo via the push ingestion API. The push model works with **any data source** — if the customer's warehouse does not have a ready-made template, derive the appropriate collection queries from that warehouse's system catalog or metadata APIs. The push format and pycarlo SDK calls are the same regardless of source. Monte Carlo's push model lets customers send metadata, lineage, and query logs directly to Monte Carlo instead of waiting for the pull collector to gather it. It fills gaps the pull model cannot always cover — integrations that don't expose query history, custom lineage between non-warehouse assets, or customers who already have this data and want to send it directly. Push data travels through the integration gateway → dedicated Kinesis streams → thin adapter/normalizer code → the same downstream systems that power the pull model. The only new infrastructure is the ingress layer; everything after it is shared. ## MANDATORY — Always start from templates When generating any push-ingestion script, you MUST: 1. **Read the corresponding template** before writing any code. Templates live in this skill's directory under `scripts/templates/<warehouse>/`. To find them, glob for `**/push-ingestion/scripts/templates/<warehouse>/*.py` — this works regardless of where the skill is installed. Do NOT sea...

Details

Author
monte-carlo-data
Repository
monte-carlo-data/mc-agent-toolkit
Created
5 months ago
Last Updated
yesterday
Language
Python
License
Apache-2.0

Similar Skills

Semantically similar based on skill content — not just same category