dt-obs-genai

Featured

Analyzes observability signals from customer GenAI applications with DQL. Reads OpenTelemetry GenAI spans and LLM evaluation bizevents. Use for: golden signals (traffic, errors, latency, saturation); LLM signals (model, provider, tokens); cost/token analytics, usage attribution, and prompt caching; agent signals (tool calls, steps, failures, loop detection, Smartscape topology); conversation/session analytics; guardrails (blocked/truncated responses); and evaluation signals (quality, pass/fail). Trigger: "LLM latency", "token usage by model", "cost by model and provider", "cost per conversation", "who is driving token spend", "do I have prompt caching", "failing agent tool calls", "find runaway agents", "responses truncated or blocked", "failed evaluations", "am I hitting rate limits", "token throughput / TPM", "provider throttling or 429s". Do NOT use for: Davis CoPilot/MCP telemetry (dt-platform), generic service metrics (dt-obs-services), logs (dt-obs-logs), or non-GenAI tracing (dt-obs-tracing).

AI & Automation 137 stars 28 forks Updated today Apache-2.0

Install

View on GitHub

Quality Score: 89/100

Stars 20%
71
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# AI Observability (GenAI) Skill Analyze AI Observability signals from customer GenAI applications using DQL — golden signals, LLM signals, token and cost analytics (with usage attribution and prompt-caching economics), agent signals (including loop/runaway detection and Smartscape topology), conversation/session-level analytics, guardrails, and evaluation quality. --- ## Core Capabilities ### Golden Signals The four classic observability signals — traffic, errors, latency, and saturation — apply directly to GenAI applications. Traffic is request throughput over time; errors are spans where `span.status_code == "error"`; latency is the `duration` field (a Grail duration value — divide by the `1ms` literal, `duration / 1ms`, for a numeric millisecond value); saturation is proxied by total token throughput per minute (input + output tokens combined). ```dql fetch spans, from: now()-24h | filter isNotNull(gen_ai.request.model) | summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model} | fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0) | sort error_rate_pct desc ``` → **Full traffic, latency, and saturation queries:** See [references/golden-signals.md](references/golden-signals.md) ### LLM Signals LLM signals describe which model and provider served each request, what operation type was invoked (`chat`, `execute_tool`, `invoke_agent`, `create_agent`), and how tokens were consumed. Use these to benchma...

Details

Author
Dynatrace
Repository
Dynatrace/dynatrace-for-ai
Created
5 months ago
Last Updated
today
Language
JavaScript
License
Apache-2.0

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

instrument-llm-observability

Use this when adding tracing/observability to an LLM or AI-agent application - capturing prompts, tool calls, token usage, latency, and cost per step. Trigger whenever someone wants to "add tracing", "instrument", "monitor", "see what my agent is doing", or debug an LLM app in production. Prefer vendor-neutral OpenTelemetry unless a specific platform is already in use.

33 Updated 4 days ago
ContextJet-ai
AI & Automation Listed

llm-observability

LLM observability & LLMOps capability pack. Gives AI agents the judgment rules for production-grade distributed tracing, real-time cost attribution & budget governance, OpenTelemetry GenAI semantic conventions, latency/TTFT profiling, prompt registry & versioning, and online evaluation with groundedness & drift detection. Research-grounded rules from LangSmith, Langfuse, Arize Phoenix, Helicone, MLflow, Braintrust, TrueFoundry, vLLM, and the OpenTelemetry GenAI semantic conventions. Use for any LLM monitoring, tracing, cost governance, prompt versioning, latency profiling, or production drift/hallucination task.

3 Updated today
Sheldon-92
AI & Automation Listed

grafana-llmops-forge

Runs end-to-end AI/LLM observability on any Grafana (OSS, Cloud, Enterprise) with a single prerequisite, a reachable Grafana. Auto-discovers the instance and which LLM telemetry dialects exist (OpenTelemetry GenAI gen_ai.*, LiteLLM, vLLM, TGI, GPU/DCGM, eval scores), then generates and deploys dashboards (FinOps and cost by provider sovereignty across US/EU/Asia, gateway SLOs, agent and RAG tracing, adoption, quality evaluations, self-hosted inference, and governance evidence read against the EU AI Act, ISO/IEC 42001 and NIST AI RMF). Also emits burn-rate SLO alerts, cost recording rules for Prometheus or the Prometheus Operator, English or French labels, and a visual verification pass. Use it whenever the user mentions Grafana, dashboards, AI or LLM monitoring, token costs, AI FinOps, LLMOps, agents or RAG, model adoption, AI Act, ISO 42001 or NIST AI RMF evidence, or Prometheus/Loki/Tempo applied to AI, even without the word dashboard, or to audit a stack emitting nothing yet.

1 Updated 2 days ago
alebgl77