← ClaudeAtlas

vllm-benchmark-result-analysislisted

Compare vLLM serving benchmark outputs before and after a code change. Parses plain-text vLLM benchmark output (the "Serving Benchmark Result" block), computes per-metric percentage changes with improvement/regression markers, generates a Markdown report with a full metrics table and a key-changes summary, and saves it to ./outputs/. Use when the user pastes or provides vLLM benchmark output and asks to compare, summarize, or analyze performance differences between two runs (e.g., "before this PR" vs "after this PR", or any before/after wording).
shen-shanshan/vllm-dev-skills · ★ 16 · AI & Automation · score 80
Install: claude install-skill shen-shanshan/vllm-dev-skills
# vLLM Benchmark Summary ## Workflow 1. **Identify the input** — the user pastes combined text containing a "before" block and an "after" block (or provides two separate files). 2. **Save the input** to `/tmp/bench_input.txt`, then run the script. 3. **Run the script** using the bundled `./scripts/compare_benchmarks.py`. This generates the Performance Metrics table. 4. **Generate the Summary** — after the script runs, analyze the table data and write a narrative `## AI Summary` section (see guidelines below). 5. **Assemble and save** — append the Summary to the output file, then show the complete report to the user. ## Running the Script ### Combined text input (most common) Save the user's pasted text to `/tmp/bench_input.txt`, then: ```bash python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \ /tmp/bench_input.txt \ --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \ --title "vLLM Benchmark Comparison" ``` ### Two separate files ```bash python3 /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/scripts/compare_benchmarks.py \ --before before.txt --after after.txt \ --output-dir /Users/shanshan-shen/.claude/skills/vllm-benchmark-result-analysis/outputs \ --title "vLLM Benchmark Comparison" ``` ## Input Format The script expects the standard vLLM benchmark output block: ``` Before this PR: ← any line containing "before" (case-i