nasde-benchmark-from-history

Solid

Generate benchmark tasks from git history of the current or specified repository. Use this skill when the user wants to: - Create benchmark tasks based on real problems their team already solved (closed PRs, past commits, resolved issues) - Mine git history for good evaluation candidates - Turn a commit range or set of PRs into a NASDE benchmark - Build a regression test suite from their team's actual work Even if the user doesn't say "benchmark" — if they're talking about turning past work into evaluation tasks, or want to test AI agents against problems they've already solved, this skill applies.

AI & Automation 11 stars 0 forks Updated today MIT

Install

View on GitHub

Quality Score: 79/100

Stars 20%
36
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# NASDE Benchmark from Git History Generate NASDE benchmark tasks by mining git history. You analyze commits, diffs, and PR descriptions to identify self-contained changes that make good evaluation candidates, then generate task files with user approval. ## Prerequisites - A git repository with meaningful commit history (the repo you're currently in, or a path to another local repo) - An existing NASDE benchmark project (run `nasde init` first, or use the `nasde-benchmark-creator` skill) - If the benchmark project doesn't exist yet, create it first — this skill generates tasks, not the project scaffold ## Critical: line endings on Windows (read this first) When generating `tests/test.sh`, `solution/solve.sh`, or `environment/Dockerfile` on a Windows host, write them with **LF** line endings or every trial fails with `bash: required file not found` (the kernel reads `#!/bin/bash\r` as the shebang). See the full explanation and `.gitattributes` template in the `nasde-benchmark-creator` skill. Quick rules: - The benchmark project MUST have a `.gitattributes` enforcing `*.sh text eol=lf` and `Dockerfile text eol=lf`. `nasde init` creates this. If the existing project lacks it, **create `.gitattributes` before generating any task files**. - When writing files programmatically, use `path.write_text(content, encoding="utf-8", newline="")` — never the bare default which translates `\n`→`\r\n` on Windows. - Sanity-check after generation: `find tasks/<new-task> -name '*.sh' -o -n...

Details

Author
NoesisVision
Repository
NoesisVision/nasde-toolkit
Created
4 months ago
Last Updated
today
Language
Python
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Solid

nasde-benchmark-from-public-repos

Build diverse benchmark task suites from public GitHub repositories for testing universal skills. Use this skill when the user wants to: - Create a benchmark that spans multiple public repositories and languages - Test a universal skill (refactoring, test writing, code review, etc.) across diverse codebases - Curate a representative set of repos and tasks for cross-codebase validation - Build an evaluation suite for a skill that should work in any repository Even if the user doesn't say "benchmark" — if they're building a skill meant to work everywhere and want to validate it across many different projects, this skill applies.

11 Updated today
NoesisVision
AI & Automation Solid

nasde-benchmark-creator

Create coding agent benchmarks for evaluation with nasde. Use this skill when the user wants to: - Create a new benchmark project (set of tasks for evaluating coding agents) - Add tasks to an existing benchmark - Create or modify agent variants (configurations that control agent behavior) - Set up assessment dimensions and scoring criteria - Verify that a new benchmark's Docker environment and tests work Even if the user doesn't say "benchmark" — if they're talking about creating coding challenges for AI agents or setting up evaluation criteria, this skill applies.

11 Updated today
NoesisVision
AI & Automation Solid

nasde-benchmark-runner

Run coding agent benchmarks and verify results with nasde. Use this skill when the user wants to: - Run a benchmark (all tasks, single task, specific variant) - Re-run assessment evaluation on existing trial results - Check or verify results in Opik (traces, feedback scores, experiments) - Troubleshoot a failed benchmark run - View or compare trial results Even if the user doesn't say "benchmark" — if they're talking about running evaluations, checking scores, or analyzing agent performance, this skill applies. After every run that uses --with-opik, ALWAYS verify results via Opik REST API — don't wait for the user to ask.

11 Updated today
NoesisVision