← ClaudeAtlas

nasde-benchmark-creatorlisted

Create coding agent benchmarks for evaluation with nasde. Use this skill when the user wants to: - Create a new benchmark project (set of tasks for evaluating coding agents) - Add tasks to an existing benchmark - Create or modify agent variants (configurations that control agent behavior) - Set up assessment dimensions and scoring criteria - Verify that a new benchmark's Docker environment and tests work Even if the user doesn't say "benchmark" — if they're talking about creating coding challenges for AI agents or setting up evaluation criteria, this skill applies.
NoesisVision/nasde-toolkit · ★ 10 · AI & Automation · score 79
Install: claude install-skill NoesisVision/nasde-toolkit
# NASDE Benchmark Creator Create and configure coding agent benchmarks for evaluation with `nasde`. A benchmark is a set of coding tasks that AI agents solve inside isolated Docker containers, scored both by functional tests (pass/fail) and by an LLM-as-a-Judge architecture assessment. ## Critical: line endings on Windows (read this first) Benchmark scripts execute inside **Linux** sandboxes (Docker, Daytona). If `tests/test.sh`, `solution/solve.sh`, or `environment/Dockerfile` are checked out with **CRLF** line endings (the Windows git default when `core.autocrlf=true` and there is no `.gitattributes`), every trial fails immediately with: ``` bash: line 1: /tests/test.sh: cannot execute: required file not found ``` …because the kernel reads the shebang as `#!/bin/bash\r` and tries to execute a non-existent `/bin/bash\r`. The agent finishes its work, but the verifier never runs and Harbor reports `RewardFileNotFoundError`. **Mitigation (always do this for a new benchmark — `nasde init` does it for you, but verify):** 1. The benchmark repo MUST have a `.gitattributes` file enforcing LF for shell scripts and Dockerfiles. The minimum content: ```gitattributes * text=auto eol=lf *.sh text eol=lf *.bash text eol=lf Dockerfile text eol=lf *.dockerfile text eol=lf docker-compose.yaml text eol=lf docker-compose.yml text eol=lf *.ps1 text eol=crlf *.bat text eol=crlf *.cmd text eol=crlf ``` `nasde init` wri