10x-eval-modellisted

Set up and run benchmark evaluations for new LLM models in the 10xBench project. Use when the user wants to add a new model to the benchmark, prepare evaluation directories, update metadata, or launch evaluation runs. Triggers on phrases like "eval model", "add model to benchmark", "run benchmark for [model]", "evaluate [model-name]", "set up [model] for eval", or any request involving adding a new model to the Przeprogramowani.pl benchmark pipeline.
przeprogramowani/10x-bench · ★ 7 · AI & Automation · score 76

Install: claude install-skill przeprogramowani/10x-bench

# 10x-eval-model Orchestrates the full setup and execution pipeline for evaluating a new LLM model in the 10xBench benchmark. This includes updating metadata, verifying pricing, preparing attempt directories, and launching evaluation runs via the appropriate harness. ## Inputs The user provides: - **Model ID** — the short identifier used in directory names (e.g., `glm-51`, `gemini-31-pro`) - **Display name** — human-readable name (e.g., `GLM-5.1`, `Gemini 3.1 Pro`) - **Harness** — the coding agent environment used to run the model (e.g., `opencode`, `cursor`, `claude-code`, `claude-desktop`, `codex-desktop`) - **Supersedes** (optional) — the model ID this new model replaces (e.g., `glm-5` is superseded by `glm-51`) If the user provides a natural model name like "GLM-5.1 via opencode", derive the model ID yourself (e.g., `glm-51`) and confirm with the user before proceeding. ## Pipeline Steps ### Step 1: Check if model already exists in metadata Read `eval-attempts/metadata.ts` and check whether the model ID is already present. If it exists but is commented out, offer to uncomment it instead of adding new entries. If it already exists and is active, inform the user and skip to Step 4. ### Step 2: Look up pricing Search the web for current API pricing for the model (per 1M tokens, input and output, USD, no cache). Present findings to the user for confirmation before writing to metadata. Check multiple sources (official docs, OpenRouter, Artificial Analysis) to cross-re