compare-llm-modelslisted

# Compare LLM models for your task The best model on a public leaderboard is often not the best model for *your* task at *your* cost and latency. Public benchmarks narrow the field; your own eval set makes the call. ## Use benchmarks to shortlist, not to decide - **General leaderboards** (Chatbot Arena / LMArena for human preference, HELM for multi-metric, MMLU/GPQA for reasoning) tell you the rough tier a model is in. Use them to pick 2-4 candidates, not to declare a winner for your app. - **Watch for contamination and overfitting** to popular benchmarks. A high MMLU score does not mean the model is good at your specific extraction/RAG/agent task. - **Task-relevant benchmarks** beat general ones: if you do code, look at code evals; if RAG, look at long-context/faithfulness; if tools, look at agent/tool-use benchmarks. ## Then evaluate the shortlist on YOUR eval set This is the part that actually decides it. Run each candidate model through your own eval suite (see `build-eval-dataset` and `add-llm-evals`) and compare on the axes that matter: | Axis | How to measure | |---|---| | **Quality** | Your eval scores on your dataset (not a leaderboard) | | **Cost** | Tokens x price on your real prompts (see `reduce-llm-cost`) | | **Latency** | p50/p95 on your prompt sizes | | **Reliability** | Structured-output adherence, refusal rate, error rate | | **Context/limits** | Context window, rate limits, region/availability | | **Fit** | Tool-calling quality, multilingual, safety,