benchmarklisted

# LemonCrow benchmark Measures what LemonCrow actually buys you on **your** repo and prompts — offline from session history (free) or online as a real A/B run against the host you are in, vanilla vs LemonCrow (real API spend). Does **not** benchmark LemonCrow internals — dev suite commands at the bottom for that. Three channels: | Mode | What it measures | Cost | | -------------------------------------- | --------------------------------------------------------------------------- | -------------------------------- | | **Offline** (`lc eval fitness`) | How many `grep` calls LemonCrow's `code_search` collapses in YOUR session history | Free (reads local session files) | | **Online** (`lc benchmark local`) | Side-by-side A/B cost + quality delta on YOUR prompts | Real API spend | | **Competitor** (`--competitor`) | 3-way: baseline vs LemonCrow vs **any GitHub tool** you point it at, on YOUR prompts | Real API spend (Claude Code host only) | --- ## Offline mode — session history analysis Reads Claude Code session files from `~/.claude/projects/` **and Codex rollouts from `~/.codex/sessions/`** (both are scanned on every run, whatever host you are in), extracts every `mcp__lc__grep`, `mcp__lc__code_search`, and `ToolSearch` call, groups them into "search episodes" between prompts, shows: - Individual `grep` call