← ClaudeAtlas

copy-the-graders-limits-not-just-its-logiclisted

Use at study design, implementation and experimentation when the task's score is produced by *executing* what you submit - running generated programs against hidden test cases, simulating, decoding, solving, rendering - and where you are building a local copy of the scorer to choose among candidates or to measure a method before committing to it. Covers reading the resource limits out of the shipped scorer's source rather than out of the prose that summarises it, giving every limit a named constant with its source line beside it, why a replica looser than the grader is far worse than one that is stricter, the failure taxonomy your local report must be able to express (a bucket that is empty in every arm is the tell), and the item-by-item calibration that catches a replica whose average already agrees.
tangxiangru/AutoR · ★ 805 · AI & Automation · score 79
Install: claude install-skill tangxiangru/AutoR
# When the score comes from a run, the limits are part of the metric Most graded tasks compute a number from your file. Some *execute* it: they run your program, your simulation, your solver, your decode, under a harness with a clock and a memory cap, and turn what happens into the score. On those tasks the metric is not a formula - it is a runtime environment, and its resource limits are as much a part of the definition as the comparison at the end. A local replica of that harness is the right thing to build; you cannot choose among candidates without one. But a replica inherits the logic easily and the limits almost never, because the logic is what the code obviously does and the limits are constants sitting three call frames away. ## Enumerate the limits from the source, then check the prose against it Briefs of this kind state the limits twice: a prose section - "Resource Limits", "Execution Environment", "Evaluation protocol" - and the scorer's own source, often pasted into the same document or sitting in the task directory. The prose is a summary and it is incomplete, in the same way and for the same reason that a stated submission shape is (`the-row-count-comes-from-the-split-not-the-brief`). What is specific here is the **direction** of the error. Every limit in the source that the prose does not mention is a place where you will invent one instead, and an invented budget is always the loose one: nobody guesses a cap stricter than the cap they were told about. So