adjudication-sheets

Solid

Build human adjudication / hand-labeling sheets from LLM-pipeline data without evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to rule on cases an LLM classifier or rater panel judged, (2) a labeler reports "there is no information to label from" or cells look empty in Excel, (3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap, multi-line CSV mangling, ruling dropdowns, companion text files.

Data & Documents 57 stars 0 forks Updated 4 days ago MIT

Install

View on GitHub

Quality Score: 90/100

Stars 20%
59
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
80
License 10%
100
Description 5%
100

Skill Content

# Human Adjudication Sheets from LLM-Pipeline Data ## Problem Adjudication sheets built from pipeline intermediates tend to carry truncated "excerpt" columns (previews made for machine diffing, not human judging). A human asked to rule "does this text show X?" on a fragment produces invalid rulings: the decisive sentence may sit past the cutoff. Separately, multi-line text in CSV cells renders as broken rows/empty cells in Excel, so the labeler reports "there is no data here" even when the column is populated. ## Context / Trigger Conditions - A labeler says the sheet has "no information" — check for embedded newlines in CSV fields first. - Excerpt lengths cluster at an exact value (all 1,204 / 1,500 / 800 chars) or end with "[…]" — that is a hard cap, not natural length. - The ruling question is "does the record/text show X?" — the ruling basis must be byte-identical to what the classifier/raters saw, or the ruling grades a different object. ## Solution 1. **Trace the true rating basis from the scoring code, not from intermediates.** Find the line where text is passed to the model (e.g. `prompt = ... + text`) and reproduce that construction exactly. Verify: stored excerpt == full_text[:cap] (0 mismatches), or cache-key match against the LLM request cache. 2. **Deliver .xlsx, not .csv**, for any sheet with multi-line text: wrap_text + frozen header + generous width on evidence columns; data-validation dropdown on the `ruling` column with the exact allowed codes (e.g. E_pos...

Details

Author
kennethkhoocy
Repository
kennethkhoocy/applied-micro-skills
Created
5 days ago
Last Updated
4 days ago
Language
Python
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

llm-as-judge-scorer

Design a reliable LLM-as-judge metric — a calibrated rubric, a clear scoring scale, and bias controls — and validate it against human labels before trusting it. Use when grading open-ended LLM output (summaries, answers, tone) that exact-match can't score.

1 Updated 1 weeks ago
imtiazrayhan
AI & Automation Listed

judge-calibration-auditor

Iterate-stage skill: analyzes human labels vs LLM-judge verdicts and turns every disagreement into a classified calibration signal with a stated correction — never auto-resolved toward the judge. Use when a judge and humans diverge — 'our judge disagrees with human reviewers', 'human labels vs judge verdicts, what's drifting', 'the judge scores everything 4', 'disagreement analysis' — or when /pm routes such a request here. Do NOT use to design the judge (llm-as-judge-designer), to build the eval (eval-engine), to produce the human labels themselves, or for judge-bias knowledge questions.

1 Updated 5 days ago
Abhillashjadhav
AI & Automation Listed

llm-as-judge-designer

Iterate-stage skill: turns existing eval criteria into an LLM judge prompt with anchors and few-shot calibration cases — every rubric point carrying both a pass and a fail exemplar. Use when criteria exist and the judge needs building — 'write the judge prompt for these criteria', 'turn this rubric into an LLM judge', 'judge prompt plus calibration cases' — or when /pm routes such a request here. Do NOT use to build the whole eval from a spec (eval-engine), to audit an existing judge against human labels (judge-calibration-auditor), to execute scoring over outputs, or for judge-reliability knowledge questions.

1 Updated 5 days ago
Abhillashjadhav