# Skill eval loop (Agent Skill)

> For skill maintainers: harden an Agent Skill from its automatic agent evals. Score every run against a fixed rubric read from the agent's log, fix the skill at the root cause, release, and repeat until every agent scores full.

Page: https://timerise.ai/skills/skill-eval-loop

Agent Skill `skill-eval-loop`: https://github.com/timerise-ai/skill-eval-loop. Category: AI & integrations. Runs on Next.js; backend: No data store, the target skill's repository and its eval runs via the `gh` CLI.

Latest release: v0.1.3 (2026-09-28).

Install: npx skills add timerise-ai/skill-eval-loop. A tool for an agent, not a module: nothing for Timerise to build or quote.

A skill with automatic agent evals reports a result for every release: Claude Code, Codex and Gemini CLI each install it into an empty app, build the module and run the checks. When every run passes, the skill looks finished.

It is not. A passing run proves that the app type-checks, builds and runs its tests. It does not prove that the agent used the skill as written. Agents patch the shipped templates, rewrite the tests to suit themselves, widen the defaults and skip the handover, and the checks still pass. The deviations are in the agent's log, and each one traces back to a sentence in the skill that allowed it.

This skill gives a maintainer's agent the loop we run on our own skills: score every eval run against a rubric fixed before the first round, read from the agent's log, fix the skill at the root cause of each deviation, cut a patch release, let the release re-run the evals, and stop only when every agent scores full.

Harden an Agent Skill from its automatic agent evals: score every eval run against a fixed fidelity rubric read from the agent's own log, fix the skill at the root cause of each deviation, cut a patch release, let the release re-run the evals, and repeat until every agent scores full. Use when: (1) a skill's evals all "pass" but agents still patch its templates, rewrite its tests, widen its defaults or skip its handover, (2) a skill has an eval workflow that runs Claude Code, Codex and Gemini on every published release, (3) the user mentions: fix the skill based on evals, eval loop, iterate until full score, score the eval runs, rubric, agent deviations, "based on last evals fix skill", "loop to full score", "2:1 majority". Carries the rubric, the log readers for three agents, a root-cause table for deviations, the release recipe and the stop rule. Skill repositories with a CHANGELOG, tagged releases and an `evals/` folder; GitHub Actions and the gh CLI. Not a way to write a new skill and not a way to change the eval harness.

This skill was written by the maintainer who has run the loop on a published skill, from its first scored release to a unanimous full score. What it carries holds by construction: a rubric fixed before round one and the same for every agent, so scores compare across rounds; every agent's template edit reproduced against the template before it is adopted; a template check that type-checks the templates and matches the documented test count before any round ships; and a stop rule that is set in advance and never moved. [`references/provenance.md`](references/provenance.md) has the record.

What it builds:
- A skill hardened from its automatic agent evals
- Each run scored against a fixed fidelity rubric read from the agent's log
- Each deviation fixed at its root cause in the skill
- A patch release per round
- Repeated until every agent scores full

When to use:
- A skill that already has automatic agent evals, with results committed to its `evals/` folder and a workflow that runs on every published release. Run it after a release's evals land, or when asked to iterate a skill to a full score. A target named but not present is cloned from its public repository; that clone and the package registry are not external services in an eval's sense. Clone it to a scratch directory, and leave a working directory that is not the target as it was: no dependency, script, config or copied code, only the loop's notes.
- Pushing tags and publishing releases is outward-facing. Confirm once, before the first round, that every round may push and publish unattended; then report per round and stop early only if a run fails outright or a fix would weaken one of the skill's non-negotiables.

Non-negotiables:
- Result frontmatter is never edited and a run is never deleted. The frontmatter is what was measured, so scores go in the result's body, in a `chore(evals)` commit; `git diff` on the frontmatter stays empty.
- The eval is never changed to make a skill pass. Prompt, unattended note, harness and workflow are the same for every skill, so a fix there proves nothing about this one; every fix lands in the skill.
- An agent's template edit is reproduced before it is adopted. An edit is a claim, and a probe against the shipped template settles it: a real defect gets a template fix and a test that fails on the old code; an improvisation gets a sentence that forbids it.
- No round ships without the template check. The templates type-check and the documented test count holds under every runner the skill names, because a round's fix can break the templates it did not touch.
- Evals never bump a version. A version marks a change to the skill; a round with nothing to fix re-runs by dispatch, and score commits never ride a release commit.
- The stop rule is fixed before round one and followed to the end. At least three rounds, unanimous through round five, two of three after; moving it mid-loop would let a round's scores choose its own finish line.

Not this:
- Writing a new skill, or turning an app's module into one: `extract-skill` or `skill-creator`
- Cutting a release without scoring evals: `bumpv`
- Checking an app's code against its docs: `code-audit`
- Changing the eval harness, prompts or workflow: The skills index repository, by a maintainer

What to tell your agent:
- Before we start iterating on one of our skills, write down the rubric you will score its agent eval runs against, and say which items you still need the skill itself to fill in.
- Every agent eval of a skill I maintain passes, but I doubt the agents follow the skill. Tell me how you would score the runs of its latest release and what you need from me to do it; do not fix or release anything.
- In the last evals of a skill I maintain, Codex rewrote the shipped tests to node:assert and still passed. Explain what in a skill usually lets that happen, and where the fix belongs, without changing or releasing anything.

Agent eval results (newest per agent and prompt):
- Gemini CLI (gemini-3.8-flash) on v0.1.3, 2026-09-28: pass
- Codex CLI (gpt-6-astra) on v0.1.3, 2026-09-28: pass
- Claude Code (claude-opus-5-5) on v0.1.3, 2026-09-28: pass

Recent releases:
- v0.1.3 (2026-09-28): Patch release: records the prompt-1 agent eval runs of 0.1.2. The skill itself is unchanged.
- v0.1.2 (2026-09-28): Patch release: the skill is framed as hardening any Agent Skill, and its own eval prompts no longer name a target skill.
- v0.1.1 (2026-09-28): Patch release, from scoring the prompt-1 agent eval runs against 0.1.0.