Micro-averaged across all 35 benchmarks × 3 models (175 graded skill / benchmark / model results, 3 trials each). The mean hides wide variance: per-skill uplift ranges from −38pp (frontend-design) to +33pp (brand-guidelines).
The three coding agents benchmarked below, by parameter count — all open-weight mixture-of-experts models. The large "total" figure is the full parameter pool; "active" is what actually fires per token.
skillproof turns a skill folder (a SKILL.md plus supporting scripts/references) into a set of graded, reproducible coding tasks, then measures whether mounting that skill actually changes an agent's pass rate. Nothing here is hand-written per skill — the whole pipeline below runs the same way for every skill, from chunking the docs to grading the final trial.
Each skill's markdown is split into heading-bounded chunks (one chunk per section/subsection). Every chunk is embedded with the configured embedding model, then clustered with k-means, sweeping k over a small range and keeping whichever k gives the best silhouette score. From those clusters, skillproof greedily selects the k_benchmarks most mutually dissimilar clusters (max-min selection on cluster-centroid distance) — the goal is one benchmark per genuinely distinct capability the skill covers, not five benchmarks that all probe the same paragraph.
For each selected cluster, a headless coding agent (Codex CLI or Claude Sonnet in headless mode) is given that cluster's chunk text and a strict output contract, and asked to author a complete, self-contained benchmark: a task prompt, input fixtures, a reference solution, and a deterministic grader — without seeing the skill's own guidance beyond that cluster, so the benchmark tests the capability rather than parroting the skill's phrasing.
An authored benchmark is only kept if it clears three checks: the bundled reference solution must pass the grader; an untouched/empty workspace must fail the grader (otherwise the task is gradeable by doing nothing); and the grader must return the same verdict on two independent runs of the reference solution (determinism — no flaky timing- or ordering-dependent checks). If a benchmark fails validation, the authoring agent gets the failure output back and retries up to max_attempts times; if it still can't produce something that passes the gate, the benchmark is dropped and a FAILED.md explaining why is written instead, excluding it from evaluation entirely.
Evaluation is a single model-agnostic loop that drives any model over OpenRouter through a small bash/read_file/write_file tool surface (see schemas below) inside the sandbox. Every benchmark is run in two arms: with_skill, where the skill's SKILL.md is injected into the system prompt and its supporting files are mounted read-only at /skill; and without_skill, where the agent gets neither the injected instructions nor the mount — same task, same tools, same model, skill withheld. Uplift for a given benchmark × model is simply pass_rate(with_skill) − pass_rate(without_skill).
Every trial — authoring, validation, and each evaluation arm — runs in a fresh, network-less Docker container built from the same image, so nothing can cheat by calling out to the internet and nothing leaks state between trials. Grading follows a snapshot-then-grade pattern: the agent's final /workspace is captured, then copied into a brand-new container to run the grader, so a grader can't be influenced by leftover agent processes or a corrupted shell.
Some capabilities (tone, completeness, "did it avoid overclaiming") don't reduce to a substring match. For those, the grader exits with code 3 and prints one or more yes/no questions about the produced artifact; skillproof routes each question to a three-model judge panel and takes a majority vote. The panel is instructed to be strict: for a multi-condition question, every condition has to be satisfied for a YES, and any doubt resolves to NO — there's no partial credit for grading purposes.
One YAML file controls the seed, embedding provider/model, clustering k range, authoring backend/model/attempt budget, and evaluation defaults (trial count, model list, timeouts) for every skill run through the pipeline.