From dfbd45993fae9eaa946306f699a06dadcf0bd92a Mon Sep 17 00:00:00 2001 From: cliffhall Date: Fri, 4 Sep 2026 20:33:34 -0400 Subject: [PATCH] docs(skills): record the hand-off measurement honestly after the turn rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-measured the two committed chain cases under the stricter turn-boundary scoring from 313a5809: 100% and 33% at RUNS=3, against 33% and 33% before. That rise is NOT an effect of the change and the doc now says so. The turn rule only ever rejects matches a flatter reading accepted, so it cannot raise a score — the two runs simply straddle a 67-point swing on the same prompt at RUNS=3, where one sample is worth 33 points. Reading it as an improvement would be exactly the mistake the eval exists to prevent. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe Signed-off-by: cliffhall --- docs/skill-authoring.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/skill-authoring.md b/docs/skill-authoring.md index 0bab8872..6e515833 100644 --- a/docs/skill-authoring.md +++ b/docs/skill-authoring.md @@ -289,12 +289,21 @@ pointed at the second, and the column would stop carrying signal. Read a hand-off number as a description-strength measurement, not a verdict — and read it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points. -**Both committed cases currently sit at 33% and are therefore red, and that is -the intended state rather than an oversight.** `skills:eval` is not a gate (see -below), and the number is the finding: `testing` points at `test-servers` in its -first paragraph and the model follows that pointer about a third of the time. -Strengthening it is its own change against its own issue; lowering the bar to -turn the column green would throw away the only signal this feature adds. +**The committed cases have measured 33% / 33% on one `RUNS=3` run and 100% / +33% on another, and at least one of them being red is the intended state rather +than an oversight.** `skills:eval` is not a gate (see below), and the number is +the finding: `testing` points at `test-servers` in its first paragraph and the +model follows that pointer *sometimes*. Strengthening it is its own change +against its own issue (#2247); lowering the bar to turn the column green would +throw away the only signal this feature adds. + +⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One +sample is 33 points there, and the two runs above straddle a 67-point swing on +the same prompt with no change to the pointer. Note in particular that the +turn-boundary rule added later can only ever *lower* a chained score — it +rejects matches a flatter reading accepted — so a higher number after it is +noise by construction, not an effect. `RUNS=5` is the smallest honest setting +for a hand-off, and the cost is real: each sample is up to 14 turns. ⚠️ **Expect a hand-off to cost far more than a first move.** Each sample is up to 14 turns rather than one, so a chained case is the most expensive line in the @@ -363,7 +372,7 @@ The summary is two lines, never one: ``` 7/7 first-move cases at or above 80%. -0/2 hand-off cases above 50%. +1/2 hand-off cases above 50%. ``` Narrowing the run never narrows what a **negative** case is scored against — a