docs(skills): record the hand-off measurement honestly after the turn rule
Re-measured the two committed chain cases under the stricter turn-boundary
scoring from 313a5809: 100% and 33% at RUNS=3, against 33% and 33% before.
That rise is NOT an effect of the change and the doc now says so. The turn
rule only ever rejects matches a flatter reading accepted, so it cannot
raise a score — the two runs simply straddle a 67-point swing on the same
prompt at RUNS=3, where one sample is worth 33 points. Reading it as an
improvement would be exactly the mistake the eval exists to prevent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
parent
313a580982
commit
dfbd45993f
1 changed files with 16 additions and 7 deletions
|
|
@ -289,12 +289,21 @@ pointed at the second, and the column would stop carrying signal. Read a
|
|||
hand-off number as a description-strength measurement, not a verdict — and read
|
||||
it at `RUNS=5`, since at `RUNS=3` one sample is worth 33 points.
|
||||
|
||||
**Both committed cases currently sit at 33% and are therefore red, and that is
|
||||
the intended state rather than an oversight.** `skills:eval` is not a gate (see
|
||||
below), and the number is the finding: `testing` points at `test-servers` in its
|
||||
first paragraph and the model follows that pointer about a third of the time.
|
||||
Strengthening it is its own change against its own issue; lowering the bar to
|
||||
turn the column green would throw away the only signal this feature adds.
|
||||
**The committed cases have measured 33% / 33% on one `RUNS=3` run and 100% /
|
||||
33% on another, and at least one of them being red is the intended state rather
|
||||
than an oversight.** `skills:eval` is not a gate (see below), and the number is
|
||||
the finding: `testing` points at `test-servers` in its first paragraph and the
|
||||
model follows that pointer *sometimes*. Strengthening it is its own change
|
||||
against its own issue (#2247); lowering the bar to turn the column green would
|
||||
throw away the only signal this feature adds.
|
||||
|
||||
⚠️ **Do not read a rise between two `RUNS=3` runs as an improvement.** One
|
||||
sample is 33 points there, and the two runs above straddle a 67-point swing on
|
||||
the same prompt with no change to the pointer. Note in particular that the
|
||||
turn-boundary rule added later can only ever *lower* a chained score — it
|
||||
rejects matches a flatter reading accepted — so a higher number after it is
|
||||
noise by construction, not an effect. `RUNS=5` is the smallest honest setting
|
||||
for a hand-off, and the cost is real: each sample is up to 14 turns.
|
||||
|
||||
⚠️ **Expect a hand-off to cost far more than a first move.** Each sample is up
|
||||
to 14 turns rather than one, so a chained case is the most expensive line in the
|
||||
|
|
@ -363,7 +372,7 @@ The summary is two lines, never one:
|
|||
|
||||
```
|
||||
7/7 first-move cases at or above 80%.
|
||||
0/2 hand-off cases above 50%.
|
||||
1/2 hand-off cases above 50%.
|
||||
```
|
||||
|
||||
Narrowing the run never narrows what a **negative** case is scored against — a
|
||||
|
|
|
|||
Loading…
Reference in a new issue