feat(skills): measure a skill reached from another skill (#2204)

`skills:eval` ran every case with `--max-turns 1`, so it measured one
thing: whether a skill is the model's FIRST tool call. That is right for a
skill a user reaches directly, and it leaves a whole class unmeasured — a
skill reached from inside another skill's body. `testing` tells the model
that picking a fixture is `/test-servers` and that it has to load it, and
`test-servers` scores 5/5 while every one of those five cases asks for it
by name. A skill only ever reachable through a hand-off would score a
clean 100% with the hand-off silently never firing.

An eval case is now one of two shapes, and exactly one:

  { "prompt": "…", "expect": "test-servers" }              first move, 1 turn
  { "prompt": "…", "chain": ["testing", "test-servers"] }  hand-off, 14 turns

A chain must END with the skill whose file it lives in. The case exists to
measure whether THIS skill is reachable, so the file that goes red is the
one belonging to the skill that stopped being reached; anchoring on the
first link would file a `test-servers` measurement under `testing`, where
a `test-servers` description edit would never be seen. Links are checked
against the repo's model-invoked set, which is why both `verify:skills`
and `collectCases` now validate in a second pass — `test-servers` sorts
before `testing`, so a per-directory check would reject a live chain as
unknown purely because of where the alphabet put it.

Scoring is an ordered SUBSEQUENCE, not a prefix and not a contiguous run:
the model may load something before the chain starts and something
unrelated in between, and neither changes the claim that A led to B.
`collectSkillInvocations` therefore returns an ordered array rather than a
Set — a B, A, B run has to stay distinguishable from one that never
reached B from A.

The two rates are reported in separate columns and never summed. A
hand-off is a second-hop load over many turns; a first-move rate is the
model's opening move. `CHAIN_MAX_TURNS` (14) and `CHAIN_THRESHOLD` are
their own knobs for the same reason, and `CHAIN_THRESHOLD` defaults to
**0.5** rather than inheriting 0.8: see the measurement below.

Read-only containment no longer leans on `--max-turns 1`, which was doing
much of it by itself. The deny list gains the agentic and network tools —
`Task` in particular, whose subagent the flag does not reach.

Measured, `skills:eval -- test-servers`:

  First move (1 turn)     7/7 at 100%, unchanged
  Hand-off (14 turns)     67% and 33% at RUNS=3

So the hand-off is real and unreliable — which is the fact nothing could
observe before this change, and the reason 0.5 rather than 0.8 is the
default bar: at 0.8 both committed cases are red no matter how strongly
the first skill points at the second, and the column stops carrying
signal. A `["pr-flow", "test-servers"]` probe measured 0% and was dropped
rather than kept: `pr-flow` says nothing about fixtures, so the case had
no lever short of broadening a description onto another skill's ground.
Both findings are written up in `docs/skill-authoring.md`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABdTnqyBodw6mHVVkGehqe
Signed-off-by: cliffhall <cliff@futurescale.com>

scripts/lib/skill-manifest.mjs

@@ -250,6 +250,31 @@ export function listingCost(skills) {
  */
 export const MIN_POSITIVE_CASES = 5;
 
+/**
+ * The fewest links a chained case may name.
+ *
+ * A one-link "chain" is a first-move case written the long way — it asserts
+ * nothing about a hand-off, and accepting it would let a skill satisfy the
+ * hand-off column without ever measuring one.
+ */
+export const MIN_CHAIN_LENGTH = 2;
+
+/**
+ * Whether a case measures a hand-off (`chain`) rather than a first move
+ * (`expect`).
+ *
+ * The two are scored against different turn budgets and reported in different
+ * columns, so every consumer has to tell them apart; doing it by field
+ * presence in one place keeps that decision from drifting between the
+ * validator and the runner.
+ *
+ * @param {unknown} c
+ * @returns {boolean}
+ */
+export function isChainCase(c) {
+  return c !== null && typeof c === "object" && Array.isArray(c.chain);
+}
+
 /**
  * Validate an `evals/evals.json` payload for a model-invoked skill.
  *
@@ -257,11 +282,30 @@ export const MIN_POSITIVE_CASES = 5;
  * is the failure nobody notices by hand — so negatives are required, not
  * optional.
  *
+ * A case is one of two shapes, and it must be exactly one: a **first-move**
+ * case names `expect` (the skill the model should reach with its first tool
+ * call, or `null`), and a **hand-off** case names `chain` — the ordered skills
+ * a single run should load, ending with this one. Requiring exactly one of the
+ * two rather than letting `chain` shadow `expect` means a case that carries
+ * both is a typo caught here, not a silently half-scored measurement.
+ *
+ * A hand-off case counts toward neither the positive floor nor the negative
+ * requirement. It measures a different thing (a second-hop load, over many
+ * turns) and it is scored in its own column, so letting one stand in for a
+ * first-move positive would let a skill ship with no measurement of the way
+ * users actually reach it.
+ *
  * @param {string} skillName
  * @param {unknown} cases Parsed JSON.
+ * @param {Set<string> | null} [known] Every model-invoked skill in the repo.
+ *   When supplied, each link of a `chain` is checked against it — a link naming
+ *   a skill that does not exist, or one the model cannot invoke, can never fire
+ *   and would score the case a permanent 0% that reads as a description
+ *   problem. Omitted (null), the link names are left unchecked, so a caller
+ *   that has not yet parsed the whole directory can still validate shape.
  * @returns {string[]} errors
  */
-export function validateEvalCases(skillName, cases) {
+export function validateEvalCases(skillName, cases, known = null) {
   if (!Array.isArray(cases) || cases.length === 0) {
     return ["evals.json must be a non-empty array of cases"];
   }
@@ -274,11 +318,23 @@ export function validateEvalCases(skillName, cases) {
     if (typeof c.prompt !== "string" || c.prompt.trim() === "") {
       errors.push(`case ${i}: \`prompt\` must be a non-empty string`);
     }
+    if ("expect" in c && "chain" in c) {
+      errors.push(
+        `case ${i}: carries both \`expect\` and \`chain\` — a case is either a first-move case or a hand-off case`,
+      );
+      return;
+    }
+    if ("chain" in c) {
+      errors.push(...validateChain(skillName, i, c.chain, known));
+      return;
+    }
     if (
       !("expect" in c) ||
       (c.expect !== null && typeof c.expect !== "string")
     ) {
-      errors.push(`case ${i}: \`expect\` must be a skill name or null`);
+      errors.push(
+        `case ${i}: needs \`expect\` (a skill name, or null for a negative case) or \`chain\``,
+      );
     } else if (c.expect !== null && c.expect !== skillName) {
       // A case living in this skill's evals may only expect THIS skill. A
       // foreign name passes the eval whenever that other skill fires, so the
@@ -325,6 +381,57 @@ export function validateEvalCases(skillName, cases) {
   return errors;
 }
 
+/**
+ * Validate one hand-off case's `chain`.
+ *
+ * The last link must be the owning skill, not the first. A hand-off case exists
+ * to measure whether **this** skill is reachable at all when nothing about the
+ * prompt names it, so the file that has to go red when the hand-off stops
+ * working is the one belonging to the skill that stops being reached. Anchoring
+ * on the first link instead would file the case under whichever skill happened
+ * to start the run, and a `test-servers` description edit would then be
+ * measured only inside `testing`.
+ *
+ * @param {string} skillName
+ * @param {number} i Case index, for the message.
+ * @param {unknown} chain
+ * @param {Set<string> | null} known
+ * @returns {string[]}
+ */
+function validateChain(skillName, i, chain, known) {
+  if (!Array.isArray(chain) || chain.length < MIN_CHAIN_LENGTH) {
+    return [
+      `case ${i}: \`chain\` must be an ordered array of at least ${MIN_CHAIN_LENGTH} skill names`,
+    ];
+  }
+  const errors = [];
+  if (chain.some((n) => typeof n !== "string" || n.trim() === "")) {
+    errors.push(`case ${i}: every \`chain\` link must be a non-empty string`);
+    return errors;
+  }
+  if (new Set(chain).size !== chain.length) {
+    // A repeated link cannot be observed: the run records which skills fired
+    // in what order, and a second load of one already recorded is
+    // indistinguishable from the first.
+    errors.push(`case ${i}: \`chain\` repeats a skill name`);
+  }
+  if (chain[chain.length - 1] !== skillName) {
+    errors.push(
+      `case ${i}: \`chain\` ends with \`${chain[chain.length - 1]}\`, but this file measures whether \`${skillName}\` is reached`,
+    );
+  }
+  if (known) {
+    for (const link of chain) {
+      if (!known.has(link)) {
+        errors.push(
+          `case ${i}: \`chain\` names \`${link}\`, which is not a model-invoked skill — it can never fire`,
+        );
+      }
+    }
+  }
+  return errors;
+}
+
 /**
  * Claude Code version the authoritative validator is pinned to when it has to
  * be fetched. Pinned rather than @latest: a validator that moves on its own can

scripts/lib/skill-manifest.test.mjs

@@ -9,6 +9,8 @@ import assert from "node:assert/strict";
 import {
   splitFrontmatter,
   parseSkill,
+  isChainCase,
+  MIN_CHAIN_LENGTH,
   MIN_POSITIVE_CASES,
   validateEvalCases,
   listingCost,
@@ -204,8 +206,116 @@ test("validateEvalCases rejects malformed cases", () => {
   );
   assert.match(
     validateEvalCases("x", [{ prompt: "a", expect: 7 }]).join(),
-    /expect. must be a skill name or null/,
+    /needs .expect./,
   );
+  assert.match(
+    validateEvalCases("x", [{ prompt: "a" }]).join(),
+    /needs .expect. .* or .chain./,
+  );
+});
+
+test("a hand-off case names an ordered chain ending in this skill", () => {
+  const base = [
+    ...Array.from({ length: MIN_POSITIVE_CASES }, (_, i) => ({
+      prompt: `p${i}`,
+      expect: "test-servers",
+    })),
+    { prompt: "n", expect: null },
+  ];
+  const withChain = (chain) => [...base, { prompt: "c", chain }];
+  const known = new Set(["testing", "test-servers", "pr-flow"]);
+
+  assert.deepEqual(
+    validateEvalCases(
+      "test-servers",
+      withChain(["testing", "test-servers"]),
+      known,
+    ),
+    [],
+  );
+  // The case belongs to the skill that must be REACHED, so a chain anchored on
+  // its first link would file a `test-servers` measurement under `testing` and
+  // leave a `test-servers` description edit unmeasured by its own file.
+  assert.match(
+    validateEvalCases(
+      "test-servers",
+      withChain(["test-servers", "testing"]),
+      known,
+    ).join(),
+    /ends with .testing., but this file measures .* .test-servers./,
+  );
+  assert.match(
+    validateEvalCases(
+      "test-servers",
+      withChain(["test-servers"]),
+      known,
+    ).join(),
+    new RegExp(`at least ${MIN_CHAIN_LENGTH} skill names`),
+  );
+  assert.match(
+    validateEvalCases(
+      "test-servers",
+      withChain(["testing", "testing", "test-servers"]),
+      known,
+    ).join(),
+    /repeats a skill name/,
+  );
+  assert.match(
+    validateEvalCases(
+      "test-servers",
+      withChain(["nope", "test-servers"]),
+      known,
+    ).join(),
+    /.nope., which is not a model-invoked skill/,
+  );
+  assert.match(
+    validateEvalCases(
+      "test-servers",
+      withChain(["", "test-servers"]),
+      known,
+    ).join(),
+    /non-empty string/,
+  );
+  // Without the known set the shape is still checked; only the link names go
+  // unverified, so a caller that has not parsed the directory can still run.
+  assert.deepEqual(
+    validateEvalCases("test-servers", withChain(["nope", "test-servers"])),
+    [],
+  );
+});
+
+test("a hand-off case satisfies neither floor and never doubles as a first move", () => {
+  // A hand-off is a second-hop load over many turns and is scored in its own
+  // column. Letting one stand in for a first-move positive would let a skill
+  // ship with no measurement of the way users actually reach it.
+  const chained = Array.from({ length: MIN_POSITIVE_CASES + 1 }, (_, i) => ({
+    prompt: `c${i}`,
+    chain: ["testing", "test-servers"],
+  }));
+  const errors = validateEvalCases("test-servers", chained).join(" ");
+  assert.match(errors, /no positive case/);
+  assert.match(errors, /no negative case/);
+});
+
+test("a case carrying both shapes is a typo, not a half-scored measurement", () => {
+  assert.match(
+    validateEvalCases("test-servers", [
+      {
+        prompt: "c",
+        expect: "test-servers",
+        chain: ["testing", "test-servers"],
+      },
+    ]).join(),
+    /carries both .expect. and .chain./,
+  );
+});
+
+test("isChainCase tells the two shapes apart", () => {
+  assert.equal(isChainCase({ chain: ["a", "b"] }), true);
+  assert.equal(isChainCase({ expect: "a" }), false);
+  assert.equal(isChainCase({ expect: null }), false);
+  assert.equal(isChainCase(null), false);
+  assert.equal(isChainCase("chain"), false);
 });
 
 test("parseClaudeVersion reads the CLI's version banner", () => {