feat: flag unportable tool schemas in all three clients

Closes #1005

A tool schema can be perfectly legal JSON Schema and still be rejected by
the client the server is meant to run against — Go's `jsonschema` emits a
bare `true` for `interface{}`, which the SDK accepts and Claude Code
refuses with an opaque "Invalid input". The Inspector passed it through
silently.

Adds `core/json/schemaLint.ts`, a portability lint over a tool's
inputSchema/outputSchema, and surfaces one shared verdict everywhere:
the CLI's `--strict` report (exit 6 on an error-severity finding), the
TUI's tool list marker + detail pane, and the web Tools sidebar flag +
detail section.

Deliberately not a JSON Schema validator: a census of 617 public servers
found zero that fail the SDK's own parse, so a conformance check would
report nothing on real servers. Each rule is instead a construct that is
legal JSON Schema and known to be refused or mishandled by a shipping
client — a bare boolean in a schema position, an array-form `type`, a
remote `$ref`, a schema with no validation keyword, a non-object root.

`test-servers/configs/unportable-schemas-http.json` exercises every rule,
via a new `rawToolSchemas` override — the Zod-built presets cannot emit
any of these shapes, which is the same reason a real server only hits
this when its schemas come from another generator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UL9retfvAXRgvi6EWk4SY4
Signed-off-by: cliffhall <cliff@futurescale.com>
This commit is contained in:
cliffhall 2026-08-25 01:18:53 -04:00
commit 2f9e0901c1
7 changed files with 1979 additions and 13 deletions

View file

@ -120,6 +120,7 @@ Options that specify the MCP server (catalog/config file, ad-hoc command/URL, en
| `--tool-metadata <key=value>` | Tool-specific `_meta` entries for `tools/call`. Same JSON-parsed value handling as `--metadata`. |
| `--connect-timeout <ms>` | Connection timeout in ms. Defaults to `15000` for ad-hoc `--server-url`/target runs (so a black-holed host fails fast) and to the file-level timeout for `--catalog`/`--config` runs. `0` disables the timeout. |
| `--app-info` | Probe a tool's MCP App UI metadata without invoking it. With `--method tools/call --tool-name <name>`: prints one JSON line (`hasApp`, `resourceUri`, `csp`, `permissions`, `domain`, …) and exits `0` if the tool has an app or `2` (`no_app`) if not. With `--method tools/list`: emits NDJSON — one app-info line per tool over a single connection. |
| `--strict` | With `--method tools/list`: report tool-schema portability problems in full (path, issue, suggested fix) on stderr, and exit `6` if any is error-severity. Without it, a one-line count is printed instead. See [Schema portability](#schema-portability---strict). |
| `--format <text\|json>` | Output format. `text` (default) pretty-prints the result. `json` emits a single JSON object on stdout (`{ "result": … }`, plus `{ "appInfo": … }` as a sibling key for App tools) with no banners, so the whole output pipes cleanly into `jq`. |
| `--relogin` | Delete stored OAuth for this server URL from the shared store before connect; interactive login still only runs if the server requires auth. Requires an HTTP/SSE URL (rejected for stdio). Conflicts with `--stored-auth-only` / `--use-stored-auth` / `--wait-for-auth` / catalog short-circuits. |
| `--stored-auth-only` | **CI / non-interactive safe:** never start interactive OAuth / step-up (and never auto-open a browser); use the shared store if present, otherwise fail immediately with `auth_required`. Prefer this over a bare pipe/CI run that would otherwise attempt interactive login. |
@ -152,6 +153,50 @@ mcp-inspector --cli <server> --method tools/call --tool-name my_app_tool --forma
A `tools/call` that returns `isError:true` still prints its payload but exits `5` (`tool_is_error`) so `&&` chains don't proceed on a failed call.
#### Schema portability (`--strict`)
A tool schema can be perfectly legal JSON Schema and still be refused by the
client the server is meant to run against. `--strict` (with `--method
tools/list`) names those constructs — path, what is wrong, and a concrete fix —
on **stderr**, and exits `6` if any is error-severity:
```bash
mcp-inspector --cli <server> --method tools/list --strict
```
```text
Error: tool "info"
Path: outputSchema.properties.data
Issue: Bare `true` used where a schema object is expected.
Suggestion: Spell the "anything" schema out — `{"type": "object", "additionalProperties": true}` …
1 error, 3 warnings across 2 tools.
```
Severity decides the exit code, not the report: **errors** are constructs a
shipping MCP client refuses outright (a bare `true` in a `properties` map, a
non-object schema root), **warnings** are ones handled unevenly (an array-form
`type`, a remote `$ref`, a schema carrying no validation keyword at all). Only
errors fail the run — a `--strict` that failed on warnings would be unusable as
a CI gate against servers that are in fact fine.
The report goes to stderr, so the result on stdout stays parseable. Under
`--format json` the findings are folded into the same envelope instead
(`{"result":…,"schemaFindings":[…]}`), so a caller reads one document rather
than correlating two.
**Without `--strict` nothing changes except one line.** A `tools/list` whose
schemas have findings prints a single stderr summary — `Schema portability: 1
error, 3 warnings across 2 tools. Re-run with --strict for details.` — and
still exits `0`. A clean list prints nothing at all.
This is deliberately not a JSON Schema validator. A census of 617 public
servers found **zero** that fail the SDK's own parser, so a conformance check
would report nothing on essentially every real server; what bites is the
narrower subset each consumer accepts, which is what these rules encode. The
same verdict drives the TUI's tool detail pane and the web Tools tab — all
three read `core/json/schemaLint`.
### CLI-specific (OAuth for HTTP servers)
The CLI runs the same loopback callback server as the TUI (`http://127.0.0.1:6276/oauth/callback` by default).
@ -250,6 +295,7 @@ prose from stderr:
| `3` | Server requires authentication (401/403, `WWW-Authenticate`, OAuth). |
| `4` | Server unreachable (DNS, connection refused, timeout, `fetch failed`). |
| `5` | Tool error (`tools/call` returned `isError:true`, or the tool was not found). |
| `6` | `--strict` found an error-severity tool-schema portability problem. |
On any non-zero exit the CLI also writes a single JSON line to **stderr** — the
`ErrorEnvelope`:

View file

@ -0,0 +1,251 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { emitResult, runCli } from "../src/cli.js";
import { CliExitCodeError, EXIT_CODES } from "../src/error-handler.js";
import {
lintListResult,
toolsFromResult,
writeSchemaLintReport,
} from "../src/handlers/schema-lint-report.js";
/** A `tools/list` result with one bare-`true` property — the issue's case. */
const DIRTY_LIST = {
tools: [
{
name: "info",
inputSchema: { type: "object", properties: {} },
outputSchema: { type: "object", properties: { data: true } },
},
],
};
/** Same shape, but every schema portable. */
const CLEAN_LIST = {
tools: [{ name: "ok", inputSchema: { type: "object", properties: {} } }],
};
describe("toolsFromResult", () => {
it("returns the tools array", () => {
expect(toolsFromResult(DIRTY_LIST).map((t) => t.name)).toEqual(["info"]);
});
it.each([
["a missing tools key", {}],
["a non-array tools value", { tools: "nope" }],
])("returns nothing for %s", (_label, result) => {
expect(toolsFromResult(result)).toEqual([]);
});
it("skips entries that are not tool-shaped", () => {
const result = {
tools: [null, 3, {}, { name: 7 }, { name: "real", inputSchema: {} }],
};
expect(toolsFromResult(result).map((t) => t.name)).toEqual(["real"]);
});
});
describe("lintListResult", () => {
it("reports the offending tool", () => {
const results = lintListResult(DIRTY_LIST);
expect(results.map((r) => r.toolName)).toEqual(["info"]);
});
it("reports nothing for a clean list", () => {
expect(lintListResult(CLEAN_LIST)).toEqual([]);
});
});
describe("writeSchemaLintReport", () => {
let stderr: string;
let originalWrite: typeof process.stderr.write;
beforeEach(() => {
stderr = "";
originalWrite = process.stderr.write;
process.stderr.write = ((chunk: unknown): boolean => {
stderr += typeof chunk === "string" ? chunk : String(chunk);
return true;
}) as typeof process.stderr.write;
});
afterEach(() => {
process.stderr.write = originalWrite;
});
it("writes nothing at all when there are no findings", () => {
writeSchemaLintReport([], true);
expect(stderr).toBe("");
});
it("writes a one-line hint without --strict", () => {
writeSchemaLintReport(lintListResult(DIRTY_LIST), false);
expect(stderr.trimEnd().split("\n")).toHaveLength(1);
expect(stderr).toContain("Re-run with --strict");
});
it("writes the full report under --strict", () => {
writeSchemaLintReport(lintListResult(DIRTY_LIST), true);
expect(stderr).toContain('Error: tool "info"');
expect(stderr).toContain("Path: outputSchema.properties.data");
expect(stderr).toContain("Suggestion: ");
expect(stderr).not.toContain("Re-run with --strict");
});
});
describe("emitResult — schema lint wiring (#1005)", () => {
let stdout: string;
let stderr: string;
let originalOut: typeof process.stdout.write;
let originalErr: typeof process.stderr.write;
beforeEach(() => {
stdout = "";
stderr = "";
originalOut = process.stdout.write;
originalErr = process.stderr.write;
process.stdout.write = ((chunk: unknown, ...rest: unknown[]): boolean => {
stdout += typeof chunk === "string" ? chunk : String(chunk);
(
rest.find((r) => typeof r === "function") as (() => void) | undefined
)?.();
return true;
}) as typeof process.stdout.write;
process.stderr.write = ((chunk: unknown): boolean => {
stderr += typeof chunk === "string" ? chunk : String(chunk);
return true;
}) as typeof process.stderr.write;
});
afterEach(() => {
process.stdout.write = originalOut;
process.stderr.write = originalErr;
});
it("exits 6 under --strict when a finding is error-severity", async () => {
const promise = emitResult(DIRTY_LIST, undefined, {
method: "tools/list",
strict: true,
format: "text",
});
await expect(promise).rejects.toBeInstanceOf(CliExitCodeError);
await promise.catch((e: CliExitCodeError) => {
expect(e.exitCode).toBe(EXIT_CODES.SCHEMA_INVALID);
expect(e.envelope?.code).toBe("schema_invalid");
});
// The result still goes to stdout; only the report is on stderr.
expect(JSON.parse(stdout)).toEqual(DIRTY_LIST);
expect(stderr).toContain("Path: outputSchema.properties.data");
});
it("succeeds under --strict when only warnings were found", async () => {
const warnOnly = {
tools: [
{
name: "w",
inputSchema: {
type: "object",
properties: { a: { type: ["null", "boolean"] } },
},
},
],
};
await expect(
emitResult(warnOnly, undefined, {
method: "tools/list",
strict: true,
format: "text",
}),
).resolves.toBeUndefined();
expect(stderr).toContain('Warning: tool "w"');
});
it("prints only the hint, and does not fail, without --strict", async () => {
await expect(
emitResult(DIRTY_LIST, undefined, {
method: "tools/list",
format: "text",
}),
).resolves.toBeUndefined();
expect(stderr).toContain("Re-run with --strict");
expect(stderr).not.toContain("Suggestion:");
});
it("folds findings into the --format json envelope under --strict", async () => {
await emitResult(DIRTY_LIST, undefined, {
method: "tools/list",
strict: true,
format: "json",
}).catch(() => {
// exit 6 is expected here; the envelope is what this test asserts.
});
const envelope = JSON.parse(stdout) as {
result: unknown;
schemaFindings?: { toolName: string }[];
};
expect(envelope.result).toEqual(DIRTY_LIST);
expect(envelope.schemaFindings?.[0]?.toolName).toBe("info");
});
it("leaves the json envelope alone when nothing was found", async () => {
await emitResult(CLEAN_LIST, undefined, {
method: "tools/list",
strict: true,
format: "json",
});
expect(JSON.parse(stdout)).toEqual({ result: CLEAN_LIST });
expect(stderr).toBe("");
});
it("does not lint a method other than tools/list", async () => {
// The same payload under `tools/call` must produce no report at all —
// `--strict` is a tools/list flag and the CLI rejects it elsewhere.
await emitResult(DIRTY_LIST, undefined, {
method: "tools/call",
strict: true,
format: "text",
});
expect(stderr).toBe("");
});
it("does not lint an --app-info run", async () => {
await emitResult(
DIRTY_LIST,
{ hasApp: true, toolName: "info" },
{ method: "tools/list", strict: true, appInfo: true, format: "text" },
);
expect(stderr).toBe("");
});
it("reports isError before the schema verdict", async () => {
// Both would throw; the tool-level failure is the more specific one and
// must win, so a caller branching on exit 5 is not shadowed by exit 6.
const promise = emitResult({ ...DIRTY_LIST, isError: true }, undefined, {
method: "tools/list",
strict: true,
toolName: "info",
format: "text",
});
await promise.catch((e: CliExitCodeError) => {
expect(e.exitCode).toBe(EXIT_CODES.TOOL_ERROR);
});
await expect(promise).rejects.toBeInstanceOf(CliExitCodeError);
});
});
describe("--strict argument validation", () => {
it("is rejected with a method other than tools/list", async () => {
await expect(
runCli([
"node",
"cli",
"--cli",
"--method",
"tools/call",
"--tool-name",
"x",
"--strict",
"--server-url",
"http://127.0.0.1:1/mcp",
]),
).rejects.toThrow("--strict requires --method tools/list.");
});
});

View file

@ -684,6 +684,10 @@ async function parseArgs(argv?: string[]): Promise<ParseResult> {
"--app-info",
"Probe the tool's MCP App UI metadata (resourceUri, csp, permissions, domain) and emit it as one JSON line; exit 2 when the tool has no app. Use with --method tools/call --tool-name <name> (the tool itself is not invoked) or --method tools/list (one NDJSON line per tool).",
)
.option(
"--strict",
"Report tool-schema portability problems in full (path, issue, suggested fix) on stderr, and exit 6 if any is error-severity. Use with --method tools/list. Without it, a one-line count is printed instead.",
)
.option(
"--connect-timeout <ms>",
`Connection timeout in ms (default ${DEFAULT_CONNECT_TIMEOUT_MS} for ad-hoc --server-url / target invocations; 0 = no timeout).`,
@ -784,6 +788,7 @@ async function parseArgs(argv?: string[]): Promise<ParseResult> {
serverUrl?: string;
header?: Record<string, string>;
appInfo?: boolean;
strict?: boolean;
connectTimeout?: number;
format?: OutputFormat;
toolArgsJson?: string;
@ -1004,6 +1009,10 @@ async function parseArgs(argv?: string[]): Promise<ParseResult> {
);
}
if (options.strict && options.method !== "tools/list") {
throw new Error("--strict requires --method tools/list.");
}
// --tool-args-json passes arguments verbatim with no key=value coercion (so
// `"012"` stays a string and nested objects work without shell escaping).
let toolArg = options.toolArg;
@ -1047,6 +1056,7 @@ async function parseArgs(argv?: string[]): Promise<ParseResult> {
metadata: options.metadata,
toolMeta: options.toolMetadata,
appInfo: options.appInfo === true,
strict: options.strict === true,
format: options.format,
};

View file

@ -8,6 +8,7 @@
* - 3: server requires authentication (401 / WWW-Authenticate / OAuth)
* - 4: server unreachable (DNS, connect refused, timeout, fetch failure)
* - 5: tool error (`tools/call` returned `isError:true`, or tool not found)
* - 6: `--strict` found an error-severity tool-schema portability finding
*/
export const EXIT_CODES = {
OK: 0,
@ -16,6 +17,7 @@ export const EXIT_CODES = {
AUTH_REQUIRED: 3,
UNREACHABLE: 4,
TOOL_ERROR: 5,
SCHEMA_INVALID: 6,
} as const;
/** Machine-readable error envelope written as one JSON line on stderr. */

View file

@ -1,10 +1,13 @@
import { awaitableLog } from "../utils/awaitable-log.js";
import { CliExitCodeError, EXIT_CODES } from "../error-handler.js";
import { lintListResult, writeSchemaLintReport } from "./schema-lint-report.js";
import { countFindings } from "@inspector/core/json/schemaLint.js";
import type { CliAppInfo, McpResponse, MethodArgs } from "./method-types.js";
/**
* Write the method result (and any app-info) to stdout, honouring `--format`
* and `--app-info`, then map `isError`/no-app outcomes onto the exit-code map.
* and `--app-info`, then map `isError`/no-app/schema outcomes onto the
* exit-code map.
*/
export async function emitResult(
result: McpResponse,
@ -28,14 +31,23 @@ export async function emitResult(
return;
}
// Lint before writing, because `--format json --strict` folds the findings
// into the same envelope as the result rather than emitting a second
// document a caller would have to correlate.
const lint =
args.method === "tools/list" ? lintListResult(result) : undefined;
if (json) {
const envelope: Record<string, unknown> = { result };
if (appInfo?.hasApp) envelope.appInfo = appInfo;
if (args.strict && lint && lint.length > 0) envelope.schemaFindings = lint;
await awaitableLog(JSON.stringify(envelope) + "\n");
} else {
await awaitableLog(JSON.stringify(result, null, 2) + "\n");
}
if (lint) writeSchemaLintReport(lint, args.strict === true);
if ((result as { isError?: unknown }).isError === true) {
throw new CliExitCodeError(
EXIT_CODES.TOOL_ERROR,
@ -43,4 +55,18 @@ export async function emitResult(
{ code: "tool_is_error" },
);
}
// Only `--strict` fails the run. Warnings never do: they mark constructs
// that are handled unevenly rather than refused, so failing on them would
// make `--strict` unusable as a CI gate on servers that are in fact fine.
if (args.strict && lint) {
const { errors } = countFindings(lint);
if (errors > 0) {
throw new CliExitCodeError(
EXIT_CODES.SCHEMA_INVALID,
`${errors} tool schema error${errors === 1 ? "" : "s"} found (--strict).`,
{ code: "schema_invalid" },
);
}
}
}

View file

@ -24,6 +24,11 @@ export type MethodArgs = {
toolMeta?: RequestMetadata;
metadata?: RequestMetadata;
appInfo?: boolean;
/**
* `--strict`: report tool-schema portability findings in full and exit
* non-zero when any is error-severity (#1005). `tools/list` only.
*/
strict?: boolean;
format?: OutputFormat;
/** Task id for tasks/get, tasks/cancel, tasks/result. */
taskId?: string;

View file

@ -0,0 +1,62 @@
import type { Tool } from "@modelcontextprotocol/client";
import {
formatSchemaLintReport,
lintTools,
summarizeFindings,
type ToolSchemaFindings,
} from "@inspector/core/json/schemaLint.js";
import type { McpResponse } from "./method-types.js";
/**
* Read the `tools` array out of a `tools/list` result. The result is typed as
* an opaque `McpResponse` at this layer, and a server can return anything, so
* each entry is narrowed to "an object with a string `name`" before it is
* treated as a {@link Tool}. Everything else is skipped rather than crashing
* the report — an unlintable entry is not the CLI's problem to surface here.
*/
export function toolsFromResult(result: McpResponse): Tool[] {
const tools = (result as { tools?: unknown }).tools;
if (!Array.isArray(tools)) return [];
return tools.filter(
(t): t is Tool =>
typeof t === "object" &&
t !== null &&
typeof (t as { name?: unknown }).name === "string",
);
}
/**
* Lint the tools in a `tools/list` result (#1005).
*
* Always run, `--strict` or not — it is a pure in-memory walk over a list the
* CLI already holds, and the non-strict hint below needs the count. What
* `--strict` changes is what happens with the result, in
* {@link writeSchemaLintReport}.
*/
export function lintListResult(result: McpResponse): ToolSchemaFindings[] {
return lintTools(toolsFromResult(result));
}
/**
* Write the schema-lint outcome to **stderr**, so it never contaminates the
* result on stdout that a `--format json` consumer is parsing.
*
* Under `--strict` this is the full report from the issue — path, issue,
* suggestion, one block per finding. Without it, a single line naming the
* count and how to see the detail: a server author who has not asked for the
* lint should still learn it found something, but a multi-page report nobody
* requested would be worse than silence.
*/
export function writeSchemaLintReport(
results: readonly ToolSchemaFindings[],
strict: boolean,
): void {
if (results.length === 0) return;
if (!strict) {
process.stderr.write(
`Schema portability: ${summarizeFindings(results)}. Re-run with --strict for details.\n`,
);
return;
}
process.stderr.write(`${formatSchemaLintReport(results)}\n`);
}