modelcontextprotocol/conformance
Watch
1
0
Fork
You've already forked conformance
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix Windows compatibility in tier-check client conformance (#168)

Browse source 
* Fix Windows compatibility in tier-check client conformance

Two issues prevented client conformance tests from running on Windows:

1. Single-quote quoting in execSync: The --command argument was wrapped in single quotes, which cmd.exe treats as literal characters rather than string delimiters. This caused the CLI to receive only the first word of the command. Switched to execFileSync with an args array to bypass shell quoting entirely.

2. Backslash path separators in parseOutputDir: dirname() on Windows returns auth\scenario-name but reconcileWithExpected compares against auth/scenario-name, causing auth scenarios to be double-counted. Normalize backslashes to forward slashes in parsed scenario names.

* fix: use process.argv[1] for CLI path in subprocess calls

Combines the npx-compatibility fix from #175 with this PR's execFileSync
approach. process.argv[1] resolves to the absolute path of the running CLI
regardless of working directory, so tier-check works when invoked via npx
or from any cwd.

---------

Co-authored-by: Paul Carleton <paulc@anthropic.com>
This commit is contained in:
Jeff Handley 2026-03-25 05:03:34 -07:00 • committed by GitHub
commit a2ec2d44ad
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 17 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

23
src/tier-check/checks/test-conformance-results.ts
View file

@ -1,4 +1,4 @@
import { execSync } from 'child_process';
import { execFileSync } from 'child_process';
import { mkdtempSync, readFileSync, existsSync, globSync } from 'fs';
import { join, dirname } from 'path';
import { tmpdir } from 'os';
@ -43,7 +43,7 @@ function parseOutputDir(outputDir: string): ConformanceResult {
const checksFiles = globSync('**/checks.json', { cwd: outputDir });
for (const checksFile of checksFiles) {
const scenarioName = dirname(checksFile);
const scenarioName = dirname(checksFile).replace(/\\/g, '/');
const checksPath = join(outputDir, checksFile);
try {
@ -181,8 +181,9 @@ export async function checkConformance(options: {
const outputDir = mkdtempSync(join(tmpdir(), 'tier-check-server-'));
try {
execSync(
`node "${process.argv[1]}" server --url ${options.serverUrl} -o ${outputDir}`,
execFileSync(
process.execPath,
[process.argv[1], 'server', '--url', options.serverUrl, '-o', outputDir],
{
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 120_000
@ -220,8 +221,18 @@ export async function checkClientConformance(options: {
const outputDir = mkdtempSync(join(tmpdir(), 'tier-check-client-'));
try {
execSync(
`node "${process.argv[1]}" client --command '${options.clientCmd}' --suite all -o ${outputDir}`,
execFileSync(
process.execPath,
[
process.argv[1],
'client',
'--command',
options.clientCmd,
'--suite',
'all',
'-o',
outputDir
],
{
stdio: ['pipe', 'pipe', 'pipe'],
timeout: 120_000