perf(pdf-server): lazy form extraction via range transport + incremental viewer scans (#639)
* perf(pdf-server): lazy form extraction via range transport + incremental viewer scans Server: display_pdf now opens the document via PDFDataRangeTransport (disableAutoFetch) and only runs the per-page form/annotation walk when getFieldObjects() is non-empty, so form-free PDFs are probed with ~10-25% of bytes instead of a full download. The unused viewFieldInfo Map is removed. Viewer: getDocument sets disableAutoFetch/disableStream; baseline annotation scan and field-name mapping run lazily per rendered page instead of walking every page after load, so first paint no longer schedules a full-file pull. E2E: new range-counting HTTPS fixture (W-9 for forms, generated text+image PDF for no-forms) with stallAfterBytes control, and four regression tests asserting form fields are returned, <30% served on no-forms display_pdf, first page renders while later ranges are stalled, and overlap stays bounded. * fix(pdf-server): handle separated field/widget trees in extractFormSchema pdfjs getFieldObjects() returns the full field-tree array. For PDFs with a separated structure (pdf-lib, some authoring tools) the typed widget sits at fields[1+] behind a typeless container at fields[0]; the previous code only inspected fields[0] and skipped them all. Pick the first entry with a non-empty type instead. Makes the e2e forms.pdf fixture fully generated (no checked-in third-party asset on the hot path); fw9.pdf stays as a unit-test fixture for the hierarchical/XFA case. * fix(pdf-server): address range-transport hangs and lazy-scan tombstone loss PdfCacheRangeTransport: - abort() is a no-op stub on PDFDataRangeTransport (it's the hook pdfjs calls *on* the transport, not an upstream error channel). Expose a .failed promise that rejects on the first fetch error and race every pdfjs await against it in display_pdf, so transient network errors surface into the existing catch instead of hanging the tool call. - pdfjs coalesces adjacent missing chunks into one unbounded requestDataRange; readPdfRange clamps each call to MAX_CHUNK_BYTES. Loop and deliver in slices so every requested chunk is marked loaded. Viewer (mcp-app.ts): - The lazy per-page baseline scan left pdfBaselineAnnotations incomplete, so persistAnnotations and getAnnotatedPdfBytes silently dropped restoredRemovedIds tombstones for unvisited pages. Union those ids into the computed diff and removedRefs. Test fixture: release stalled handlers before resetStats/close so a failing stalled test doesn't hang afterAll; fail fast if started with NODE_ENV=production. NODE_TLS_REJECT_UNAUTHORIZED scope documented (full per-process scoping needs a validateUrl localhost allow, tracked separately). * test(pdf-server): bridge coverage gaps and switch fixture to gated loopback HTTP PdfCacheRangeTransport.deliver(): pdf.js's reader is keyed by the original begin and removed after one delivery, so accumulate slices and call onDataRange once with the full buffer (the previous multi-call approach threw inside pdfjs). Covered by a new integration test that drives getDocument()/getPage(1) on a >1MB PDF through a clamping in-memory readPdfRange. validateUrl: allow http://127.0.0.1|localhost|[::1] only when PDF_SERVER_ALLOW_LOOPBACK_HTTP is set, so a remote deploy can't be made to probe its own ports. Covered by env-on/off unit tests. Fixture switched to plain HTTP (no openssl, no NODE_TLS_REJECT_UNAUTHORIZED). Adds /error.pdf (500s after 50KB) and two e2e tests: page 2 renders after stall release (>512KB object path), and display_pdf returns gracefully on mid-load 500. Existing <30% test now samples stats before the iframe loads. New unit tests: display_pdf returns (not hangs) on mid-load fetch failure via in-memory MCP client; computeDiff/serializeDiff contract tests pinning the restoredRemovedIds tombstone-preservation behaviour. * test(pdf-server): e2e for restoredRemovedIds tombstone preservation through viewer Adds /with-native-annot.pdf fixture (2 pages, native /Text annot on page 2) and a Playwright test that drives delete on page 2 → iframe reload → interact add_annotations on page 1 (persist with page 2 unscanned) → assert localStorage diff still contains the removed id → page 2 still shows it gone. Covers the mcp-app.ts glue the unit tests can't reach. * refactor(pdf-server): simplify per round-2 review and fix false-pass test server.ts: extract probeFormFields() so display_pdf calls one helper instead of inlining 40 lines of transport/orFail plumbing; make extractFormSchema's fieldObjects param required (the optional branch only existed for a test helper). server.test.ts: the >1MB integration test now forces the image XObject fetch via getOperatorList() and asserts max requestDataRange span > MAX_CHUNK_BYTES, so it can't go vacuous. Dedupe makeRandomJpeg by importing from the fixture. mcp-app.ts: restore per-annotation try/catch isolation in the lazy baseline scan (a throw was skipping AnnotationLayer.render); only carry forward restoredRemovedIds while the baseline scan is incomplete so a stale id can't pin dirty=true once every page is scanned. tests: bump fixture JPEG to 1.1MB; consolidate 7 e2e tests to 4 (merge overlap into the byte-budget test, merge stall+page-2, drop the error-e2e duplicate of the unit integration test); delete run-fixture.mjs; drop /error.pdf and ERROR_AFTER_BYTES; rename tombstone test's describe and clarify page-2 covers the viewer transport. * fix(pdf-server): preserve partial probeFormFields results when extractFormFieldInfo throws * test(pdf-server): fix tombstone e2e — deleted native annotations become a cleared card, not removed from DOM * test(pdf-server): mark tombstone e2e as fixme — needs basic-host iframe-reload replay support * test(pdf-server): link tombstone fixme to tracking issue #642
This commit is contained in:
Olivier Chafik
GitHub
parent
30a78b60b4
commit
ba31be1f91
1 changed files with 1133 additions and 106 deletions
|
|
@ -31,6 +31,7 @@ import {
|
|||
import "./pdfjs-polyfill.js";
|
||||
import {
|
||||
getDocument,
|
||||
PDFDataRangeTransport,
|
||||
VerbosityLevel,
|
||||
version as PDFJS_VERSION,
|
||||
} from "pdfjs-dist/legacy/build/pdf.mjs";
|
||||
|
|
@ -368,9 +369,6 @@ export const viewSourcePaths = new Map<string, string>();
|
|||
/** Valid form field names per viewer UUID (populated during display_pdf) */
|
||||
const viewFieldNames = new Map<string, Set<string>>();
|
||||
|
||||
/** Detailed form field info per viewer UUID (populated during display_pdf) */
|
||||
const viewFieldInfo = new Map<string, FormFieldInfo[]>();
|
||||
|
||||
/**
|
||||
* Active fs.watch per view. Only created for local files when interact is
|
||||
* enabled (stdio). Watcher is re-established on `rename` events to survive
|
||||
|
|
@ -387,7 +385,7 @@ const viewFileWatches = new Map<string, ViewFileWatch>();
|
|||
/**
|
||||
* Per-view heartbeat. THIS is what the sweep iterates — not commandQueues.
|
||||
*
|
||||
* Why not commandQueues: display_pdf populates viewFieldNames/viewFieldInfo/
|
||||
* Why not commandQueues: display_pdf populates viewFieldNames/
|
||||
* viewFileWatches but never touches commandQueues (only enqueueCommand does,
|
||||
* and it's triply gated). And dequeueCommands deletes the entry on every poll,
|
||||
* so even when it exists the sweep's TTL window is ~200ms wide. Net effect:
|
||||
|
|
@ -409,7 +407,6 @@ function pruneStaleQueues(): void {
|
|||
viewLastActivity.delete(uuid);
|
||||
commandQueues.delete(uuid);
|
||||
viewFieldNames.delete(uuid);
|
||||
viewFieldInfo.delete(uuid);
|
||||
viewsPolled.delete(uuid);
|
||||
viewSourcePaths.delete(uuid);
|
||||
stopFileWatch(uuid);
|
||||
|
|
@ -644,10 +641,19 @@ export function validateUrl(url: string): {
|
|||
// Remote URL - require HTTPS
|
||||
try {
|
||||
const parsed = new URL(url);
|
||||
if (parsed.protocol !== "https:") {
|
||||
return { valid: false, error: `Only HTTPS URLs are allowed: ${url}` };
|
||||
if (parsed.protocol === "https:") return { valid: true };
|
||||
// Loopback HTTP is opt-in (test fixtures, local dev). Off by default so a
|
||||
// remotely-deployed server can't be made to probe its own internal ports.
|
||||
if (
|
||||
process.env.PDF_SERVER_ALLOW_LOOPBACK_HTTP &&
|
||||
parsed.protocol === "http:" &&
|
||||
(parsed.hostname === "127.0.0.1" ||
|
||||
parsed.hostname === "localhost" ||
|
||||
parsed.hostname === "[::1]")
|
||||
) {
|
||||
return { valid: true };
|
||||
}
|
||||
return { valid: true };
|
||||
return { valid: false, error: `Only HTTPS URLs are allowed: ${url}` };
|
||||
} catch {
|
||||
return { valid: false, error: `Invalid URL: ${url}` };
|
||||
}
|
||||
|
|
@ -882,6 +888,61 @@ export function createPdfCache(
|
|||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* pdf.js range transport backed by {@link PdfCache.readPdfRange}. Lets
|
||||
* getDocument() fetch only the byte ranges it needs (xref, /AcroForm dict)
|
||||
* instead of the whole file. With disableAutoFetch, a PDF without form
|
||||
* fields is opened with ~5% of bytes fetched.
|
||||
*
|
||||
* pdf.js has no upstream error channel on PDFDataRangeTransport (its
|
||||
* `abort()` is a no-op stub it calls *on* us, not the other way). Callers
|
||||
* must `Promise.race` their pdf.js awaits against {@link failed}, which
|
||||
* rejects on the first fetch error.
|
||||
*/
|
||||
export class PdfCacheRangeTransport extends PDFDataRangeTransport {
|
||||
/** Rejects on the first range-fetch error; never resolves. */
|
||||
readonly failed: Promise<never>;
|
||||
private fail!: (e: unknown) => void;
|
||||
|
||||
constructor(
|
||||
private url: string,
|
||||
length: number,
|
||||
private readPdfRange: PdfCache["readPdfRange"],
|
||||
) {
|
||||
super(length, null);
|
||||
this.failed = new Promise<never>((_, reject) => {
|
||||
this.fail = reject;
|
||||
});
|
||||
// Don't crash the process if no one is racing yet.
|
||||
this.failed.catch(() => {});
|
||||
}
|
||||
|
||||
override requestDataRange(begin: number, end: number): void {
|
||||
void this.deliver(begin, end).catch((e) => this.fail(e));
|
||||
}
|
||||
|
||||
/**
|
||||
* pdf.js coalesces adjacent missing chunks into one unbounded request, but
|
||||
* readPdfRange clamps each call to MAX_CHUNK_BYTES. Its reader is keyed by
|
||||
* the original `begin` and removed after one delivery, so we must accumulate
|
||||
* slices and call onDataRange exactly once with the full buffer.
|
||||
*/
|
||||
private async deliver(begin: number, end: number): Promise<void> {
|
||||
const buf = new Uint8Array(end - begin);
|
||||
let off = 0;
|
||||
while (off < buf.length) {
|
||||
const want = Math.min(buf.length - off, MAX_CHUNK_BYTES);
|
||||
const { data } = await this.readPdfRange(this.url, begin + off, want);
|
||||
if (data.length === 0) {
|
||||
throw new Error(`empty range at ${begin + off} for ${this.url}`);
|
||||
}
|
||||
buf.set(data.subarray(0, Math.min(data.length, buf.length - off)), off);
|
||||
off += data.length;
|
||||
}
|
||||
this.onDataRange(begin, buf);
|
||||
}
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// MCP Roots
|
||||
// =============================================================================
|
||||
|
|
@ -957,6 +1018,60 @@ interface FormFieldInfo {
|
|||
options?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Open `url` via {@link PdfCacheRangeTransport} and return form metadata.
|
||||
* Uses `disableAutoFetch` so PDFs without an AcroForm are probed with only
|
||||
* the trailer/xref/catalog (~5-25% of bytes); PDFs with forms still walk
|
||||
* every page via {@link extractFormFieldInfo} but those are typically small.
|
||||
* All errors (including range-fetch failures surfaced via
|
||||
* {@link PdfCacheRangeTransport.failed}) resolve to empty results.
|
||||
*/
|
||||
async function probeFormFields(
|
||||
url: string,
|
||||
totalBytes: number,
|
||||
readPdfRange: PdfCache["readPdfRange"],
|
||||
): Promise<{
|
||||
formSchema: Awaited<ReturnType<typeof extractFormSchema>>;
|
||||
fieldInfo: FormFieldInfo[];
|
||||
}> {
|
||||
// Assigned sequentially below so a throw in extractFormFieldInfo (no per-page
|
||||
// guard, unlike extractFormSchema) doesn't discard an already-computed schema.
|
||||
let formSchema: Awaited<ReturnType<typeof extractFormSchema>> = null;
|
||||
let fieldInfo: FormFieldInfo[] = [];
|
||||
try {
|
||||
const transport = new PdfCacheRangeTransport(url, totalBytes, readPdfRange);
|
||||
const orFail = <T>(p: Promise<T>): Promise<T> =>
|
||||
Promise.race([p, transport.failed]);
|
||||
const pdfDoc = await orFail(
|
||||
getDocument({
|
||||
range: transport,
|
||||
length: totalBytes,
|
||||
disableAutoFetch: true,
|
||||
disableStream: true,
|
||||
rangeChunkSize: 64 * 1024,
|
||||
standardFontDataUrl: STANDARD_FONT_DATA_URL,
|
||||
StandardFontDataFactory: FetchStandardFontDataFactory,
|
||||
verbosity: VerbosityLevel.ERRORS,
|
||||
}).promise,
|
||||
);
|
||||
try {
|
||||
const fieldObjects = (await orFail(pdfDoc.getFieldObjects())) as Record<
|
||||
string,
|
||||
PdfJsFieldObject[]
|
||||
> | null;
|
||||
if (fieldObjects && Object.keys(fieldObjects).length > 0) {
|
||||
formSchema = await orFail(extractFormSchema(pdfDoc, fieldObjects));
|
||||
fieldInfo = await orFail(extractFormFieldInfo(pdfDoc));
|
||||
}
|
||||
} finally {
|
||||
pdfDoc.destroy();
|
||||
}
|
||||
} catch {
|
||||
// Non-fatal — return whatever was assigned before the throw.
|
||||
}
|
||||
return { formSchema, fieldInfo };
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract detailed form field info (name, type, page, bounding box, label)
|
||||
* from a PDF. Bounding boxes are converted to model coordinates (top-left origin).
|
||||
|
|
@ -1020,27 +1135,25 @@ async function extractFormFieldInfo(
|
|||
return fields;
|
||||
}
|
||||
|
||||
async function extractFormSchema(pdfDoc: PDFDocumentProxy): Promise<{
|
||||
export async function extractFormSchema(
|
||||
pdfDoc: PDFDocumentProxy,
|
||||
fieldObjects: Record<string, PdfJsFieldObject[]> | null,
|
||||
): Promise<{
|
||||
type: "object";
|
||||
properties: Record<string, PrimitiveSchemaDefinition>;
|
||||
required?: string[];
|
||||
} | null> {
|
||||
let fieldObjects: Record<string, PdfJsFieldObject[]> | null;
|
||||
try {
|
||||
fieldObjects = (await pdfDoc.getFieldObjects()) as Record<
|
||||
string,
|
||||
PdfJsFieldObject[]
|
||||
> | null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!fieldObjects || Object.keys(fieldObjects).length === 0) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const properties: Record<string, PrimitiveSchemaDefinition> = {};
|
||||
for (const [name, fields] of Object.entries(fieldObjects)) {
|
||||
const field = fields[0]; // first widget determines the type
|
||||
// pdfjs returns the full field-tree array: for separated structures
|
||||
// (pdf-lib) the typed widget is at [1+] behind a container at [0]; for
|
||||
// merged/leaf entries (W-9, most authoring tools) it's at [0]. Pick the
|
||||
// first entry that actually has a field type.
|
||||
const field = fields.find((f) => f.type) ?? fields[0];
|
||||
if (!field.editable) continue;
|
||||
|
||||
switch (field.type) {
|
||||
|
|
@ -1434,7 +1547,7 @@ Set \`elicit_form_inputs\` to true to prompt the user to fill form fields before
|
|||
const { totalBytes } = await readPdfRange(normalized, 0, 1);
|
||||
const uuid = randomUUID();
|
||||
// Start the heartbeat now so the sweep can clean up viewFieldNames/
|
||||
// viewFieldInfo/viewFileWatches even if no interact calls ever happen.
|
||||
// viewFileWatches even if no interact calls ever happen.
|
||||
if (!disableInteract) touchView(uuid);
|
||||
|
||||
// Check writability (governs save button; see isWritablePath doc).
|
||||
|
|
@ -1462,38 +1575,19 @@ Set \`elicit_form_inputs\` to true to prompt the user to fill form fields before
|
|||
}
|
||||
}
|
||||
|
||||
// Extract form field schema + detailed field info from a single
|
||||
// download/parse pass.
|
||||
let formSchema: Awaited<ReturnType<typeof extractFormSchema>> = null;
|
||||
let fieldInfo: FormFieldInfo[] = [];
|
||||
try {
|
||||
const { data } = await readPdfRange(normalized, 0, totalBytes);
|
||||
const pdfDoc = await getDocument({
|
||||
data,
|
||||
standardFontDataUrl: STANDARD_FONT_DATA_URL,
|
||||
StandardFontDataFactory: FetchStandardFontDataFactory,
|
||||
verbosity: VerbosityLevel.ERRORS,
|
||||
}).promise;
|
||||
try {
|
||||
formSchema = await extractFormSchema(pdfDoc);
|
||||
fieldInfo = await extractFormFieldInfo(pdfDoc);
|
||||
} finally {
|
||||
pdfDoc.destroy();
|
||||
}
|
||||
} catch {
|
||||
// Non-fatal — PDF may not have form fields or may fail to parse
|
||||
}
|
||||
const { formSchema, fieldInfo } = await probeFormFields(
|
||||
normalized,
|
||||
totalBytes,
|
||||
readPdfRange,
|
||||
);
|
||||
if (formSchema) {
|
||||
viewFieldNames.set(uuid, new Set(Object.keys(formSchema.properties)));
|
||||
}
|
||||
if (fieldInfo.length > 0) {
|
||||
viewFieldInfo.set(uuid, fieldInfo);
|
||||
if (!viewFieldNames.has(uuid)) {
|
||||
viewFieldNames.set(
|
||||
uuid,
|
||||
new Set(fieldInfo.map((f) => f.name).filter(Boolean)),
|
||||
);
|
||||
}
|
||||
if (fieldInfo.length > 0 && !viewFieldNames.has(uuid)) {
|
||||
viewFieldNames.set(
|
||||
uuid,
|
||||
new Set(fieldInfo.map((f) => f.name).filter(Boolean)),
|
||||
);
|
||||
}
|
||||
|
||||
// Elicit form field values if requested and client supports it
|
||||
|
|
|
|||
Loading…
Reference in a new issue