feat(preview): SSE-streamed generation, no CF 100s edge cap
All checks were successful
Deploy to Production / deploy (push) Successful in 1m27s

Architectural fix for "spec_too_large" / preview_timeout — the sync
endpoint had to fit the whole model run into Cloudflare's ~100s edge
window, which made the system fragile against any prompt that produced
a verbose spec. The new streaming path pipes Anthropic's token deltas
as Server-Sent Events; every chunk resets CF's idle timer and a 15s
keepalive comment guarantees activity even during slow first-token
windows.

@bmm/llm: new streamSpecFromAnthropic() exposes the SDK's .stream()
flow with the same typed-error contract as generateSpec — same
SpecTruncatedError / SpecValidationError / SpecTimeoutError raised from
the relevant moment.

API: POST /v1/servers/preview/stream returns text/event-stream with
events 'text' (deltas), 'spec' (final success payload, same shape as
the sync endpoint), 'error' (typed). Anthropic-only — GLM/hobby falls
back to the sync route via 409 streaming_unavailable.

Frontend: apiSseStream() handles the POST + ReadableStream + SSE
parser. The wizard's analyze() prefers the stream and only uses the
sync endpoint on the explicit 409 fallback.

nginx (api.buildmymcpserver.com): the /v1/builds/ location block (which
already had proxy_buffering off + 600s read timeout for the WS build
stream) now also matches /v1/servers/preview/stream so the SSE
response isn't buffered.
This commit is contained in:
Marco Sadjadi
2026-05-28 21:11:05 +02:00
parent b930a454e8
commit 0c6d738a6b
4 changed files with 385 additions and 12 deletions

View File

@@ -295,6 +295,79 @@ async function generateWithAnthropic(
return { spec: parsed.data, source: 'claude' };
}
// ──────────────────────────────────────────────────────────────────────────
// Streaming generation (Anthropic only)
// ──────────────────────────────────────────────────────────────────────────
export interface StreamHandlers {
/** Called for each text delta emitted by the model. */
onText: (text: string) => void;
/** Called once when the stream completes successfully with the final spec. */
onSpec: (result: GenerationResult) => void;
/** Called once on any terminal error (timeout, truncation, validation). */
onError: (err: Error) => void;
}
/**
* Stream a spec from Anthropic, piping text deltas to a handler and finally
* surfacing the parsed/validated spec or the relevant typed error.
*
* Why streaming for /preview: Cloudflare's edge timeout is ~100s on the Free
* tier and our previous sync call could approach that for ambitious prompts.
* With streaming the TCP connection writes bytes from the first model token,
* which keeps CF (and nginx) from cutting us off — runtime is bounded only by
* the model itself and our own AbortController, not by CF.
*/
export async function streamSpecFromAnthropic(
prompt: string,
opts: { apiKey: string; model: string; maxTokens: number; signal?: AbortSignal },
handlers: StreamHandlers,
): Promise<void> {
const client = new Anthropic({ apiKey: opts.apiKey });
let accumulated = '';
try {
const stream = client.messages.stream({
model: opts.model,
max_tokens: opts.maxTokens,
system: SYSTEM_PROMPT,
messages: [{ role: 'user', content: prompt }],
});
if (opts.signal) {
opts.signal.addEventListener('abort', () => stream.abort(), { once: true });
}
stream.on('text', (delta) => {
accumulated += delta;
handlers.onText(delta);
});
const final = await stream.finalMessage();
if (final.stop_reason === 'max_tokens') {
throw new SpecTruncatedError(
`model hit max_tokens (${opts.maxTokens}) before finishing the spec`,
);
}
const json = extractJson(accumulated);
const parsed = GeneratorSpec.safeParse(json);
if (!parsed.success) {
const preview = accumulated.slice(0, 400).replace(/\s+/g, ' ');
throw new SpecValidationError(`${parsed.error.message} :: raw="${preview}"`);
}
scanForInjection(parsed.data);
handlers.onSpec({ spec: parsed.data, source: 'claude' });
} catch (err) {
if (err instanceof Anthropic.APIConnectionTimeoutError) {
handlers.onError(new SpecTimeoutError('spec generation exceeded the time budget'));
return;
}
handlers.onError(err instanceof Error ? err : new Error(String(err)));
}
}
const GLM_ENDPOINT = 'https://open.bigmodel.cn/api/paas/v4/chat/completions';
async function generateWithGlm(