feat(preview): SSE-streamed generation, no CF 100s edge cap
All checks were successful
Deploy to Production / deploy (push) Successful in 1m27s
All checks were successful
Deploy to Production / deploy (push) Successful in 1m27s
Architectural fix for "spec_too_large" / preview_timeout — the sync endpoint had to fit the whole model run into Cloudflare's ~100s edge window, which made the system fragile against any prompt that produced a verbose spec. The new streaming path pipes Anthropic's token deltas as Server-Sent Events; every chunk resets CF's idle timer and a 15s keepalive comment guarantees activity even during slow first-token windows. @bmm/llm: new streamSpecFromAnthropic() exposes the SDK's .stream() flow with the same typed-error contract as generateSpec — same SpecTruncatedError / SpecValidationError / SpecTimeoutError raised from the relevant moment. API: POST /v1/servers/preview/stream returns text/event-stream with events 'text' (deltas), 'spec' (final success payload, same shape as the sync endpoint), 'error' (typed). Anthropic-only — GLM/hobby falls back to the sync route via 409 streaming_unavailable. Frontend: apiSseStream() handles the POST + ReadableStream + SSE parser. The wizard's analyze() prefers the stream and only uses the sync endpoint on the explicit 409 fallback. nginx (api.buildmymcpserver.com): the /v1/builds/ location block (which already had proxy_buffering off + 600s read timeout for the WS build stream) now also matches /v1/servers/preview/stream so the SSE response isn't buffered.
This commit is contained in:
@@ -295,6 +295,79 @@ async function generateWithAnthropic(
|
||||
return { spec: parsed.data, source: 'claude' };
|
||||
}
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────────
|
||||
// Streaming generation (Anthropic only)
|
||||
// ──────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface StreamHandlers {
|
||||
/** Called for each text delta emitted by the model. */
|
||||
onText: (text: string) => void;
|
||||
/** Called once when the stream completes successfully with the final spec. */
|
||||
onSpec: (result: GenerationResult) => void;
|
||||
/** Called once on any terminal error (timeout, truncation, validation). */
|
||||
onError: (err: Error) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream a spec from Anthropic, piping text deltas to a handler and finally
|
||||
* surfacing the parsed/validated spec or the relevant typed error.
|
||||
*
|
||||
* Why streaming for /preview: Cloudflare's edge timeout is ~100s on the Free
|
||||
* tier and our previous sync call could approach that for ambitious prompts.
|
||||
* With streaming the TCP connection writes bytes from the first model token,
|
||||
* which keeps CF (and nginx) from cutting us off — runtime is bounded only by
|
||||
* the model itself and our own AbortController, not by CF.
|
||||
*/
|
||||
export async function streamSpecFromAnthropic(
|
||||
prompt: string,
|
||||
opts: { apiKey: string; model: string; maxTokens: number; signal?: AbortSignal },
|
||||
handlers: StreamHandlers,
|
||||
): Promise<void> {
|
||||
const client = new Anthropic({ apiKey: opts.apiKey });
|
||||
let accumulated = '';
|
||||
|
||||
try {
|
||||
const stream = client.messages.stream({
|
||||
model: opts.model,
|
||||
max_tokens: opts.maxTokens,
|
||||
system: SYSTEM_PROMPT,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
});
|
||||
|
||||
if (opts.signal) {
|
||||
opts.signal.addEventListener('abort', () => stream.abort(), { once: true });
|
||||
}
|
||||
|
||||
stream.on('text', (delta) => {
|
||||
accumulated += delta;
|
||||
handlers.onText(delta);
|
||||
});
|
||||
|
||||
const final = await stream.finalMessage();
|
||||
|
||||
if (final.stop_reason === 'max_tokens') {
|
||||
throw new SpecTruncatedError(
|
||||
`model hit max_tokens (${opts.maxTokens}) before finishing the spec`,
|
||||
);
|
||||
}
|
||||
|
||||
const json = extractJson(accumulated);
|
||||
const parsed = GeneratorSpec.safeParse(json);
|
||||
if (!parsed.success) {
|
||||
const preview = accumulated.slice(0, 400).replace(/\s+/g, ' ');
|
||||
throw new SpecValidationError(`${parsed.error.message} :: raw="${preview}"`);
|
||||
}
|
||||
scanForInjection(parsed.data);
|
||||
handlers.onSpec({ spec: parsed.data, source: 'claude' });
|
||||
} catch (err) {
|
||||
if (err instanceof Anthropic.APIConnectionTimeoutError) {
|
||||
handlers.onError(new SpecTimeoutError('spec generation exceeded the time budget'));
|
||||
return;
|
||||
}
|
||||
handlers.onError(err instanceof Error ? err : new Error(String(err)));
|
||||
}
|
||||
}
|
||||
|
||||
const GLM_ENDPOINT = 'https://open.bigmodel.cn/api/paas/v4/chat/completions';
|
||||
|
||||
async function generateWithGlm(
|
||||
|
||||
Reference in New Issue
Block a user