Request lifecycle¶
How one POST /v1/chat/completions request travels through the edge Worker and the
origin, in the order the code runs, and where every status code comes from.
Component overview: SYSTEM_ARCHITECTURE.md.
Defaults in this document are the values in deploy/cloudflare-worker/wrangler.toml,
deploy/cloudflare-worker/src/config.ts, src/fastapi_assistant/config.py, and
the Cloud Run overrides in deploy/cloudrun/service.yaml.
1. Sequence¶
sequenceDiagram
autonumber
participant C as Client
participant W as Edge Worker
participant R as Upstash Redis, optional
participant M as Origin middleware
participant H as Route handler
participant A as Admission
participant E as llama.cpp engine
C->>W: POST /v1/chat/completions with API key
Note over W: classifyPath. /metrics and unknown paths get 404
opt no key and REQUIRE_API_KEY is true
W-->>C: 401 authentication_error
end
Note over W: readBodyCapped, limit 64 KiB
opt body larger than MAX_REQUEST_BYTES
W-->>C: 413 request_too_large
end
alt Upstash configured and answering
W->>R: one pipeline: INCR and EXPIRE rate key, GET cache entry, GET key marker
R-->>W: count, cached body, marker
else Upstash absent or failing
W->>W: LocalRateLimiter.hit, counted in this isolate
end
opt count above the per-minute limit
W-->>C: 429 rate_limited with Retry-After
end
opt cached answer present and key marker present
W-->>C: 200 from cache, X-Cache HIT
end
W->>M: forward with X-Origin-Secret, X-Request-ID, X-Forwarded-For
Note over W,M: forwardWithFailover skips origins with an open circuit
opt every origin failed at the network level
W-->>C: 502 upstream_unavailable or 504 upstream_timeout
end
opt platform in front of the origin answered a non-JSON 429 or 5xx
W-->>C: same status as a JSON error with Retry-After
end
opt X-Origin-Secret missing or wrong
M-->>W: 403
end
opt body larger than MAX_REQUEST_BYTES while reading
M-->>W: 413
end
M->>H: JSON decode, require_principal, schema validation
opt key missing or invalid
H-->>W: 401 authentication_error
end
opt invalid schema, over 6000 characters, or last message not from the user
H-->>W: 400 invalid_request_error
end
H->>E: service.fit counts prompt tokens with the model tokenizer
opt prompt leaves fewer than 16 tokens of the 2048-token context
H-->>W: 400 context_length_exceeded
end
H->>A: service.admit
opt queue full
A-->>W: 429 overloaded with Retry-After
end
opt no slot within QUEUE_TIMEOUT_S
A-->>W: 503 overloaded with Retry-After
end
A->>E: generate in a worker thread while holding the slot
alt stream is true
E-->>C: SSE chunks through origin and Worker, then a final DONE event
else stream is false
E-->>H: text and token counts
H-->>W: 200 chat.completion JSON
W-)R: waitUntil: SET cache entry, SET key marker
W-->>C: 200 with X-Cache MISS or BYPASS
end
An origin response of 429, 502, 503 or 504 is retried on the next origin when one
exists. The last origin's response, whatever its status, is returned to the client,
rewritten into the edge's JSON error shape only if it is a non-JSON platform error.
Today ORIGINS lists only Cloud Run (audit F-11), so origin 429 and 503 responses
reach the client directly.
2. Edge steps¶
All in deploy/cloudflare-worker/src/index.ts (handle), unless noted.
| Step | Behaviour | Code |
|---|---|---|
| 1. Route | /metrics is blocked and any path outside the contract is unknown: both get 404 without touching Redis or the origin. Built UI files never reach the Worker; Cloudflare serves them from web/dist. |
src/routes.ts, wrangler.toml [assets] |
| 2. Key presence | On a protected path (/generate, /v1/*, not OPTIONS), a missing key is a 401 when REQUIRE_API_KEY is true (the default). The edge does not validate the key; the origin does. |
src/identity.ts extractApiKey |
| 3. Identity | key identity: the first 32 hex characters of SHA-256(key), limit RATE_LIMIT_PER_MINUTE (20). Otherwise ip identity from CF-Connecting-IP, limit RATE_LIMIT_PER_MINUTE_ANON (10). |
src/identity.ts |
| 4. Body cap | A declared Content-Length over 65,536 bytes is refused before reading; a chunked body is counted and abandoned once it crosses the cap. 413 request_too_large. The body is buffered so it can be hashed and replayed to a second origin. |
src/body.ts |
| 5. Redis pipeline | One Upstash /pipeline call: INCR rl:<id>:<minute>, EXPIRE ... 70, and for cacheable requests GET c:<model>:<hash> and GET kv:<id>. 2 s timeout. |
src/ratelimit.ts, src/cache.ts, src/upstash.ts |
| 6. Fallback limit | When Upstash is not configured or the pipeline throws, the same fixed window is counted in isolate memory. Rate limiting never turns off. | src/ratelimit.ts LocalRateLimiter |
| 7. 429 | Count above the limit: 429 rate_limited, Retry-After (seconds to the end of the window), X-RateLimit-Limit, -Remaining, -Reset. |
src/ratelimit.ts |
| 8. Cache HIT rule | A request is cacheable only when Redis answered, it is a POST to /generate or /v1/chat/completions, the body is a JSON object of at most 64 KiB, and stream is absent, null or false. A stored answer is served only if the key's kv: marker exists, that is, the origin accepted this key within KEY_VERIFIED_TTL_SECONDS (300 s). Unknown keys always reach the origin. |
src/cache.ts, index.ts step 5b |
| 9. Forward | Origins tried in order. Open circuits are skipped while another origin is available; if every circuit is open, all are tried. The timeout (ORIGIN_TIMEOUT_MS, 150 s as deployed; 60 s code default) covers the wait for response headers only, never a streaming body. |
src/proxy.ts forwardWithFailover |
| 10. Circuit breaker | Three consecutive network failures or timeouts open an origin's circuit for 30 s. Any HTTP response, even a 503, closes it. State is per isolate. | src/proxy.ts CircuitBreaker, src/config.ts |
| 11. Write-back | On a cache MISS: the key marker is written on any 2xx; the answer is written on a 200 JSON body of at most 64 KiB, with TTL CACHE_TTL_SECONDS (86,400). Both writes go in one pipeline inside waitUntil, so they do not delay the response. |
index.ts, src/cache.ts |
| 12. Platform errors | If the response that survives failover is a 429, 502, 503 or 504 without a JSON content type, it came from the platform in front of the origin (for example Cloud Run's front end while a new instance starts), not from the app. The Worker discards the body and answers the same status in its JSON shape: type overloaded for 429, upstream_unavailable otherwise, with the platform's Retry-After or 10 s, and logs upstream_normalized: true. The app's own 429 and 503 are JSON and pass through unchanged. |
index.ts (PLATFORM_RETRY_AFTER_SECONDS) |
| 13. Response headers | Strips x-cloud-trace-context, server, via, alt-svc; sets nosniff, X-Cache (HIT, MISS or BYPASS), X-Served-By (edge-cache or origin-<n>), X-Request-ID, rate-limit headers; rewrites an origin Location to the edge hostname. |
src/http.ts, src/proxy.ts |
Upstash is configured on the deployed Worker (since 2026-11-14), so limits are shared
and deterministic requests can be cache hits. If Upstash fails, a request is
X-Cache: BYPASS and rate limiting runs in the per-isolate fallback; see
DEPLOYMENT_ARCHITECTURE.md, section 1.
3. Origin steps¶
Middleware order was confirmed by building the app and reading
app.user_middleware: the request_context HTTP middleware is outermost, then
BodySizeLimitMiddleware, then CORSMiddleware.
| Step | Behaviour | Code |
|---|---|---|
| 1. Origin lock | If ORIGIN_SHARED_SECRET is set and the path is not /health, /ready or /metrics, the X-Origin-Secret header must match (hmac.compare_digest), or the response is 403 {"detail": "Direct origin access is not allowed"}. |
api.py request_context |
| 2. Body cap | receive is wrapped: a declared length over MAX_REQUEST_BYTES (65,536), or a streamed count over it, raises 413 on the first read, before JSON parsing. |
limits.py |
| 3. JSON decode | Malformed JSON is a 400 on /v1/* (a 422 on /generate). |
FastAPI, api.py validation_error |
| 4. Authentication | require_principal: the static API_KEY by constant-time compare, else SHA-256 lookup in Postgres through a TTL cache. 401 with WWW-Authenticate: Bearer. If neither API_KEY nor DATABASE_URL is configured, authentication is off. |
auth.py, api.py |
| 5. Schema | Pydantic validation: 1 to 100 messages, each at most 8,000 characters; max_tokens 1 to 1,024; n must be 1. 400 on /v1/*, 422 on /generate. |
schemas.py |
| 6. Character cap | The sum of message contents over MAX_INPUT_CHARS (6,000) is a 400 "Conversation is too long". |
api.py chat_completions |
| 7. Retrieval | Only when enabled (RAG_ENABLED, false on Cloud Run). Failures degrade to no context. |
service.py retrieve |
| 8. Normalise | Client system messages are dropped, the last MAX_CHAT_TURNS (6) turns kept, retrieved context attached. A conversation that does not end with a user message is a 400. |
prompts.py normalize_chat |
| 9. Token budget | service.fit runs before admission. The llama.cpp tokenizer counts content tokens plus a deliberate over-estimate of template tokens (8 per message, 8 fixed). If fewer than 16 tokens of LLAMACPP_N_CTX (2,048) remain: 400 with code: context_length_exceeded. Otherwise max_tokens is clamped to the remaining room. |
engines/base.py fit, engines/llamacpp_engine.py |
| 10. Admission | Up to MAX_CONCURRENCY (1) run; up to MAX_QUEUE (3 on Cloud Run) wait, each at most QUEUE_TIMEOUT_S (60 s on Cloud Run). Queue full: 429; wait expired: 503. Both use type overloaded and a Retry-After from a moving average of service time. |
admission.py, api.py admission_error |
| 11. Generation | Runs in a worker thread holding the slot. GENERATION_TIMEOUT_S (60 s on Cloud Run, 120 s default) is checked per token and ends the answer with finish_reason length on the OpenAI API (timeout internally). |
service.py, engines/base.py Deadline |
| 12. Response | Non-streaming: JSON chat.completion; usage recorded in a background task. Streaming: see section 5. Every response gets X-Request-ID, X-Process-Time-Ms, nosniff, X-Frame-Options: DENY, Referrer-Policy: no-referrer. |
api.py |
A request that arrives while the model is still loading waits inside step 9:
count_prompt_tokens calls load(), which blocks on the load lock until the weights
are ready. If the load fails, the request gets 503 model_not_ready with
Retry-After: 30.
4. Status codes¶
Edge errors use {"error": {"message", "type"}} (src/http.ts). Origin errors on
/v1/* use the OpenAI envelope {"error": {"message", "type", "code"}}; on other
paths {"detail": ...} (api.py _error).
| Status | Type or code | Origin of the response | Condition | Code |
|---|---|---|---|---|
| 200 | chat.completion or SSE |
Origin | Success | api.py |
| 200 | cached JSON, X-Cache: HIT |
Edge | Cache HIT rule satisfied | index.ts |
| 400 | invalid_request_error |
Origin | Malformed JSON, schema violation, over 6,000 characters, or last message not from the user | api.py, schemas.py, prompts.py |
| 400 | invalid_request_error, code context_length_exceeded |
Origin | Prompt does not fit the context window with 16 tokens to spare | engines/base.py, api.py |
| 401 | authentication_error |
Edge | Protected path, no key, REQUIRE_API_KEY true |
index.ts |
| 401 | authentication_error |
Origin | Key missing or not recognised | auth.py, api.py |
| 403 | {"detail": ...} |
Origin | X-Origin-Secret missing or wrong (direct origin access) |
api.py request_context |
| 404 | not_found |
Edge | /metrics, or a path outside the contract |
routes.ts, index.ts |
| 413 | request_too_large |
Edge | Body over 65,536 bytes | body.ts, index.ts |
| 413 | invalid_request_error, code 413 |
Origin | Body over MAX_REQUEST_BYTES; reachable directly or if the edge cap is raised |
limits.py |
| 429 | rate_limited |
Edge | Per-minute limit exceeded (Redis or per-isolate) | ratelimit.ts, index.ts |
| 429 | overloaded |
Origin | Admission queue full | admission.py, api.py |
| 429 | overloaded |
Edge, normalised | The platform in front of the origin answered a non-JSON 429 (seen during a cold scale-out) | index.ts |
| 500 | server_error |
Origin | Engine raised during a non-streaming generation ("Generation failed") | api.py |
| 500 | edge_error |
Edge | Uncaught exception in the Worker | index.ts createHandler |
| 502 | upstream_unavailable |
Edge | No origin configured, or every attempt failed at the network level without a timeout on the last one | index.ts, proxy.ts |
| 503 | overloaded |
Origin | No slot within QUEUE_TIMEOUT_S |
admission.py, api.py |
| 503 | model_not_ready |
Origin | Model failed to load; Retry-After: 30 |
api.py not_ready_error |
| 502, 503, 504 | upstream_unavailable |
Edge, normalised | The platform in front of the origin answered a non-JSON 5xx | index.ts |
| 504 | upstream_timeout |
Edge | The last attempt did not return headers within ORIGIN_TIMEOUT_MS |
index.ts, proxy.ts |
POST /generate follows the same path with these differences (api.py generate,
schemas.py GenerateRequest):
- schema and JSON errors are 422 with FastAPI's
{"detail": [...]}list; - the question is capped at 4,000 characters by the schema, and there is no separate character check or turn handling;
- retrieval is chosen per request with
use_retrieval; - streaming uses named SSE events (
token,done,error) instead of OpenAI chunks.
The Cloud Run front end can also refuse a request by itself, for example with a
plain-text 429 Rate exceeded. while a second instance starts
(docs/audit/PERFORMANCE_BASELINE.md). Through the
edge, clients receive those as the normalised edge errors in the table above.
5. Streaming¶
- The Worker never caches a streaming request (
wantsStreaminsrc/cache.ts), and passes the origin's body through unbuffered (src/proxy.ts). - The origin acquires the admission slot, then returns a
StreamingResponse. The first chunk is{"role": "assistant"}, then one chunk per token, a final chunk withfinish_reason, anddata: [DONE](api.py). - A failure after the 200 is reported in-band:
{"error": {"type": "invalid_request_error", "code": "context_length_exceeded"}}(the backstop inengines/llamacpp_engine.py_context_errors) or{"error": {"type": "server_error"}}, followed by[DONE]. - When the client's disconnect reaches the app,
service.streamsets the cancel event, the engine stops at the next token (Deadline), and the slot is released (see "Timeouts" below for what that takes on Cloud Run).
6. Timeouts and budgets¶
| Budget | Value | Covers | Code |
|---|---|---|---|
| Upstash pipeline | 2 s | One Redis round trip | src/config.ts REDIS_TIMEOUT_MS |
ORIGIN_TIMEOUT_MS |
150 s per origin (wrangler.toml; 60 s code default) |
Time until response headers | wrangler.toml, src/proxy.ts |
QUEUE_TIMEOUT_S |
60 s on Cloud Run (30 s default) | Wait for an admission slot | deploy/cloudrun/service.yaml |
GENERATION_TIMEOUT_S |
60 s on Cloud Run (120 s default) | Decode loop | deploy/cloudrun/service.yaml, config.py |
| Cloud Run request timeout | 300 s | Whole request, including a stream | deploy/cloudrun/service.yaml timeoutSeconds |
These budgets interact. For a non-streaming request the origin sends headers only
after generation finishes, so queue wait plus generation must fit inside the edge's
timeout. The deployed values keep that invariant: 60 s of queue plus 60 s of
generation is 120 s, under the edge's 150 s, so the origin always answers first. A
long answer is cut at 60 s of generation (about 500 tokens at the measured 8 to 10
tokens/s) and returned with finish_reason length, rather than lost behind a 504.
A disconnect that reaches the app stops the generation, streaming or not.
service.complete and service.stream both watch the connection
(api._client_gone) and set the engine's cancel event; the answer is recorded with
finish_reason cancelled (#26, #38). Three things had to be true for that to work
on the live path, and after #37 none was:
- The edge passes the disconnect on. The Worker aborts its origin request when
the incoming request's
signalaborts, and logsclient_gonewith status 499. - Cloud Run tells the container. Over HTTP/1.1 it does not (Google's troubleshooting guide). Over HTTP/2 it resets the request's stream, so the image serves h2c with Hypercorn (DECISIONS.md, ADR-009).
- The app acts on it. The response layer alone does not: behind the request-context middleware it either keeps writing a stream nobody reads, or stops consuming it without closing it. The watch therefore also closes the engine's generator and releases the slot if the stream has not ended one second after the disconnect.
Measured after #37, a request sent behind an abandoned one waited 59.8 s
(non-streaming) and 58.4 s (streamed); after #38 it was answered in 1.3 s and
2.2 s. Changing
any of the three values must preserve
QUEUE_TIMEOUT_S + GENERATION_TIMEOUT_S < ORIGIN_TIMEOUT_MS.