Performance baseline¶
Measured on 2026-11-13 against the live deployment:
- Edge: Cloudflare Worker.
- Origin: Cloud Run, 2 vCPU and 2 GiB per instance,
maxScale: 2,containerConcurrency: 4,MAX_CONCURRENCY=1,MAX_QUEUE=3. - Model: TinyLlama-1.1B-Chat Q4_K_M with llama.cpp,
n_ctx2048.
These figures were measured on the TinyLlama-1.1B model served at the time. Since
2026-11-15 the deployed model is v2026.11.14-72d6b37, a fine-tune of
Qwen2.5-Coder-1.5B-Instruct, also Q4_K_M on llama.cpp. The measurements below have not
been repeated for it; on Cloud Run (2 vCPU) it generates about 7.6 tokens/s (Grafana,
mean over an hour), and two live answers took 14.2 s (104 tokens) and 26.3 s
(192 tokens).
Every figure is one observed run from a single client, not a statistical benchmark. No sustained load test has been run, and none should be run against the public deployment without a plan, a time window and a cost check.
Single requests (warm instance)¶
| Request | Result |
|---|---|
GET /health through the edge |
683 ms (includes edge and TLS setup from the client) |
POST /generate, 64 new tokens, blocking |
8.6 s first call, 6.3 s repeat |
| OpenAI SDK chat completion, 64 tokens | 7.3 s, finish_reason: length |
| Native SSE stream, 64-token budget | time to first token 298 ms, 6.8 s total, 65 events |
| OpenAI SDK stream, 48 tokens | time to first token 1.06 s, 5.5 s total |
POST /generate with retrieval, 128 tokens |
521 prompt tokens (retrieved context included) |
Implied decode rate for one request: roughly 8 to 10 tokens per second.
Concurrency probes¶
Two controlled, one-off bursts of short requests (32 new tokens each), one minute apart:
| Burst | Outcome | Latencies |
|---|---|---|
| 6 simultaneous, one warm instance | 2 x 200, 4 x 429 | 200s at 3.8 s and 7.2 s; the 429s all at about 10.5 s |
| 4 simultaneous, a minute later | 4 x 200 | 3.5, 5.8, 8.8, 11.9 s (served one after another) |
What the probes show¶
- One generation at a time per instance. The 4-request burst finished in a staircase,
about 3 s per 32-token answer. That is
MAX_CONCURRENCY=1: the queue holds the others, so latency grows linearly with queue position. - Scale-out is not instant. In the 6-request burst, Cloud Run routed part of the burst towards a second instance that was still starting.
- What the client saw: after about 10 seconds, Cloud Run's front end answered
those requests itself with
429 Rate exceeded.(plain text, noRetry-After). - Why: a cold instance has to download and load the model before it is ready. The application's own queue never saw those requests.
- The app's admission control behaves as designed once traffic reaches it. It returned no 429 or 503 in either burst.
Changes made because of this¶
- Edge: the Worker now rewrites a non-JSON 429, 502, 503 or 504 from the origin's
platform into the API's JSON error shape (
overloadedorupstream_unavailable). It addsRetry-After: 10when the platform sent none and logsupstream_normalized: true. Clients see one error contract, whichever layer refused the request. - New metrics:
assistant_queue_wait_secondsandassistant_time_to_first_token_secondsmake the staircase visible without probing.
Options not taken (they cost money or need owner approval)¶
| Option | Effect | Cost |
|---|---|---|
minScale: 1 |
No cold start for the first instance | An always-allocated instance, billed while idle |
Higher maxScale |
More parallel generations during bursts | Raises the worst-case bill |
| Bake weights into the image | Shorter cold start (no 0.7 GB download; about 1 GB for the Qwen release) | Image exceeds Artifact Registry's 0.5 GB free storage |
After the hardening release (same day)¶
Measured after PR #13 was deployed (Cloud Run revision built from b146406, Worker
from main), with the same 64-token requests as above:
| When | Per-request speed at the origin (x-process-time-ms) |
|---|---|
| First samples after the deploy | 12.3 to 13.6 s per 64-token answer, about 5 tokens/s |
| Samples a few minutes later | 2.6 to 3.2 s per answer of up to 48 tokens, about 15 to 18 tokens/s |
The code path did not change speed. The CI eval, which runs the same llama.cpp workload on GitHub runners, has a p50 of 5.8 to 7.0 s per 256-token answer both before and after the hardening commits. The image's llama.cpp build was a cache hit in both releases (same base image digest, identical wheel).
The spread therefore comes from the platform: which host an instance lands on, and how long it has been warm. Plan capacity against the low end, about 5 tokens/s per request.
Capacity statement¶
From this evidence:
- Concurrency: the deployment serves about one generation at a time per instance, with at most 2 instances.
- Short bursts: a burst of a few short requests completes, serially.
- Larger bursts: a burst that forces a cold scale-out can see some requests refused with 429 and a retry hint while the second instance starts.
No number of supported users is claimed. Establishing one needs a controlled load test
(see tests/load/k6.js), with an explicit request budget, against a non-production
copy or an agreed window.