System architecture¶
This document set describes the system as the code implements it, for engineers who evaluate or contribute to the project. Every behaviour below is traced to a file in the repository. Measured figures come only from docs/audit/INITIAL_AUDIT.md.
| Document | Covers |
|---|---|
| SYSTEM_ARCHITECTURE.md (this file) | Components, responsibilities, runtime topology |
| REQUEST_LIFECYCLE.md | One /v1/chat/completions request through the edge and the origin, every status code |
| DEPLOYMENT_ARCHITECTURE.md | What is live, the Release pipeline, Cloud Run settings |
| SECURITY_ARCHITECTURE.md | Trust boundaries and the controls at each one |
| MODEL_PIPELINE.md | Data, training, export, eval gates, release and rollback of weights |
| DECISIONS.md | Architecture decision records |
Related: docs/ARCHITECTURE.md (the original high-level design and its scaling path), docs/DEPLOYMENT.md (the setup runbook), docs/IMPLEMENTATION_STATUS.md (what is live today), README.md (quickstart and API).
These documents describe the code on main. The hardening changes described here
were merged in PR #13 and are live, together with the follow-ups #14 to #16: the
Release workflow deploys both Cloud Run and the edge Worker from main.
1. What the system is¶
A FastAPI-specialised small language model served as an HTTP service with a native
API (POST /generate) and an OpenAI-compatible API (/v1/chat/completions,
/v1/models, /v1/feedback). The model is Qwen2.5-Coder-1.5B-Instruct with a
LoRA fine-tune merged in, served as a quantized GGUF by llama.cpp on CPU.
The deployed model is v2026.11.14-72d6b37 ("finetuned": true in
model-release.json), live since 2026-11-15 (#53). It replaced the first fine-tuned
release, v2026.11.13-2b04940, a TinyLlama-1.1B-Chat fine-tune that resolved audit
finding F-04 in #33. See MODEL_PIPELINE.md.
2. Context diagram¶
Dashed nodes and links are optional and not deployed. Solid nodes and links are live or in use, according to the implementation status (Upstash and Neon since 2026-11-14; fine-tuned releases published to the Hub since #33).
flowchart LR
Client["API clients<br/>curl, OpenAI SDKs"]
Browser["Browser"]
subgraph CF["Cloudflare"]
Assets["Static assets<br/>web UI build, web/dist"]
Worker["Edge Worker<br/>deploy/cloudflare-worker"]
end
Redis[("Upstash Redis<br/>optional, live")]
Backup["Backup origins<br/>HF Space, Oracle VM<br/>config only, not deployed"]
subgraph GCP["Google Cloud"]
Run["Cloud Run origin<br/>FastAPI service, src/fastapi_assistant"]
Engine["llama.cpp engine<br/>in-process, GGUF Q4_K_M"]
SM[("Secret Manager")]
AR[("Artifact Registry")]
end
PG[("Neon Postgres + pgvector<br/>optional, live")]
Hub[("Hugging Face Hub<br/>GGUF at a pinned revision")]
GHA["GitHub Actions<br/>CI, Release, Security, Eval, Train"]
GHCR[("GHCR<br/>multi-arch image")]
Kaggle["Kaggle GPU kernel<br/>trains each model release"]
Browser -->|"GET /, /assets"| Assets
Browser -->|"same-origin API calls"| Worker
Client -->|HTTPS| Worker
Worker -->|"REST pipeline"| Redis
Worker -->|"X-Origin-Secret"| Run
Worker -.->|"failover"| Backup
Run --> Engine
Run -->|"keys, usage, feedback"| PG
Run -->|"download GGUF, verify SHA-256"| Hub
SM -->|"secrets as env vars"| Run
AR -->|"amd64 image"| Run
GHA -->|"build and push"| GHCR
GHA -->|"copy image, sync secrets, deploy"| GCP
GHA -->|"wrangler deploy, k6 smoke"| Worker
GHA -->|"train.yml"| Kaggle
Kaggle -->|"publish tagged GGUF"| Hub
classDef optional stroke-dasharray: 5 5
class Backup optional
3. Components and responsibilities¶
Edge (Cloudflare Worker)¶
| Module | Responsibility |
|---|---|
deploy/cloudflare-worker/src/index.ts |
The request pipeline: route check, key presence, body cap, rate limit, cache lookup, forward, normalisation of non-JSON platform errors, cache write-back, response header hygiene. One JSON log line per request. |
src/routes.ts |
Classifies paths as protected (/generate, /v1/*), public (/health, /ready, /docs, /redoc, /openapi.json), blocked (/metrics) or unknown (404 at the edge, so scanners never wake a scaled-to-zero origin). |
src/identity.ts |
Reads the key from X-API-Key or Authorization: Bearer. The rate-limit identity is the first 32 hex characters of the key's SHA-256, or ip:<CF-Connecting-IP> for anonymous calls. The raw key never leaves this module. |
src/body.ts |
Buffers the request body once, capped at MAX_REQUEST_BYTES (64 KiB). Checks the declared length first, then counts streamed bytes. |
src/ratelimit.ts |
Fixed one-minute window: INCR plus EXPIRE 70 on Redis, or LocalRateLimiter (per isolate, at most 10,000 identities) when Upstash is absent or fails. |
src/upstash.ts |
Minimal Upstash REST client: one /pipeline HTTP call per use, 2 s timeout (REDIS_TIMEOUT_MS). |
src/cache.ts |
Shared answer cache keyed by MODEL_VERSION and the SHA-256 of path plus canonical JSON body. Verified-key marker kv:<identity>. |
src/proxy.ts |
Builds forwarded headers (strips hop-by-hop, cf-* and client-supplied x-origin-secret; sets the real secret, x-request-id, x-forwarded-for). Ordered failover across ORIGINS, per-isolate CircuitBreaker, Location rewrite. |
src/http.ts |
Edge error envelope {"error": {"message", "type"}}, X-Content-Type-Options: nosniff, strips x-cloud-trace-context, server, via, alt-svc from origin responses. |
src/config.ts |
Parses [vars] and secrets with defaults (DEFAULTS). |
wrangler.toml |
Worker name fastapi-assistant-edge; [assets] directory = "../../web/dist", so built UI files are answered by Cloudflare without invoking the Worker. |
Web UI¶
| Path | Responsibility |
|---|---|
web/src/ |
React + Vite playground: streaming answers, sources, usage, Retry-After handling, feedback, client snippets. The API key is kept in browser localStorage (web/src/state/settings.ts). |
web/public/_headers |
CSP and other browser security headers for the static files. |
Origin (Python package src/fastapi_assistant/)¶
| Module | Responsibility |
|---|---|
api.py |
App factory create_app, routes, middleware (origin lock, request id, access log, metrics, security headers), error mapping to the native or OpenAI envelope. |
limits.py |
BodySizeLimitMiddleware: 413 before the body is parsed. |
auth.py, storage/ |
Static API_KEY (constant-time compare) and per-user keys stored as SHA-256 hashes in Postgres, with an in-process TTL cache. Usage events and feedback. SQL migrations in storage/migrations/. |
service.py |
One code path for both APIs: retrieval, token budget (fit), admission, generation, metrics, usage accounting. |
admission.py |
Bounded concurrency plus a bounded wait queue. 429 or 503 with Retry-After. |
engines/base.py |
Engine contract, load-once lifecycle, Deadline (timeout and cancellation), fit (prompt token budget). |
engines/llamacpp_engine.py |
Production backend: GGUF on CPU, real tokenizer for fit, one generation at a time under a lock. |
engines/transformers_engine.py |
torch + LoRA adapter; used for training-time evaluation and development. |
engines/remote_engine.py |
Forwards to an OpenAI-compatible server (vLLM, TGI). Not used in production. |
hub.py |
Resolves the GGUF or adapter from local disk or the Hugging Face Hub at a pinned revision; verify_sha256 fails the load on a mismatch. |
retrieval.py, docs_corpus.py |
Documentation index (local NumPy index baked into the image, or pgvector). Retrieval is opt-in per request. |
prompts.py |
The only definition of the prompt format, shared by training, evaluation and serving. Client system messages are dropped. |
metrics.py |
Prometheus registry per app instance: request, generation, tokens/s, queue depth and wait, time to first token, admission and prompt rejections. |
config.py |
Pydantic settings from the environment, validated at startup. |
evaluation.py, datasets_tools.py, datagen.py, training.py, export.py |
Offline model pipeline: eval harness and gate, dataset validation, Claude-assisted drafting, LoRA training, merge and Hub publishing. |
__main__.py |
CLI: serve, eval, train, data, export, rag, db, keys. |
Deployment and pipeline¶
| Path | Responsibility |
|---|---|
Dockerfile |
Two-stage image: wheels built from the hash-pinned requirements/serve.txt, runtime as uid 10001 with no pip, setuptools or wheel. Weights are not baked in (BAKE_MODEL=false); the RAG index is (BAKE_RAG_INDEX=true). |
deploy/cloudrun/service.yaml |
Cloud Run service template, rendered by Release. |
deploy/gcp/bootstrap.sh |
One-time GCP setup: Artifact Registry, service accounts, Workload Identity Federation, budget alert. |
deploy/hf-space/, deploy/vm/ |
Backup origin configurations. Not deployed (F-11). |
deploy/grafana/ |
Dashboard and alert rules. |
.github/workflows/ |
ci, release, security, codeql, eval, train, uptime. |
kaggle/pipeline.py, scripts/export_gguf.sh |
GPU training, merge, GGUF conversion and quantisation. |
model-release.json |
The deployed weights, pinned to a Hub revision and SHA-256. |
4. Runtime topology and state¶
| Tier | Where it runs | State it holds | Scaling unit |
|---|---|---|---|
| Static UI | Cloudflare Workers Static Assets | None | Cloudflare |
| Edge Worker | Cloudflare isolates | Per isolate: LocalRateLimiter counters and CircuitBreaker state. Lost when the isolate is recycled. |
Cloudflare |
| Shared edge state | Upstash Redis (optional) | Rate counters rl:*, cached answers c:*, verified-key markers kv:* |
Not configured today |
| Origin | Cloud Run instance, 2 vCPU, 4 GiB | Read-only model weights, auth TTL cache, admission counters, metrics registry | Instances: minScale 0, maxScale from CLOUD_RUN_MAX_INSTANCES (default 2) |
| Persistence | Neon Postgres (optional) | api_keys, usage_events, feedback, doc_chunks |
Not required |
| Weights | Hugging Face Hub | GGUF at a pinned revision | Downloaded on cold start |
The origin is stateless apart from caches, so capacity grows by adding instances or
origins. Inside one instance, one generation runs at a time: MAX_CONCURRENCY=1 in
deploy/cloudrun/service.yaml, and LlamaCppEngine holds a lock around each
generation.
5. Invariants¶
- The prompt format lives only in
src/fastapi_assistant/prompts.py. Training, evaluation and serving all call it; the GGUF converter embeds the same tokenizer chat template (scripts/export_gguf.sh). - Every request to an origin, except
/health,/readyand/metrics, must carry the sharedX-Origin-SecretwhenORIGIN_SHARED_SECRETis set (api.py). - A prompt is measured in tokens and rejected or clamped before it takes an
admission slot (
service.fit,engines/base.py). - Deployed weights are named only by
model-release.json(repository, file, revision, SHA-256). Weights are never committed to git. - Overload is answered quickly with 429 or 503 and
Retry-After, never with an unbounded queue (admission.py).
6. Measured performance and capacity¶
From the audit's live baseline (single requests, Cloud Run, 2 vCPU, Q4_K_M):
| Measurement | Value |
|---|---|
| Decode throughput, one request | about 8 to 10 tokens/s |
| Streaming time to first token | 0.3 s (native SSE) to 1.1 s (OpenAI SDK) |
POST /generate, 64 tokens |
8.6 s first call, 6.3 s repeat |
No load test has been run. The audit's capacity statement applies: one generation
per instance, at most CLOUD_RUN_MAX_INSTANCES instances (two when the audit was
written, one on the live public demo), further requests queued up to MAX_QUEUE and
then rejected with 429. No figure for supported users is claimed. Two one-off burst
probes and what they showed about scale-out are recorded in
docs/audit/PERFORMANCE_BASELINE.md.