Architecture (High-Level Design)¶
This document describes the FastAPI Assistant as a production system: a small fine-tuned language model served behind an edge layer, across several free-tier origins, with a gated model-release pipeline. It is written for an HLD review: it states responsibilities, data flow, the scaling and failure model, and the trade-offs behind each decision.
Detailed companion documents live in architecture/: system, request lifecycle, deployment, security, model pipeline and decision records. What is live today, as opposed to designed, is in IMPLEMENTATION_STATUS.md.
1. Goals and non-goals¶
Goals
- Answer FastAPI questions with a fine-tuned small model behind a stable HTTP API (native and OpenAI-compatible).
- Run at zero cost on free tiers, while keeping the architecture that scales by adding capacity rather than by redesign.
- Make model quality a release gate: no model ships unless it beats the one it replaces on a held-out evaluation.
- Degrade gracefully: overload produces fast, explicit back-pressure (429 +
Retry-After), not timeouts; a failed origin is routed around.
Non-goals
- High-throughput multi-tenant inference on free hardware. One CPU replica produces roughly 8-15 tokens/s (8-10 measured on a 2-vCPU Cloud Run instance; see audit/PERFORMANCE_BASELINE.md); capacity grows with replicas or a GPU server (section 8), not inside this process.
- General-purpose chat. The service answers FastAPI questions with one system prompt.
2. System overview¶
flowchart LR
Client["Clients<br/>curl, OpenAI SDKs, apps"]
Browser["Browser<br/>web UI (web/)"]
subgraph Edge["Cloudflare Worker (edge)"]
Assets["static assets<br/>(built web UI)"]
W["key check, rate limit,<br/>shared answer cache,<br/>origin failover, origin lock"]
end
Redis[("Upstash Redis<br/>rate counters,<br/>cached answers")]
subgraph Origins["Origins (same image, stateless)"]
CR["Cloud Run<br/>primary, scale-to-zero"]
VM["Oracle Ampere VM<br/>backup, via Cloudflare Tunnel"]
HF["HF Space<br/>optional backup"]
end
PG[("Neon Postgres<br/>API keys, usage,<br/>feedback, pgvector")]
Hub[("Hugging Face Hub<br/>versioned adapter + GGUF")]
Obs["Grafana Cloud + Sentry"]
Client -->|HTTPS| W
Browser -->|GET /| Assets
Browser -->|same-origin API calls| W
W <--> Redis
W -->|X-Origin-Secret| CR
W -.->|on 429/5xx/timeout| VM
W -.-> HF
CR & VM & HF --> PG
CR & VM & HF -->|pinned revision| Hub
CR & VM -.-> Obs
Model build pipeline (offline):
flowchart LR
Docs["FastAPI docs<br/>(pinned release)"] --> Gen["data generate<br/>(Claude, drafts)"]
Gen --> Review["human review<br/>+ data validate"]
Review --> Data[("data/*.jsonl")]
Data --> Train["Kaggle GPU<br/>LoRA fine-tune"]
Train --> G1{"beats base<br/>model?"}
G1 -->|yes| Export["merge + GGUF Q4_K_M"]
Export --> Hub[("HF Hub<br/>tagged release")]
Hub --> G2{"CI eval gate<br/>(production backend)"}
G2 -->|yes| PR["PR: model-release.json<br/>+ eval/baseline.json"]
PR --> Deploy["merge = deploy"]
3. Components¶
| Component | Responsibility |
|---|---|
deploy/cloudflare-worker |
Edge: API-key presence check, per-key and per-IP rate limits, shared answer cache, ordered origin failover, injects X-Origin-Secret. Also serves the built web UI as static assets. |
web/ |
Browser playground (React + TypeScript, no server): streaming answers with sources, usage and edge headers, Retry-After handling, feedback, both APIs, copyable client snippets. Same hostname as the API, so there is no CORS surface and no extra host. |
api.py |
App factory, routes (/generate, /v1/chat/completions, /v1/models, /v1/feedback, probes, /metrics), middleware (request id, origin lock, access log, metrics), error mapping. |
service.py |
One code path for both APIs: retrieval, admission, generation, metrics, usage accounting. |
admission.py |
Bounded concurrency plus bounded wait queue; rejects with Retry-After. |
engines/ |
Engine contract with three backends: transformers (torch + LoRA), llamacpp (GGUF on CPU, production), remote (OpenAI-compatible server such as vLLM). |
prompts.py |
The single definition of the prompt format, shared by training, evaluation, and serving. |
retrieval.py, docs_corpus.py |
Documentation index (local NumPy or pgvector) and the docs loader/chunker. |
auth.py, storage/ |
Static key or per-user hashed keys in Postgres (TTL-cached), usage events, feedback, SQL migrations. |
evaluation.py |
Eval harness and release gate. |
datasets_tools.py, datagen.py |
Dataset validation (including train/eval leakage) and Claude-assisted drafting from the docs. |
training.py, export.py, kaggle/, scripts/export_gguf.sh |
Fine-tuning, merging, GGUF conversion, Hub publishing, release manifest. |
The most important invariant: the prompt format lives in prompts.py only.
Training, evaluation, and serving all call build_messages / normalize_chat, and
the GGUF converter embeds the same tokenizer chat template. If any of them diverged,
quality would silently collapse.
4. Request lifecycle¶
sequenceDiagram
participant C as Client
participant W as Worker
participant R as Upstash
participant O as Origin
participant A as Admission
participant E as Engine
C->>W: POST /generate (X-API-Key)
W->>W: no key? 401 (no Redis, no origin)
W->>R: pipeline: INCR rate counter, GET cached answer, GET key marker
alt over limit
W-->>C: 429 + Retry-After
else cache hit and key verified recently
W-->>C: 200 cached (X-Cache: HIT)
else
W->>O: forward + X-Origin-Secret
O->>O: origin secret, key auth (Postgres, TTL cache)
O->>O: retrieve docs (optional)
O->>A: acquire slot
alt queue full
A-->>W: 429 + Retry-After
W->>O: try next origin
else slot granted
A->>E: generate (worker thread, deadline)
E-->>O: answer + token counts
O-->>W: 200 JSON (or SSE stream)
W->>R: SET cache, SET key marker (waitUntil)
W-->>C: 200 (X-Cache: MISS)
end
end
Streaming responses (stream: true) bypass the cache and are passed through
unbuffered. If the client disconnects, a cancellation event stops the engine at the
next token and the slot is released, for streamed and non-streaming requests alike.
On Cloud Run that needs HTTP/2 to the container
(architecture/DECISIONS.md, ADR-009).
5. Back-pressure and concurrency¶
- An origin runs at most
MAX_CONCURRENCYgenerations (1 on CPU) and lets at mostMAX_QUEUEmore wait, each for at mostQUEUE_TIMEOUT_S. Anything beyond is rejected immediately with 429 and aRetry-Afterestimated from the moving average service time. - Overload is therefore visible (metric
assistant_admission_rejections_total), cheap (no work done), and actionable: the edge retries the next origin, and clients back off. - Each generation has a wall-clock budget (
GENERATION_TIMEOUT_S) enforced inside the decode loop (finish_reason: "timeout"), so one long request cannot hold a slot indefinitely. - Model loading is load-once under a lock and runs in the background at startup.
/readyis non-blocking: 503 until loaded, so platform probes never time out while weights download.
6. Caching and its safety rule¶
The edge caches non-streaming answers in Redis for CACHE_TTL_SECONDS, keyed by
MODEL_VERSION and the canonical request body, and shared across users (answers do
not depend on the caller). A shared cache must not become an authentication bypass,
so a hit is served only to a key the origin accepted within the last
KEY_VERIFIED_TTL_SECONDS. Unknown keys always reach the origin. Releasing a new
model changes MODEL_VERSION, which moves the cache namespace.
7. Data model (Postgres)¶
| Table | Purpose |
|---|---|
api_keys |
key_hash (SHA-256, unique), display prefix, active flag. Plaintext keys are shown once and never stored. |
usage_events |
Per request: key, endpoint, status, prompt/completion tokens, latency, finish reason, model version. Question/answer text only if STORE_TRANSCRIPTS=true. |
feedback |
Rating (+1/-1) and comment, joined to usage by request id: the raw material for the next dataset iteration. |
doc_chunks |
Documentation chunks with vector(384) embeddings and an HNSW cosine index (pgvector backend). |
Migrations are ordered SQL files applied by fastapi-assistant db migrate.
8. Scaling path¶
The origins are stateless (the only in-memory state is the read-only model and short-lived auth cache), so capacity scales by replication. In order of cost:
- Cache (free): repeated questions never reach a model.
- More free origins: Cloud Run instances (
maxScale), the Oracle VM, a Space. The edge fails over on 429/5xx/timeouts, so replicas absorb each other's load. - Paid CPU capacity: raise
maxScale; no code change. - GPU inference server: run vLLM or TGI with the merged model and set
ENGINE_BACKEND=remote. Continuous batching raises throughput by orders of magnitude; the service keeps auth, admission, metrics, and the API contract. - Bigger base model: the training pipeline is base-model agnostic
(
BASE_MODEL_NAME); the eval gate decides whether it ships.
9. Model lifecycle and release gates¶
- Data: hand-written and Claude-drafted pairs from the pinned docs, reviewed, then
data validate(schema, parseable code, duplicates, eval leakage) in CI. - Train on a free Kaggle GPU (
train.yml->kaggle/pipeline.py). - Gate 1 (Kaggle): the fine-tune must beat the base model on the eval set.
- Export: merge, convert, quantize to Q4_K_M, push adapter and GGUF to the Hub under an immutable tag.
- Gate 2 (CI,
eval.yml): score the GGUF with the production backend; it must not regress more than 0.02 against the model onmain. - Release: a PR updates
model-release.json(pinned revisions) andeval/baseline.json. Merging deploys; reverting rolls back.
10. Observability¶
- Metrics (
/metrics, Prometheus): request rate/latency by route template, generation latency, tokens/s, prompt/completion tokens, finish reasons, queue depth, in-flight, admission rejections, retrieval latency, model loaded, build info. Scraped by Alloy on the VM, and by Grafana Cloud's metrics endpoint job for Cloud Run. Dashboard and alert rules live indeploy/grafana/. - Logs: one JSON line per request with request id, status, and duration; the Worker logs cache status, origin, attempts, and rate-limit identity type (never keys).
- Errors: Sentry (optional), with release
fastapi-assistant@<version>+<model>. - Uptime: a 10-minute synthetic probe through the edge (
uptime.yml).
11. Security¶
- Edge rejects missing keys; origins validate keys (hashed, constant-time static comparison) and reject requests without the shared origin secret.
- CORS never allows credentials; auth is header-based.
- Secrets live in Secret Manager / Worker secrets / GitHub secrets; CI deploys through Workload Identity Federation (no long-lived cloud keys). Third-party Actions are pinned to commit SHAs.
- Images: non-root user, no compiler or packaging tools, Debian security updates applied, hash-pinned dependencies, Trivy scan (fails on fixable high and critical CVEs), CodeQL, gitleaks, pip-audit, npm audit, dependency review, Dependabot.
- Generated code is only executed by the evaluator when explicitly enabled
(
--execute-code), and only in disposable environments.
12. Failure modes¶
| Failure | Behavior |
|---|---|
| Origin overloaded | 429 + Retry-After; edge tries the next origin. |
| Origin down or slow | Edge times out (ORIGIN_TIMEOUT_MS) and fails over; 502 if every origin fails. |
| Model not loaded yet | /ready 503 (platform holds traffic); requests get 503 + Retry-After. |
| Weights download fails | Load state failed; retried on the next /ready or request. |
| Postgres unavailable | Static-key requests keep working; per-user keys fail closed (401/500); usage writes are logged and dropped. |
| Retrieval fails | Answer without context; logged and counted. |
| Redis unavailable or not configured | No shared cache; rate limiting falls back to per-isolate in-memory counters (weaker, never off); traffic still flows. |
| Generation exceeds budget | Truncated with finish_reason: "timeout". |
| Bad model release | Blocked by the eval gates; otherwise revert model-release.json. |
13. Decision log¶
- llama.cpp GGUF for production, torch for training. Q4_K_M is about 1 GB for the deployed 1.5B model and several times faster than fp32 torch on CPU, which is what makes free CPU instances usable. torch remains the reference backend for training-time eval.
- Admission control over unbounded queuing. Fast 429s keep latency honest and let the edge spread load; unbounded queues only move the timeout elsewhere.
- Shared edge cache with a verified-key rule. Maximizes hit rate across users without letting cache hits bypass authentication.
- Release manifest in git. Deployed weights are a reviewed, revertable change pinned to immutable Hub tags, not "latest".
- Eval gate on the production backend. Quantization changes outputs; gating the fp16 adapter alone would not catch a bad GGUF.
- OpenAI-compatible API. Existing SDKs and tools work unchanged, and the same contract fronts a future vLLM backend.
- Retrieval through the training
contextslot, opt-in by default. RAG uses the exact layout the model saw for context examples, and its effect is measured byeval --rag, not assumed. On the base model it raised keyword recall (+0.08) but lowered code validity (-0.14) and added about 60% latency, so it ships opt-in per request until a release shows a net gain. - App factory with injected engine/store/retriever. The full HTTP surface is tested without a model, database, or network.