Deployment architecture¶
What runs where, how a commit on main becomes a running revision, and which parts
are configured but not deployed. Step-by-step setup lives in
docs/DEPLOYMENT.md; this document describes the result.
1. Live versus configured¶
Status as recorded on 2026-11-13 in docs/audit/INITIAL_AUDIT.md, docs/audit/FINAL_VERIFICATION.md and docs/IMPLEMENTATION_STATUS.md.
This document set describes the code on main. The hardening changes (PR #13) and
the follow-ups #14 to #16 are merged and deployed: behaviour such as the fallback rate
limiter, the body caps, the token budget and GGUF verification is live. Release
deploys both Cloud Run and the Worker from main.
| Component | Status | Evidence |
|---|---|---|
Edge Worker fastapi-assistant-edge with the web UI, at https://fastapi-assistant-edge.fastapi-assistant-edge.workers.dev |
Live. Deployed by Release (deploy-edge) from main. |
Release run 43460008003; F-10 (resolved) |
Cloud Run service fastapi-assistant |
Live, deployed by Release from main |
Audit header; .github/workflows/release.yml |
| Model | The fine-tuned release v2026.11.14-72d6b37: a LoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct as a Q4_K_M GGUF (about 1 GB), live since 2026-11-15 (#53). It replaced v2026.11.13-2b04940, a TinyLlama-1.1B-Chat fine-tune |
model-release.json ("finetuned": true); F-04 (resolved in #33) |
| Authentication | Static owner key from Secret Manager; origin locked to the edge | Implementation status |
| Secret Manager, Artifact Registry, Workload Identity Federation, budget alert | Live | Audit section 5; deploy/gcp/bootstrap.sh |
| Upstash Redis (shared rate limit and cache) | Live since 2026-11-14 (database fastapi-assistant-edge, free tier). If it fails, rate limiting falls back to per-isolate counters and the cache is skipped. |
F-03; implementation status |
| Neon Postgres (per-user keys, usage, feedback) | Live since 2026-11-14 (database fastapi_assistant, free plan) |
Implementation status |
| Sentry, Grafana Cloud | Not connected | Implementation status |
uptime.yml |
Live: EDGE_URL is set |
Implementation status |
Release jobs deploy-edge, smoke-edge |
Live: CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are set. Both jobs succeeded in Release run 43460008003 (the merge of #15). |
F-10 (resolved); release.yml |
| Backup origins: Hugging Face Space, Oracle VM | Configuration only, not deployed. ORIGINS lists Cloud Run only. |
F-11; deploy/hf-space/, deploy/vm/ |
| Train on Kaggle | Configured: the Kaggle API token and username, HF_NAMESPACE, and HF_TOKEN attached to the kernel on Kaggle. The first runs exposed Kaggle environment problems, fixed in #15 and #16. Later runs trained v2026.11.13-2b04940 (deployed in #33) and v2026.11.14-72d6b37 (Train run 43622001406, deployed in #53). |
F-04; .github/workflows/train.yml |
2. Delivery pipeline¶
All the jobs below run on every Release today. A deploy job is skipped only when its platform variable is not set (section 3).
flowchart TB
Push["push to main, or manual dispatch"] --> Build
subgraph Release["release.yml"]
Build["build<br/>amd64 and arm64 on native runners<br/>push by digest"]
Publish["publish<br/>multi-arch manifest<br/>tags sha-12 and latest"]
Scan["scan<br/>Trivy SARIF upload<br/>gate: fixable HIGH and CRITICAL"]
DeployRun["deploy-cloudrun<br/>runs if GCP_PROJECT_ID is set"]
DeployEdge["deploy-edge<br/>runs if CLOUDFLARE_ACCOUNT_ID is set"]
Smoke["smoke-edge<br/>k6 smoke, runs if EDGE_URL is set"]
Build --> Publish --> Scan --> DeployRun --> DeployEdge --> Smoke
end
GHCR[("GHCR<br/>multi-arch image")]
Publish --> GHCR
subgraph GCP["Google Cloud project"]
AR[("Artifact Registry<br/>amd64 image")]
SM[("Secret Manager")]
CR["Cloud Run<br/>fastapi-assistant"]
end
DeployRun -->|"Workload Identity Federation"| GCP
DeployRun -->|"copy amd64 digest"| AR
DeployRun -->|"add version only if changed"| SM
DeployRun -->|"service.yaml rendered from model-release.json"| CR
AR --> CR
SM --> CR
Worker["Cloudflare Worker<br/>plus web/dist assets"]
DeployEdge -->|"wrangler deploy"| Worker
Smoke -->|"k6 through the edge"| Worker
Since #15, deploy-edge and smoke-edge run on every Release (first success: run
43460008003).
3. Release workflow stages¶
.github/workflows/release.yml runs on every push to main (skipped when every
changed file is Markdown or under docs/, graphify-out/, eval/ or tests/load/)
and on manual dispatch. Runs never overlap (concurrency: release, not cancelled).
| Job | Runs when | What it does | Gate |
|---|---|---|---|
build |
Always | Builds linux/amd64 on ubuntu-24.04 and linux/arm64 on ubuntu-24.04-arm from the same Dockerfile; pushes each by digest to GHCR. |
Build must succeed |
publish |
After build |
Creates one multi-arch manifest tagged sha-<12 chars> and latest. |
|
scan |
After publish |
Trivy scans the published image twice: once to upload SARIF to code scanning, once with exit-code: 1 on fixable CRITICAL,HIGH (ignore-unfixed: true). /models/rag_index is skipped because the baked documentation contains an example JWT (F-21). |
Fails the release on a fixable HIGH or CRITICAL CVE |
deploy-cloudrun |
vars.GCP_PROJECT_ID set; after scan |
Authenticates with Workload Identity Federation (no key file). Copies the amd64 image into Artifact Registry. Syncs five secrets into Secret Manager. Renders deploy/cloudrun/service.yaml with envsubst from repository variables and fastapi-assistant export env model-release.json. Deploys, grants allUsers the invoker role, then polls /health and /ready on the new URL. |
Smoke: /ready must answer 200 |
deploy-edge |
vars.CLOUDFLARE_ACCOUNT_ID set; not after a failure |
Builds web/ and runs wrangler deploy with MODEL_VERSION from model-release.json (the cache namespace) and ORIGINS from vars.EDGE_ORIGINS. Uploads the Worker secret ORIGIN_SHARED_SECRET, plus UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN only when both GitHub secrets are set (#15). Authenticates with secrets.CLOUDFLARE_API_TOKEN; the job condition checks only the account ID variable, so the token must exist too or wrangler fails. |
|
smoke-edge |
vars.EDGE_URL set and deploy-edge succeeded |
k6 run -e SCENARIO=smoke tests/load/k6.js through the edge. |
k6 thresholds |
Both deploy jobs use the production GitHub environment.
Secrets flow¶
GitHub encrypted secrets are the source of truth; they are created by
scripts/setup_secrets.sh and never printed (except the operator's own API key,
once). deploy-cloudrun copies them into Secret Manager, adding a version only when
the value changed. Cloud Run reads them as environment variables.
| Secret Manager name | Environment variable | Required |
|---|---|---|
origin-shared-secret |
ORIGIN_SHARED_SECRET |
Yes. The sync fails if the GitHub secret is missing. |
api-key |
API_KEY |
Yes |
metrics-token |
METRICS_TOKEN |
Yes |
database-url |
DATABASE_URL |
No. disabled placeholder, treated as unset by config.py. |
sentry-dsn |
SENTRY_DSN |
No. disabled placeholder. |
deploy/gcp/bootstrap.sh grants the deployer service account roles/run.admin,
roles/artifactregistry.writer and roles/secretmanager.admin, and the runtime
service account only roles/secretmanager.secretAccessor. The Workload Identity
provider accepts tokens only from this repository's refs/heads/main.
4. Cloud Run service¶
Values from deploy/cloudrun/service.yaml after rendering.
| Setting | Value | Why |
|---|---|---|
minScale |
0 | Scale to zero when idle |
maxScale |
CLOUD_RUN_MAX_INSTANCES, default 2 |
The only hard spending limit; the budget alert only notifies |
cpu-throttling |
true |
Request-based billing: CPU billed only while serving requests |
startup-cpu-boost |
true |
Faster cold start (model load) |
| Execution environment | gen2 |
|
| CPU, memory | 2 vCPU, 4 GiB | The 1 GB Qwen GGUF ran the instance at 99% of 2 GiB, so memory was raised to 4 GiB; LLAMACPP_N_THREADS=2 |
containerConcurrency |
4 | Sized to one running generation plus MAX_QUEUE=3 waiting (comment in the file) |
| Container port | h2c (8080) |
Cloud Run talks cleartext HTTP/2 to the container, which is how it reports a client's disconnect. The image serves it with Hypercorn (HTTP_SERVER=hypercorn); see DECISIONS.md, ADR-009 |
timeoutSeconds |
300 | Whole request, including a stream |
MAX_CONCURRENCY, MAX_QUEUE, QUEUE_TIMEOUT_S |
1, 3, 60 | Admission control (admission.py) |
RAG_ENABLED |
false |
Retrieval opt-in per request |
| Startup probe | GET /ready every 5 s, timeout 3 s, 48 failures allowed |
Up to 240 s for download, SHA-256 check and load |
| Liveness probe | GET /health every 30 s, timeout 5 s |
|
| Ingress | all, with allUsers as invoker |
The origin secret, not IAM, blocks direct use (403) |
| Template label | model-version |
Each revision is labelled with the release it serves |
Model environment (MODEL_VERSION, GGUF_REPO_ID, GGUF_FILENAME, GGUF_REVISION,
GGUF_SHA256, FINETUNED_WEIGHTS, LLAMACPP_CHAT_FORMAT) comes from
model-release.json through export.manifest_env.
5. Container image¶
Dockerfile, two stages:
| Stage | Content |
|---|---|
builder (python:3.11-slim) |
build-essential and cmake; pip wheel --require-hashes -r requirements/serve.txt; llama.cpp compiled with GGML_NATIVE=OFF so one image runs on any CPU of its architecture. |
runtime (python:3.11-slim) |
Debian security updates applied; libgomp1; wheels installed offline; documentation index baked at /models/rag_index (BAKE_RAG_INDEX=true); GGUF not baked (BAKE_MODEL=false); pip, setuptools and wheel uninstalled; runs as uid 10001; HEALTHCHECK on /health. |
Defaults baked into the image name the official Qwen Q4_K_M GGUF
(Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF, the base model without a fine-tune), with
its revision and SHA-256, so the image boots without extra configuration (CI relies
on this in its container smoke test). Release overrides them with the values from
model-release.json, which names the fine-tuned GGUF. Why weights are downloaded rather than baked:
DECISIONS.md, ADR-007.
6. Edge deployment¶
deploy/cloudflare-worker/wrangler.toml defines the Worker, its [vars] defaults
(RATE_LIMIT_PER_MINUTE 20, RATE_LIMIT_PER_MINUTE_ANON 10, CACHE_TTL_SECONDS
86,400, ORIGIN_TIMEOUT_MS 150,000, REQUIRE_API_KEY true,
KEY_VERIFIED_TTL_SECONDS 300, MAX_REQUEST_BYTES 65,536) and the static assets
directory ../../web/dist. ORIGINS is empty in the file on purpose; an unset value
fails closed with 502. Secrets (ORIGIN_SHARED_SECRET, Upstash URL and token) are set
with wrangler secret put, never in the file.
Release deploys the Worker on every run (F-10, resolved in #15), passing ORIGINS and
MODEL_VERSION, so the live Worker and its cache namespace track main. A manual
wrangler deploy is an emergency fallback only; it must pass both vars
(docs/operations/RUNBOOK.md section 7.7), and the next
Release replaces it.
7. Backup origins (not deployed)¶
| Origin | Files | Shape |
|---|---|---|
| Hugging Face Docker Space | deploy/hf-space/Dockerfile, README.md |
Runs the published GHCR image unchanged as uid 1000. |
| Oracle Cloud Always Free VM | deploy/vm/cloud-init.yaml, docker-compose.yml, config.alloy, .env.example |
app, cloudflared (outbound-only tunnel, no inbound ports) and Grafana Alloy (metrics and logs). MAX_QUEUE=4. |
To activate one, append its URL to EDGE_ORIGINS and re-run Release, which redeploys
the Worker. The
Worker's failover and circuit breaker are unit-tested
(deploy/cloudflare-worker/test/) but have not run in production (F-11).
8. Other workflows¶
| Workflow | Trigger | Purpose |
|---|---|---|
ci.yml |
Push and PR to main |
Python 3.10 to 3.12 tests, ruff, mypy (3.11), data validate; Postgres + pgvector integration; Worker and web typecheck and tests; web build; gitleaks over full history; container build, boot, real answer, 401, 400 for a token-dense prompt, 413 for a 70 KB body, Trivy. |
security.yml |
Push, PR, weekly | pip-audit on requirements/serve.txt, npm audit (Worker, web), dependency review on PRs |
codeql.yml |
Push, PR, weekly | CodeQL for Python, JavaScript/TypeScript and Actions |
eval.yml |
PRs touching the model, eval data, prompts, engines or evaluation; called by train.yml |
Model eval gate, see MODEL_PIPELINE.md |
train.yml |
Manual | Kaggle training to a release PR; the push_only input creates or updates the kernel without running it |
uptime.yml |
Every 10 minutes, if EDGE_URL is set |
GET /health through the edge, then the authenticated GET /v1/models when SMOKE_API_KEY is set; also keeps one instance warm |
9. Rollout and rollback of service code¶
A merge to main produces a new image tagged sha-<12> and a new Cloud Run
revision. Release deploys a revision only after the Trivy gate passes, and the job
fails if /ready does not answer. To roll back service code, revert the commit;
Release rebuilds and redeploys. Artifact Registry keeps the two most recent images
(cleanup policy in deploy/gcp/bootstrap.sh). Model rollouts and rollbacks are
covered in MODEL_PIPELINE.md.