Deployment and Operations¶
How to take the repository from git clone to a running, monitored, multi-origin
deployment using only free tiers, and how to operate it. Every platform step is
optional and independent: the release pipeline skips any platform whose variables
are not set, so you can adopt them one at a time.
1. Free-tier budget¶
Allowances as published in November 2026; check each provider before relying on them.
| Need | Service | Free allowance | What this project uses |
|---|---|---|---|
| Training GPU | Kaggle | about 30 GPU hours/week (not officially published) | about 1 hour per training run |
| Weights registry | Hugging Face Hub | free public repos | adapter (about 74 MB) + GGUF (about 1 GB) per release |
| Primary origin | Google Cloud Run | 180k vCPU-s, 360k GiB-s, 2M requests per month (request-based billing) | CPU time per uncached answer (no monthly answer count is claimed without a load test; see audit/PERFORMANCE_BASELINE.md) |
| Image storage | Artifact Registry | 0.5 GB | one amd64 image (cleanup policy keeps two) |
| Backup origin | Oracle Cloud Always Free | Ampere A1: 2 OCPU / 12 GB total (halved in 2026) | one VM |
| Edge | Cloudflare Workers | 100k requests/day, 10 ms CPU per request | about 1 ms CPU per request |
| Cache and rate limits | Upstash Redis | about 500k commands/month | 3-5 commands per request |
| Database | Neon Postgres | 0.5-1 GB storage, compute scales to zero | keys, usage, feedback, optional pgvector index |
| Metrics and logs | Grafana Cloud | 10k active series, 50 GB logs, 14-day retention | about 150 series |
| Errors | Sentry | developer plan | optional |
| CI/CD | GitHub Actions + GHCR | free for public repositories | build, eval, deploy, uptime |
Capacity¶
Measured (see audit/PERFORMANCE_BASELINE.md):
- one 2-vCPU Cloud Run instance generates 8 to 10 tokens/s for a single request, one generation at a time;
- an uncached 256-token answer therefore takes roughly 25 to 30 s;
- a burst that forces a cold scale-out can see some requests refused with 429 and a retry hint while the next instance loads the model.
Not claimed: no daily or concurrent-user figure until a controlled load test is run.
Scaling: the cache, extra origins and back-pressure keep behaviour predictable, and scaling up is a configuration change (section 9).
2. Accounts and where each value goes¶
| Value | Where it comes from | Where it goes |
|---|---|---|
GCP_PROJECT_ID, GCP_REGION, GCP_WIF_PROVIDER, GCP_DEPLOY_SA |
printed by deploy/gcp/bootstrap.sh |
GitHub variables |
CLOUD_RUN_MAX_INSTANCES |
your choice (default 2; the hard cost cap) | GitHub variable |
PUBLIC_DEMO |
true to let visitors use the service without a key (default false) |
GitHub variable |
CLOUDFLARE_ACCOUNT_ID |
Cloudflare dashboard | GitHub variable |
CLOUDFLARE_API_TOKEN |
Cloudflare: API token with "Edit Cloudflare Workers" | GitHub secret |
EDGE_ORIGINS |
Cloud Run URL, then backup URLs, comma-separated | GitHub variable |
EDGE_URL |
the deployed Worker URL | GitHub variable |
ORIGIN_SHARED_SECRET |
generated by scripts/setup_secrets.sh |
Worker secret + GitHub secret (Release copies it to Secret Manager); VM .env, Space secret |
UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN |
Upstash console, REST API section | GitHub secrets |
DATABASE_URL |
Neon connection string (sslmode=require) |
Secret Manager database-url, VM .env |
METRICS_TOKEN |
generated by scripts/setup_secrets.sh (saved to ~/.config/fastapi-assistant/metrics_token) |
GitHub secret, Grafana scrape job, VM .env |
SENTRY_DSN |
Sentry project settings | Secret Manager sentry-dsn, VM .env |
SMOKE_API_KEY |
issued by scripts/setup_secrets.sh |
GitHub secret |
HF_NAMESPACE |
your Hugging Face user or org | GitHub variable |
HF_TOKEN |
Hugging Face token with write access | GitHub secret (the Train workflow publishes with it); optionally also a Kaggle secret attached to the fastapi-assistant-train kernel, so the kernel publishes itself (run the Train workflow once with push_only: true to create the kernel) |
KAGGLE_API_TOKEN (or the legacy KAGGLE_KEY) |
Kaggle account settings, API | GitHub secret |
KAGGLE_USERNAME |
your Kaggle username | GitHub variable (a secret of the same name is also accepted) |
Secrets in one command¶
Secret values never go into scripts, workflow logs, or chat. After gh auth login
and wrangler login (in deploy/cloudflare-worker), run from the repository root:
It generates the origin secret, the metrics token, and your API key (printed once),
then optionally asks (hidden input) for the Upstash REST URL/token and the Neon
DATABASE_URL. Press Enter to skip either: the service then runs with your static
key and no edge cache. Values go to the Worker and GitHub's encrypted secrets; with a
database it also applies migrations and issues a smoke-test key. On every deploy,
Release copies the GitHub secrets into Google Secret Manager for Cloud Run, adding a
version only when a value changed.
A full run generates new secrets every time, so do not re-run it on a working deployment just to add Upstash or Neon: it would rotate the origin secret and your API key. Use the storage-only mode instead, then run Release:
It asks only for the Upstash and Neon values, sets them as GitHub secrets (Release
uploads the Upstash pair to the Worker and DATABASE_URL to Secret Manager), and
rotates nothing. It needs gh auth login only.
3. Database (Neon)¶
- Create a Neon project and copy the pooled connection string.
- Apply migrations and issue your first key:
export DATABASE_URL='postgresql://...neon.tech/neondb?sslmode=require'
fastapi-assistant db migrate
fastapi-assistant keys create my-laptop # printed once; store it
RAG_BACKEND=pgvector fastapi-assistant rag build --target pgvector.
The default is a local index baked into the image, which needs no database.
4. Primary origin (Cloud Run)¶
- Create a project with billing enabled (the free tier still requires a billing account), then run:
PROJECT_ID=... BILLING_ACCOUNT=... GITHUB_REPO=owner/Niche-Finetuned-Model \
bash deploy/gcp/bootstrap.sh
main or run the
Release workflow. It builds the image, copies the amd64 variant into Artifact
Registry, renders deploy/cloudrun/service.yaml from model-release.json, and
deploys. The job ends by polling /ready on the new revision.
3. Direct requests to the Cloud Run URL get 403: only the Worker (which sends the
origin secret) can use it. /health and /ready stay open for probes.
5. Edge (Cloudflare Worker + Upstash)¶
- Create an Upstash Redis database; copy the REST URL and token.
- Set the Cloudflare and Upstash values from section 2, with
EDGE_ORIGINSset to the Cloud Run URL. Run Release; the Worker deploys asfastapi-assistant-edge. - Set
EDGE_URLto the Worker URL andSMOKE_API_KEYto a key from section 3. From now on every release ends with a k6 smoke test through the edge, anduptime.ymlprobes it every 10 minutes.
Check it:
curl -s "$EDGE_URL/generate" -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
-d '{"question":"How do I declare a path parameter?"}' -D - -o /dev/null | grep -i -E 'x-cache|x-served-by|x-ratelimit'
The second identical call returns X-Cache: HIT in milliseconds.
The same deploy publishes the web UI: Release builds web/ and Wrangler uploads
web/dist as static assets, so $EDGE_URL/ opens the playground while every API
path still reaches the Worker.
By default visitors need a key: create one with fastapi-assistant keys create demo
and share it. To open the service to anyone, set the repository variable
PUBLIC_DEMO=true and run Release. Release then deploys the Worker with
REQUIRE_API_KEY=false and the origin with PUBLIC_DEMO=true:
- requests without a key are served, limited per IP by
RATE_LIMIT_PER_MINUTE_ANON(10 a minute per isolate, or across isolates with Upstash); - the owner key and per-user keys still work, and a wrong key is still rejected;
GET /healthreports"access": "public", and the web UI tells visitors no key is needed.
Keyless traffic is billed like any other. Keep CLOUD_RUN_MAX_INSTANCES low (the
live demo runs with 1) so the worst case stays bounded, and set the budget alert in
section 1. To close the demo, set PUBLIC_DEMO=false and run Release again.
6. Backup origin (Oracle VM)¶
- Create an Ampere A1 instance (Ubuntu 24.04, 2 OCPU / 12 GB) with
deploy/vm/cloud-init.yamlas user data. No inbound ports are needed. - In Cloudflare Zero Trust, create a tunnel and route a hostname (for example
origin-vm.example.com) tohttp://app:8080. - On the VM:
cp /opt/fastapi-assistant/.env.example /opt/fastapi-assistant/.env, fill it in, thensudo systemctl start fastapi-assistant. - Append
https://origin-vm.example.comtoEDGE_ORIGINSand re-run Release.
Oracle may reclaim Always Free instances that stay nearly idle for a week, and A1
capacity is often unavailable in popular regions; treat this origin as best effort.
An optional third origin is a Hugging Face Docker Space (deploy/hf-space/).
7. Observability¶
- Grafana Cloud: for Cloud Run, add a "Metrics Endpoint" scrape job for
https://<cloud-run-url>/metricswith bearer tokenMETRICS_TOKEN. The VM ships its own metrics and logs through Alloy. Importdeploy/grafana/dashboard.jsonand loaddeploy/grafana/alerts.yaml, either as data-source-managed rules (mimirtool rules load deploy/grafana/alerts.yaml) or, as the live stack does, as Grafana-managed rules (each expression as query A, plus a threshold that fires whenever A returns a series), so they use Grafana's contact points. A healthy rule returns no series, so set each rule's "No data" handling to Normal and its "Error" handling to Keep Last State; the defaults send aDatasourceNoDataemail for every quiet rule. Add an email contact point as the default notification policy. The live stack batches mail to at most about one email a day: the default policy groups every alert together (no group-by labels), with a group interval and a repeat interval of 24h. - Cloud Run's own request, latency, and instance metrics are free in Cloud Monitoring and can be added to Grafana as a Google Cloud Monitoring data source.
- Sentry receives unhandled errors once
SENTRY_DSNis set.
8. Releasing a new model¶
- Grow the data:
fastapi-assistant data generate --out drafts.jsonl --pages 30drafts doc-grounded pairs with Claude (needs Anthropic credentials and costs one request per page). Review them, then merge withfastapi-assistant data merge drafts.jsonl --split trainand runfastapi-assistant data validate. - Run the "Train on Kaggle" workflow. It trains on a free GPU, requires the
fine-tune to beat the base model, exports and publishes the GGUF, re-scores it
with the production backend, and opens a PR that updates
model-release.jsonandeval/baseline.json. Opening that PR needs the repository setting Settings > Actions > General > "Allow GitHub Actions to create and approve pull requests"; without it the workflow still pushes therelease/<version>branch and prints a link to open the PR by hand. The kernel publishes to the Hub itself whenHF_TOKENis attached to it on Kaggle; otherwise the workflow publishes with theHF_TOKENGitHub secret. If a run trained the model but failed afterwards, run the workflow withpublish_only: trueto reuse that run's output instead of training again. - Review the eval table in the PR and merge. Release redeploys Cloud Run and the
Worker; the edge job also bumps the Worker's cache namespace (
MODEL_VERSION). The edge job runs only whenCLOUDFLARE_ACCOUNT_IDandCLOUDFLARE_API_TOKENare set (both are set for this repository); without them, redeploy the Worker by hand with--var ORIGINS:<origins> --var MODEL_VERSION:<new version>.
9. Runbook¶
| Situation | Action |
|---|---|
| Bad model in production | Revert the release PR (restores the previous model-release.json); Release redeploys. Weights are immutable tags, so rollback never re-trains. |
| Leaked API key | fastapi-assistant keys revoke fa_AbC123. Origins stop accepting it within KEY_CACHE_TTL_S (60 s); cached edge answers stop within KEY_VERIFIED_TTL_SECONDS (300 s). |
Sustained 429s (AssistantSaturated) |
Raise CLOUD_RUN_MAX_INSTANCES (costs money beyond the free tier), add an origin, or lower RATE_LIMIT_PER_MINUTE. |
| Model never becomes ready | Check the logs for the download or load error (load_state: failed); verify the revision in model-release.json exists on the Hub. |
| Rotate the origin secret | Run bash scripts/setup_secrets.sh (updates the GitHub secret and the Worker secret together), update the VM .env if one exists, then re-run Release, which copies the GitHub value into Secret Manager. Do not edit Secret Manager directly: the next Release overwrites it. Brief 403s are possible while origin and edge disagree. |
| Unexpected bill | maxScale is the hard cap; the budget alert only notifies. Set CLOUD_RUN_MAX_INSTANCES=1 and re-run Release. |
| Postgres outage | Static API_KEY traffic keeps working; per-user keys fail closed. Usage writes are dropped and logged. |
10. Scaling beyond free tiers¶
In order: raise maxScale (no code change); add origins behind the Worker; move
inference to a GPU server running vLLM or TGI with the merged model and set
ENGINE_BACKEND=remote and REMOTE_BASE_URL (the service keeps auth, admission,
metrics, and the API); and fine-tune a larger base model by setting base_model
when running the Kaggle workflow. The eval gate decides whether a larger model ships.