Skip to content

Deployment and Operations

How to take the repository from git clone to a running, monitored, multi-origin deployment using only free tiers, and how to operate it. Every platform step is optional and independent: the release pipeline skips any platform whose variables are not set, so you can adopt them one at a time.

1. Free-tier budget

Allowances as published in November 2026; check each provider before relying on them.

Need Service Free allowance What this project uses
Training GPU Kaggle about 30 GPU hours/week (not officially published) about 1 hour per training run
Weights registry Hugging Face Hub free public repos adapter (about 74 MB) + GGUF (about 1 GB) per release
Primary origin Google Cloud Run 180k vCPU-s, 360k GiB-s, 2M requests per month (request-based billing) CPU time per uncached answer (no monthly answer count is claimed without a load test; see audit/PERFORMANCE_BASELINE.md)
Image storage Artifact Registry 0.5 GB one amd64 image (cleanup policy keeps two)
Backup origin Oracle Cloud Always Free Ampere A1: 2 OCPU / 12 GB total (halved in 2026) one VM
Edge Cloudflare Workers 100k requests/day, 10 ms CPU per request about 1 ms CPU per request
Cache and rate limits Upstash Redis about 500k commands/month 3-5 commands per request
Database Neon Postgres 0.5-1 GB storage, compute scales to zero keys, usage, feedback, optional pgvector index
Metrics and logs Grafana Cloud 10k active series, 50 GB logs, 14-day retention about 150 series
Errors Sentry developer plan optional
CI/CD GitHub Actions + GHCR free for public repositories build, eval, deploy, uptime

Capacity

Measured (see audit/PERFORMANCE_BASELINE.md):

  • one 2-vCPU Cloud Run instance generates 8 to 10 tokens/s for a single request, one generation at a time;
  • an uncached 256-token answer therefore takes roughly 25 to 30 s;
  • a burst that forces a cold scale-out can see some requests refused with 429 and a retry hint while the next instance loads the model.

Not claimed: no daily or concurrent-user figure until a controlled load test is run.

Scaling: the cache, extra origins and back-pressure keep behaviour predictable, and scaling up is a configuration change (section 9).

2. Accounts and where each value goes

Value Where it comes from Where it goes
GCP_PROJECT_ID, GCP_REGION, GCP_WIF_PROVIDER, GCP_DEPLOY_SA printed by deploy/gcp/bootstrap.sh GitHub variables
CLOUD_RUN_MAX_INSTANCES your choice (default 2; the hard cost cap) GitHub variable
PUBLIC_DEMO true to let visitors use the service without a key (default false) GitHub variable
CLOUDFLARE_ACCOUNT_ID Cloudflare dashboard GitHub variable
CLOUDFLARE_API_TOKEN Cloudflare: API token with "Edit Cloudflare Workers" GitHub secret
EDGE_ORIGINS Cloud Run URL, then backup URLs, comma-separated GitHub variable
EDGE_URL the deployed Worker URL GitHub variable
ORIGIN_SHARED_SECRET generated by scripts/setup_secrets.sh Worker secret + GitHub secret (Release copies it to Secret Manager); VM .env, Space secret
UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN Upstash console, REST API section GitHub secrets
DATABASE_URL Neon connection string (sslmode=require) Secret Manager database-url, VM .env
METRICS_TOKEN generated by scripts/setup_secrets.sh (saved to ~/.config/fastapi-assistant/metrics_token) GitHub secret, Grafana scrape job, VM .env
SENTRY_DSN Sentry project settings Secret Manager sentry-dsn, VM .env
SMOKE_API_KEY issued by scripts/setup_secrets.sh GitHub secret
HF_NAMESPACE your Hugging Face user or org GitHub variable
HF_TOKEN Hugging Face token with write access GitHub secret (the Train workflow publishes with it); optionally also a Kaggle secret attached to the fastapi-assistant-train kernel, so the kernel publishes itself (run the Train workflow once with push_only: true to create the kernel)
KAGGLE_API_TOKEN (or the legacy KAGGLE_KEY) Kaggle account settings, API GitHub secret
KAGGLE_USERNAME your Kaggle username GitHub variable (a secret of the same name is also accepted)

Secrets in one command

Secret values never go into scripts, workflow logs, or chat. After gh auth login and wrangler login (in deploy/cloudflare-worker), run from the repository root:

bash scripts/setup_secrets.sh

It generates the origin secret, the metrics token, and your API key (printed once), then optionally asks (hidden input) for the Upstash REST URL/token and the Neon DATABASE_URL. Press Enter to skip either: the service then runs with your static key and no edge cache. Values go to the Worker and GitHub's encrypted secrets; with a database it also applies migrations and issues a smoke-test key. On every deploy, Release copies the GitHub secrets into Google Secret Manager for Cloud Run, adding a version only when a value changed.

A full run generates new secrets every time, so do not re-run it on a working deployment just to add Upstash or Neon: it would rotate the origin secret and your API key. Use the storage-only mode instead, then run Release:

bash scripts/setup_secrets.sh --storage-only

It asks only for the Upstash and Neon values, sets them as GitHub secrets (Release uploads the Upstash pair to the Worker and DATABASE_URL to Secret Manager), and rotates nothing. It needs gh auth login only.

3. Database (Neon)

  1. Create a Neon project and copy the pooled connection string.
  2. Apply migrations and issue your first key:

export DATABASE_URL='postgresql://...neon.tech/neondb?sslmode=require'
fastapi-assistant db migrate
fastapi-assistant keys create my-laptop      # printed once; store it
3. Optional, for shared retrieval across replicas: RAG_BACKEND=pgvector fastapi-assistant rag build --target pgvector. The default is a local index baked into the image, which needs no database.

4. Primary origin (Cloud Run)

  1. Create a project with billing enabled (the free tier still requires a billing account), then run:

PROJECT_ID=... BILLING_ACCOUNT=... GITHUB_REPO=owner/Niche-Finetuned-Model \
  bash deploy/gcp/bootstrap.sh
2. Add the printed values as GitHub variables, then push to main or run the Release workflow. It builds the image, copies the amd64 variant into Artifact Registry, renders deploy/cloudrun/service.yaml from model-release.json, and deploys. The job ends by polling /ready on the new revision. 3. Direct requests to the Cloud Run URL get 403: only the Worker (which sends the origin secret) can use it. /health and /ready stay open for probes.

5. Edge (Cloudflare Worker + Upstash)

  1. Create an Upstash Redis database; copy the REST URL and token.
  2. Set the Cloudflare and Upstash values from section 2, with EDGE_ORIGINS set to the Cloud Run URL. Run Release; the Worker deploys as fastapi-assistant-edge.
  3. Set EDGE_URL to the Worker URL and SMOKE_API_KEY to a key from section 3. From now on every release ends with a k6 smoke test through the edge, and uptime.yml probes it every 10 minutes.

Check it:

curl -s "$EDGE_URL/generate" -H "X-API-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"question":"How do I declare a path parameter?"}' -D - -o /dev/null | grep -i -E 'x-cache|x-served-by|x-ratelimit'

The second identical call returns X-Cache: HIT in milliseconds.

The same deploy publishes the web UI: Release builds web/ and Wrangler uploads web/dist as static assets, so $EDGE_URL/ opens the playground while every API path still reaches the Worker.

By default visitors need a key: create one with fastapi-assistant keys create demo and share it. To open the service to anyone, set the repository variable PUBLIC_DEMO=true and run Release. Release then deploys the Worker with REQUIRE_API_KEY=false and the origin with PUBLIC_DEMO=true:

  • requests without a key are served, limited per IP by RATE_LIMIT_PER_MINUTE_ANON (10 a minute per isolate, or across isolates with Upstash);
  • the owner key and per-user keys still work, and a wrong key is still rejected;
  • GET /health reports "access": "public", and the web UI tells visitors no key is needed.

Keyless traffic is billed like any other. Keep CLOUD_RUN_MAX_INSTANCES low (the live demo runs with 1) so the worst case stays bounded, and set the budget alert in section 1. To close the demo, set PUBLIC_DEMO=false and run Release again.

6. Backup origin (Oracle VM)

  1. Create an Ampere A1 instance (Ubuntu 24.04, 2 OCPU / 12 GB) with deploy/vm/cloud-init.yaml as user data. No inbound ports are needed.
  2. In Cloudflare Zero Trust, create a tunnel and route a hostname (for example origin-vm.example.com) to http://app:8080.
  3. On the VM: cp /opt/fastapi-assistant/.env.example /opt/fastapi-assistant/.env, fill it in, then sudo systemctl start fastapi-assistant.
  4. Append https://origin-vm.example.com to EDGE_ORIGINS and re-run Release.

Oracle may reclaim Always Free instances that stay nearly idle for a week, and A1 capacity is often unavailable in popular regions; treat this origin as best effort. An optional third origin is a Hugging Face Docker Space (deploy/hf-space/).

7. Observability

  • Grafana Cloud: for Cloud Run, add a "Metrics Endpoint" scrape job for https://<cloud-run-url>/metrics with bearer token METRICS_TOKEN. The VM ships its own metrics and logs through Alloy. Import deploy/grafana/dashboard.json and load deploy/grafana/alerts.yaml, either as data-source-managed rules (mimirtool rules load deploy/grafana/alerts.yaml) or, as the live stack does, as Grafana-managed rules (each expression as query A, plus a threshold that fires whenever A returns a series), so they use Grafana's contact points. A healthy rule returns no series, so set each rule's "No data" handling to Normal and its "Error" handling to Keep Last State; the defaults send a DatasourceNoData email for every quiet rule. Add an email contact point as the default notification policy. The live stack batches mail to at most about one email a day: the default policy groups every alert together (no group-by labels), with a group interval and a repeat interval of 24h.
  • Cloud Run's own request, latency, and instance metrics are free in Cloud Monitoring and can be added to Grafana as a Google Cloud Monitoring data source.
  • Sentry receives unhandled errors once SENTRY_DSN is set.

8. Releasing a new model

  1. Grow the data: fastapi-assistant data generate --out drafts.jsonl --pages 30 drafts doc-grounded pairs with Claude (needs Anthropic credentials and costs one request per page). Review them, then merge with fastapi-assistant data merge drafts.jsonl --split train and run fastapi-assistant data validate.
  2. Run the "Train on Kaggle" workflow. It trains on a free GPU, requires the fine-tune to beat the base model, exports and publishes the GGUF, re-scores it with the production backend, and opens a PR that updates model-release.json and eval/baseline.json. Opening that PR needs the repository setting Settings > Actions > General > "Allow GitHub Actions to create and approve pull requests"; without it the workflow still pushes the release/<version> branch and prints a link to open the PR by hand. The kernel publishes to the Hub itself when HF_TOKEN is attached to it on Kaggle; otherwise the workflow publishes with the HF_TOKEN GitHub secret. If a run trained the model but failed afterwards, run the workflow with publish_only: true to reuse that run's output instead of training again.
  3. Review the eval table in the PR and merge. Release redeploys Cloud Run and the Worker; the edge job also bumps the Worker's cache namespace (MODEL_VERSION). The edge job runs only when CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN are set (both are set for this repository); without them, redeploy the Worker by hand with --var ORIGINS:<origins> --var MODEL_VERSION:<new version>.

9. Runbook

Situation Action
Bad model in production Revert the release PR (restores the previous model-release.json); Release redeploys. Weights are immutable tags, so rollback never re-trains.
Leaked API key fastapi-assistant keys revoke fa_AbC123. Origins stop accepting it within KEY_CACHE_TTL_S (60 s); cached edge answers stop within KEY_VERIFIED_TTL_SECONDS (300 s).
Sustained 429s (AssistantSaturated) Raise CLOUD_RUN_MAX_INSTANCES (costs money beyond the free tier), add an origin, or lower RATE_LIMIT_PER_MINUTE.
Model never becomes ready Check the logs for the download or load error (load_state: failed); verify the revision in model-release.json exists on the Hub.
Rotate the origin secret Run bash scripts/setup_secrets.sh (updates the GitHub secret and the Worker secret together), update the VM .env if one exists, then re-run Release, which copies the GitHub value into Secret Manager. Do not edit Secret Manager directly: the next Release overwrites it. Brief 403s are possible while origin and edge disagree.
Unexpected bill maxScale is the hard cap; the budget alert only notifies. Set CLOUD_RUN_MAX_INSTANCES=1 and re-run Release.
Postgres outage Static API_KEY traffic keeps working; per-user keys fail closed. Usage writes are dropped and logged.

10. Scaling beyond free tiers

In order: raise maxScale (no code change); add origins behind the Worker; move inference to a GPU server running vLLM or TGI with the merged model and set ENGINE_BACKEND=remote and REMOTE_BASE_URL (the service keeps auth, admission, metrics, and the API); and fine-tune a larger base model by setting base_model when running the Kaggle workflow. The eval gate decides whether a larger model ships.