Changelog¶
All notable changes to this project are documented in this file.
The format is based on Keep a Changelog, and
the project aims to follow Semantic Versioning.
Version numbers are the version field in pyproject.toml.
The deployed model is versioned separately, by model_version in
model-release.json. The deployed model is v2026.11.14-72d6b37
(finetuned: true), a LoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct, live since
2026-11-15 (#53). It replaced v2026.11.13-2b04940, a TinyLlama-1.1B-Chat fine-tune,
which had replaced the base TinyLlama (base-tinyllama-q4km) in #33. See
MODEL_CARD.md.
Unreleased¶
Added¶
- Training: the base model is now Qwen2.5-Coder-1.5B-Instruct (Apache-2.0), in place
of TinyLlama-1.1B-Chat. LoRA now also adapts the MLP projections, the Kaggle batch is
4 x 4 (same effective size, fits a T4 with the larger vocabulary), and the image's
fallback weights are the official Qwen Q4_K_M GGUF.
STOP_SEQUENCEScovers the role markers of both chat templates, so either model serves correctly. - Dataset: 151 new training rows (351 to 502) covering the FastAPI tutorial and advanced guide; the eval set is unchanged.
- Web UI: the finished conversation survives a page reload (kept in
sessionStoragefor the tab; answers still streaming and the API key are never stored), and "Download as Markdown" saves it asfastapi-assistant-YYYY-MM-DD.md(#41, #42). - Eval report: the category table compares each category with the baseline, so a release that wins overall but loses a topic shows it (#43).
- Edge Worker: the request log records
rl_key_id, the first 12 hex characters of the SHA-256 of the presented API key, so requests can be attributed to a key. The key itself and the caller's IP are still never logged (#27). - Grafana dashboard: panels for queue wait (p50 and p95), time to first token by
endpoint, and prompts rejected for length. A new alert rule,
AssistantLongQueueWait, fires when the p95 wait for a generation slot stays above 20 s.tests/test_grafana.pychecks every dashboard and alert query against the metrics the service exports (#28). examples/fetch_stream.mjs: a JavaScript example with no dependencies that streams an answer from/v1/chat/completionswith the built-infetch, parses the server-sent events, and explains401,413,429and503responses. It is tested against a real server like the other examples (#29).
Changed¶
- The served model is
v2026.11.14-72d6b37, a LoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct, merged and quantized to Q4_K_M (model-Q4_K_M.gguf, about 0.99 GB) and published on the Hugging Face Hub with its adapter. It replacesv2026.11.13-2b04940, the TinyLlama-1.1B-Chat fine-tune. Train run 43622001406 trained it from72d6b37(#44) on a Kaggle T4 GPU, and it passed both gates: - gate 1, on Kaggle (
transformers, fp16): 0.6716 against 0.5576 for the base Qwen2.5-Coder-1.5B-Instruct (keyword recall 0.7114 against 0.6219, code validity 0.8558 against 0.7212, ROUGE-L 0.4548 against 0.3450); - gate 2, the production eval in CI (GGUF on llama.cpp, greedy, 256 new tokens):
0.6663 against 0.5463 for
v2026.11.13-2b04940(keyword recall 0.6852 against 0.4259, code validity 0.8846 against 0.8462, ROUGE-L 0.4488 against 0.3866). It is slower: latency p50 5.725 s against 4.426 s on the CI runner.
eval/baseline.json now holds its scores. Release run 43696171917 deployed it
after #53; on Cloud Run it generates about 7.6 tokens/s. Its answers are better but
still often wrong.
Fixed¶
- Alerts:
AssistantThroughputDropno longer fires while the service is idle. Its denominator was clamped to1e-9, so 30 minutes without a generation read as a mean of 0 tokens/s and the rule fired, emailing every few hours. The docs now say to set "No data" handling to Normal for the Grafana-managed copies of the rules, and how the live stack limits alert mail to about one email a day. - Eval: code blocks are found by pairing fences, so a
bashortomlblock before the Python code no longer turns the prose between them into a "code block" that fails to parse (#45, fixed by an outside contributor in #47). - The eval workflow no longer fails pull requests from forks at its comment step; the report goes to the job summary instead (#50).
scripts/setup_secrets.sh --storage-onlyaccepts values pasted straight from the Upstash and Neon consoles (NAME="value",psql '...', bracketed-paste markers) and can read each value from its environment variable (#46).- Security: GitHub's own actions are pinned to commit SHAs like the third-party ones (audit item O-10, #40).
- A non-streaming request (
POST /generate, orPOST /v1/chat/completionswithoutstream) stops generating when its caller disconnects, as streamed requests already did. Until now it kept the only generation slot busy for up toGENERATION_TIMEOUT_S.Engine.generatetakes an optionalcancelevent; the remote engine honours it only before the request is sent (#26). - Abandoned requests on the live service. Measured after #37, the fix above had no effect there: a request sent behind an abandoned generation still waited about 60 s, streamed or not, because Cloud Run does not report a client's disconnect to the container over HTTP/1.1 (#38):
- the image serves HTTP/1.1 and cleartext HTTP/2 with Hypercorn
(
HTTP_SERVER=hypercorn; Uvicorn stays the default elsewhere), and Cloud Run talks h2c to it, which is how it reports disconnects; - streamed requests get the same disconnect watch as non-streaming ones. Behind the request middleware the response layer did not end an abandoned stream, and under Hypercorn it stopped consuming it without releasing the slot;
- the Edge Worker aborts its origin request when the client disconnects
(
enable_request_signal), logsclient_gonewith status 499, does not count the origin as failed and does not try a backup origin.
Measured live after the deploy, the request behind an abandoned generation was
answered in 1.3 s (non-streaming) and 2.2 s (streamed), against 59.8 s and
58.4 s before. scripts/check_abandoned_request.sh repeats the measurement.
2.2.0 - 2026-11-13¶
The first fine-tuned model in production, a public demo that needs no key, a documentation website, and a redesigned web UI. Merged in #23 to #36.
Added¶
- A dev container (
.devcontainer/) for GitHub Codespaces and VS Code Dev Containers: Python 3.11 with CPU-only torch, Node.js 22, the test dependencies and the pre-commit hooks. Dependabot keeps its features up to date. ROADMAP.md, GitHub Discussions for questions and ideas, starter issues labelledgood first issue, and a social-preview image (docs/assets/social-preview.png).- Public demo mode. With the repository variable
PUBLIC_DEMO=true, Release deploys the Worker without a key requirement and the origin withPUBLIC_DEMO=true: requests without a key are served, rate limited per IP at the edge, while a wrong key is still rejected.GET /healthreportsaccess(publicorkey) (#24). - A documentation site built with MkDocs Material and published to GitHub Pages
from
main, with a strict build that fails on broken links and anchors (#30). - A redesigned web UI: a welcome screen with the model's status and example
questions, onboarding driven by
/healthaccess, light and dark themes, a drawer on narrow screens, code blocks with copy buttons, and WCAG AA contrast (#32). fastapi-assistant export publish-outputs, and apublish_onlyinput on the Train workflow: CI can publish a finished Kaggle run to the Hugging Face Hub when the kernel could not (#23).
Changed¶
- The served model is the first fine-tuned release,
v2026.11.13-2b04940: a LoRA fine-tune of TinyLlama-1.1B-Chat, merged and quantized to Q4_K_M, published on the Hugging Face Hub. On the held-out set it scores 0.546 against the base model's 0.402 (code validity 0.846 against 0.490), scored with the production backend in the same run.eval/baseline.jsonnow holds its scores (#33). - The live deployment runs as a public demo, with
CLOUD_RUN_MAX_INSTANCES=1to bound the worst-case cost of keyless traffic. - When the Train workflow cannot open the release pull request, it now says which repository setting to enable and links to open the pull request from the pushed branch.
- The model cards published to the Hugging Face Hub link to the source, the demo and
the documentation, show how to load the exact release with llama.cpp or
peft, give the system prompt and the stop sequences, compare the scores with the base model's, and list the limitations.export pushgains--base-report, and the Kaggle pipeline passes the base model's report (#36). scripts/setup_secrets.sh --storage-onlyadds Upstash or Neon to a working deployment without rotating the origin secret or the API key, which a full run does (#36).
Fixed¶
- Answers no longer run on into an invented next turn. The chat template's role
markers (
<|user|>,<|system|>,<|assistant|>) are plain text to TinyLlama, so when the model skipped end-of-sequence it wrote a made-up follow-up question. The llama.cpp and remote engines now stop at them (prompts.STOP_SEQUENCES) (#35). - The retrieval index no longer contains the mkdocstrings directives of the FastAPI
reference pages (
fastapi.BackgroundTasksand their options blocks), which the model copied into answers. The index has 1,524 chunks instead of 1,534 (#35).
2.1.0 - 2026-11-13¶
Production hardening, driven by a baseline audit (docs/audit/INITIAL_AUDIT.md):
request limits and token-accurate prompt budgeting, a more resilient edge, model
integrity checks, stricter supply-chain gates, a working training pipeline, and the
documentation and community files of an open-source project. Merged in #13 to #16.
Added¶
- Prometheus histograms for admission queue wait and time to first token, and a counter for rejected prompts.
gguf.sha256inmodel-release.json. The llama.cpp engine verifies the downloaded GGUF against it on load and fails closed on a mismatch. The Kaggle pipeline records the hash for new releases.- Edge Worker: an in-isolate fallback rate limiter that applies when Upstash is not configured or is failing (previously rate limiting was off in that case).
- Edge Worker: a per-isolate circuit breaker over origins.
- Security workflow (
.github/workflows/security.yml):pip-auditon the serving lock,npm auditfor the Worker and the web UI, and dependency review on pull requests. It also runs weekly. - mypy configuration in
pyproject.toml; mypy runs in CI and inmake lint. examples/: the OpenAI SDK (blocking and streaming), the native/generateAPI with Server-Sent Events and sources, and curl for every endpoint.tests/test_examples.pyruns each one against a real uvicorn server;openaiis added to thedevextra for that test.- Documentation:
docs/audit/INITIAL_AUDIT.md(the baseline audit, with evidence for every finding),docs/audit/PERFORMANCE_BASELINE.md(measured single-request latency and two controlled concurrency probes),docs/architecture/(system, request lifecycle, deployment, security and model pipeline documents, and architecture decision records ADR-001 to ADR-008), anddocs/IMPLEMENTATION_STATUS.md. - Community and release files:
CONTRIBUTING.md,CODE_OF_CONDUCT.md(Contributor Covenant 2.1 by reference),SECURITY.md, this changelog,CITATION.cff,MODEL_CARD.md,docs/legal/LICENSE_AUDIT.md, issue and pull request templates,.editorconfigand.pre-commit-config.yaml. - Operations and security documentation:
docs/operations/(runbook, incident response, rollback, cost management, backup and recovery) anddocs/security/SECURITY_AUDIT.md(STRIDE threat model). - License notices in shipped artifacts:
THIRD_PARTY_NOTICES.md(llama.cpp, the FastAPI documentation index, the embedding model) is copied into the image withLICENSE; the web build writesTHIRD_PARTY_NOTICES.txtfor its npm dependencies; published model folders get an Apache-2.0 attribution section and aNOTICEfile. - The uptime workflow also calls the authenticated
/v1/modelsthrough the edge, so an origin-secret or key mismatch is detected, not only a dead origin. - Continuous delivery of the edge: with
CLOUDFLARE_API_TOKENandCLOUDFLARE_ACCOUNT_IDconfigured, the Release workflow deploys the Worker and runs the k6 smoke test after every Cloud Run deploy. - Train workflow: a
push_onlyinput creates the Kaggle kernel without running it, so theHF_TOKENsecret can be attached before the first training run. The Kaggle username is read from a repository variable, and release pull requests carry themodel-releaselabel. SUPPORT.md,.github/CODEOWNERS, and.github/release.yml(categories for generated release notes). The issue chooser links to private vulnerability reporting and to SUPPORT.md..env.exampledocuments every setting, including the 22 that were missing, among themGGUF_SHA256,MAX_REQUEST_BYTESandVALIDATION_FRACTION. A test fails when a new setting is added without an entry, and another loads the file to check that its values are valid.- README: CI, release, security and CodeQL badges, quick links, a screenshot of the live UI, a table showing which services the live deployment uses, and sections on contributing, security reporting, citation and acknowledgements.
- Tests for the hardening changes: Python 93 to 134 passing (plus 2 Postgres-only tests and 1 POSIX-only test), Worker 55 to 75.
Changed¶
max_tokensis clamped to the room left in the model's context window.- Edge Worker: returns 504 instead of 502 when the last origin attempt timed out.
- Edge Worker:
ORIGINSdefaults to empty, so an unconfigured Worker fails closed instead of forwarding to placeholder hosts. - Edge Worker: a non-JSON 429, 502, 503 or 504 from the origin's platform is turned
into the JSON error shape with a
Retry-Afterhint, so clients see one error contract. ruff checkandruff format --checkcoverexamples/in CI and in the Makefile.- Timeout budget: on Cloud Run, queue wait (60 s) plus generation (60 s) now stays
under the edge's
ORIGIN_TIMEOUT_MS(raised from 60 to 150 s), so the edge no longer returns 504 while the origin keeps generating an answer nobody will receive. - Kaggle gate 1 requires the fine-tune to beat the base model by at least 0.005
(
min_improvement); a tie no longer passes. - JSON logs carry a
severityfield, which Cloud Logging maps to the entry severity. - README: the deployed base-model status, measured throughput, a full error table, the
new settings (
GGUF_SHA256,LLAMACPP_N_CTX,MAX_REQUEST_BYTES), and a documentation index. The unsupported claim of "hundreds to a few thousand answers per day" is replaced by the measured capacity statement, in the README and indocs/DEPLOYMENT.md. - Corrected comments in
api.py(middleware order) anddeploy/gcp/bootstrap.sh, and stale lines indocs/ARCHITECTURE.md. - CI and Release Trivy gates fail on fixable HIGH as well as CRITICAL CVEs.
- Container image: the runtime stage applies Debian security updates, and pip, setuptools and wheel are removed from the final image.
.dockerignorekeepseval/, tooling and local entry points out of the build context;.gitignoreignores per-run eval reports; the README listsmain.py, the serving lock, the Docker files andgraphify-out/in the project structure.- Dependency floors raised by Dependabot:
transformers>=5.20.2,peft>=0.21.2,sentencepiece>=0.2.3,datasets>=5.2.0,pytest>=9.2.0,pytest-cov>=7.1.0, and the build requirementsetuptools>=84.2.1. Thetorchfloor was raised to 2.14.1 and then restored to 2.2.0 (see Fixed). Dependabot now raises minimum versions only when they stop allowing the latest release (versioning-strategy: increase-if-necessary). - Dependabot updates:
highlight.js11.12.0 (web UI);wrangler4.159.0,vitest5.1.0 and@cloudflare/workers-types5.20261110.1 (Worker tooling);grafana/alloyv1.21.3 (deploy/vm).
Fixed¶
- The web UI no longer describes the model as fine-tuned while the base model is serving; the health badge states which one is live.
- A prompt under the character limit but over the llama.cpp context window (2,048
tokens by default) returned HTTP 500 (or, when streaming, an in-band error after a
200). The prompt is now counted
with the loaded tokenizer before admission and rejected with 400
context_length_exceeded. - 5xx errors under
/v1are typedserver_errorinstead ofinvalid_request_error. - 29 mypy errors resolved.
kaggle/pipeline.pycloned withgit clone --branch <ref>, which rejects the commit SHA that the Train workflow passes, so the first training run would have failed at the clone. It now runsgit init, a shallowgit fetchof the ref, and a checkout ofFETCH_HEAD.- Training no longer reads the held-out eval file. Checkpoint selection used it as the
trainer's evaluation split (
load_best_model_at_endon its loss), which leaked the benchmark into model selection. A seeded validation slice of the training file (VALIDATION_FRACTION, default 0.1) is used instead. No fine-tune had been trained with the old behaviour. - The
eval.ymlerror for a changed eval set now says how to regenerate the baseline. - The package reported
__version__1.0.0 (and so did the OpenAPI document) whilepyproject.tomlsaid 2.0.0; both now read 2.1.0, and a test keeps them equal. - The first Kaggle training runs failed before training: a raised
torchfloor made the training install replace the image's CUDA torch (breaking its torchvision), and the image's oldtorchaomade peft refuse to apply LoRA. The pipeline now keeps the preinstalled torch, torchvision and torchaudio as pip constraints and removestorchao; the Kaggle CLI is installed with its undeclaredsixdependency. - The Release workflow's edge deploy failed when the optional Upstash secrets were absent; only configured secrets are uploaded to the Worker now.
- Streams report a mid-generation context overflow with fixed text, so no exception
text reaches a client (CodeQL
py/stack-trace-exposure).
Security¶
- Request bodies are capped at 64 KiB (413), at the origin before parsing and authentication, and at the edge with a streaming byte counter.
- API responses carry
X-Content-Type-Options: nosniff,X-Frame-Options: DENYandReferrer-Policy: no-referrer. The edge strips origin infrastructure headers and addsnosniffto every response. - The web UI is served with a strict Content-Security-Policy and related headers.
- Without a database, feedback logging records the rating and the comment length, not the user's comment text.
- Edge: when Upstash answers but the
INCRreply is an error, the request is counted by the in-isolate limiter instead of passing unlimited. API_KEYset to a placeholder (disabled,none,null,change-me) is rejected at startup instead of being read as "unset", which silently disabled authentication when no database was configured. An origin with no authentication at all now logs a warning at startup.
2.0.0 - 2026-11-13¶
The production scale-out, merged to main in pull request #1 (684a10d). This
version was set in pyproject.toml but not tagged in git.
Added¶
- Dataset grown from 32 train / 8 eval to 351 train / 108 eval examples. Eval rows
carry an
id, acategory, andmust_includeidentifiers; CI checks that the eval set does not leak into the training set. - An
Enginecontract with three backends: llama.cpp GGUF (production CPU serving), torch with a LoRA adapter, and remote OpenAI-compatible servers (vLLM, TGI). Models load once in the background, with deadlines and cancellation. - Admission control: bounded concurrency and a bounded queue; overload returns 429
with
Retry-After. - OpenAI-compatible
/v1/chat/completionsand/v1/models, Server-Sent Events streaming on both APIs,/v1/feedback, an origin lock (X-Origin-Secret), andPORTsupport. - Per-user hashed API keys, usage and feedback in Postgres (connection pool, SQL migrations), with an in-memory store for development and tests.
- Retrieval over the FastAPI documentation (local NumPy index or pgvector), opt-in per request.
- Prometheus metrics with a per-app registry; optional Sentry.
- Evaluation harness and release gate, with the recorded baseline in
eval/baseline.jsonandeval/baseline-rag.json. fastapi-assistant data generate: drafts training pairs from the FastAPI docs with Claude, for human review.model-release.json, which pins the deployed weights; merge and publish tooling.- A production image that builds llama.cpp portably, ships without torch, and installs
from a hash-pinned lock (
requirements/serve.txt). - A Cloudflare Worker edge: key checks, per-key and per-IP rate limits, a shared answer cache in Upstash, ordered origin failover, and the origin lock.
- Deployment targets: Cloud Run (scale to zero,
maxScalecost cap), a GCP bootstrap with Workload Identity Federation, an Oracle VM origin behind a Cloudflare Tunnel with Grafana Alloy, and an optional Hugging Face Space origin. - A Grafana dashboard, alert rules, and a k6 load test.
- A Kaggle GPU pipeline: train, gate against the base model, merge, convert to GGUF, publish to the Hugging Face Hub, and write the release manifest.
- Workflows: CI (Python 3.10 to 3.12, Postgres with pgvector, Worker, gitleaks, image boot and smoke test, Trivy), multi-architecture release with gated deploys, the eval gate on the production backend, Kaggle training, uptime checks, CodeQL, and Dependabot. Third-party actions are pinned to commit SHAs.
- A React and TypeScript web playground in
web/, served by the Worker as static assets. scripts/setup_secrets.sh, with a non-interactive mode, which generates and stores deployment secrets without printing them.docs/DEPLOYMENT.md, the deployment runbook.
Changed¶
- README and architecture document rewritten for the multi-origin system;
ARCHITECTURE.mdmoved todocs/, and the k6 test totests/load/. - The Release workflow copies GitHub secrets into Google Secret Manager, adding a
version only when a value changed.
deploy/gcp/bootstrap.shno longer generates or prints secrets. - Every secret pasted into the setup script is optional; a
disabledplaceholder is treated as unset, and a non-URLSENTRY_DSNleaves Sentry off.
Fixed¶
- Client errors return fixed messages instead of exception text in
/ready,/generate(JSON and SSE) and/v1errors (CodeQLpy/stack-trace-exposure). - Worker: trailing slashes are trimmed with a linear loop instead of a regular
expression (CodeQL
js/polynomial-redos). - The
anthropicSDK import is optional, so CI passes without thedatagenextra. - Training fixes for transformers 5.x.
1.0.0 - 2026-07-05¶
Tag v1.0.0 (commit 0c6094c). The GitHub release "v1.0.0: tested, containerized
FastAPI model service" was published on 2026-11-13.
Added¶
- An installable
src-layout package,fastapi_assistant, with settings from environment variables (pydantic-settings), structured JSON logging, a thread-safe engine, a FastAPI app factory, schemas, and a CLI. - HTTP API:
/health,/ready, and/generate, with request-id and timing middleware, optionalX-API-Keyauthentication, CORS, and bounded generation parameters. - A pytest suite that uses a fake engine (no model or network), ruff, GitHub Actions
CI on Python 3.10 and 3.11, a non-root Dockerfile with a health check, a Makefile,
the MIT license, and
ARCHITECTURE.md. - Dataset grown from 5 train / 2 eval to 32 train / 8 eval question-answer pairs.
- A compatibility shim for the
torch_dtypetodtyperename in transformers 5.x. .gitattributesthat normalises text files to LF; thegraphify-out/knowledge graph outputs are tracked.
Fixed¶
- Inference loaded the adapter-only directory with
AutoModelForCausalLM. It now loads the base model and applies the LoRA adapter withpeft. - Training and inference share the tokenizer's native chat template (
prompts.py). - Training masks the loss to answer tokens (completion-only loss) and pads dynamically.
Removed¶
- Training outputs (the LoRA adapter and checkpoint in
outputs/) are no longer tracked in git; the adapter is regenerated withmake train.
Prototype (untagged) - 2025-11-17¶
The original project, before any version number: scripts that fine-tuned
TinyLlama-1.1B-Chat with LoRA (transformers and peft) on 5 training and 2 eval
question-answer pairs, an interactive CLI (main.py), and the resulting adapter
committed under outputs/ (a 2-step training run). A Gradio app.py for Hugging Face
hosting was added and removed the same day.