Model pipeline¶
How training data becomes deployed weights, which gates a candidate must pass, and how a model release rolls out and rolls back.
1. Current state¶
The deployed model is v2026.11.14-72d6b37, a LoRA fine-tune of
Qwen2.5-Coder-1.5B-Instruct, live since 2026-11-15. This pipeline produced it end to
end: Train run 43622001406 trained it from commit 72d6b37 (#44) on a free Kaggle T4
GPU, gated, merged, quantized and published it to the Hub, and it was re-scored in CI
and deployed by Release run 43696171917 after release pull request #53 was merged.
model-release.json has "finetuned": true and points at
syed-waleed-ahmed/fastapi-assistant-gguf at that tag, with its SHA-256.
Both gates, each scoring both models in the same run:
| Metric | Gate 2 (CI, llama.cpp): v2026.11.14-72d6b37 |
Gate 2: previous release v2026.11.13-2b04940 |
Gate 1 (Kaggle, fp16): fine-tune | Gate 1: base Qwen2.5-Coder-1.5B-Instruct |
|---|---|---|---|---|
| Items | 108 | 108 | 108 | 108 |
| Composite score | 0.6663 | 0.5463 | 0.6716 | 0.5576 |
| Keyword recall | 0.6852 | 0.4259 | 0.7114 | 0.6219 |
| Code validity | 0.8846 | 0.8462 | 0.8558 | 0.7212 |
| ROUGE-L | 0.4488 | 0.3866 | 0.4548 | 0.3450 |
The fine-tune's Gate 2 scores in eval/baseline.json are now the bar the next release
must clear.
The previous release, v2026.11.13-2b04940, was the first fine-tuned release: a LoRA
fine-tune of TinyLlama-1.1B-Chat, deployed by #33, which scored 0.5463 against the
base TinyLlama's 0.4023 in Gate 2. Before it, base-tinyllama-q4km served the base
TinyLlama (audit finding F-04). The first training runs exposed environment problems
in the Kaggle image, fixed in #15 and #16.
2. Pipeline¶
flowchart LR
Data[("data/*.jsonl<br/>502 train, 108 eval")] --> Validate["data validate<br/>in CI and on Kaggle"]
Validate --> Train["LoRA fine-tune<br/>Kaggle GPU kernel"]
Train --> G1{"Gate 1<br/>fine-tune scores at least<br/>the base model?"}
G1 -->|no| Stop1["stop<br/>nothing published"]
G1 -->|yes| MergeStep["merge adapter<br/>into base, fp16"]
MergeStep --> Export["convert to GGUF<br/>quantize Q4_K_M<br/>record SHA-256"]
Export --> Hub[("Hugging Face Hub<br/>adapter and GGUF repos<br/>immutable version tag")]
Hub --> G2{"Gate 2, eval.yml<br/>llama.cpp, greedy<br/>regression at most 0.02"}
G2 -->|no| Stop2["no release PR"]
G2 -->|yes| PR["release PR<br/>model-release.json<br/>eval/baseline.json"]
PR --> Main["merge to main"]
Main --> Deploy["release.yml<br/>new Cloud Run revision"]
| Stage | Where | Code |
|---|---|---|
| Validate | CI on every push, and first on Kaggle | fastapi-assistant data validate, datasets_tools.py |
| Train | Kaggle GPU kernel, started by train.yml |
kaggle/pipeline.py, training.py |
| Gate 1 | Kaggle | fastapi-assistant eval --baseline, evaluation.py gate |
| Merge | Kaggle | fastapi-assistant export merge, export.py merge_adapter |
| GGUF export | Kaggle | scripts/export_gguf.sh |
| Publish | Kaggle to Hugging Face | fastapi-assistant export push, export.py push_folder |
| Gate 2 | GitHub Actions | .github/workflows/eval.yml, called by train.yml |
| Release PR | GitHub Actions | train.yml pull-request job |
| Rollout | GitHub Actions to Cloud Run | .github/workflows/release.yml |
3. Data¶
| File | Rows | Fields |
|---|---|---|
data/fastapi_qa_train.jsonl |
502 | instruction, input, output |
data/fastapi_qa_eval.jsonl |
108 | id, category, instruction, input, output, must_include (API identifiers a correct answer must name) |
fastapi-assistant data validate (datasets_tools.py) fails on:
- schema violations, empty fields, or an output over 2,000 characters;
- a fenced Python block that does not parse;
- an eval row without an
id, with a duplicateid, or with amust_includeentry missing from its own reference answer; - exact duplicates within a split, or an eval question that also appears in train;
- near-duplicate leakage between eval and train: questions compared by content words (Jaccard at least 0.7 by default), answers by 5-word shingles (Jaccard at least 0.6).
Near-duplicate questions inside the training set are reported as warnings only.
More pairs can be drafted from the FastAPI documentation with
fastapi-assistant data generate (datagen.py, Claude), then reviewed by hand and
merged with data merge.
4. Training on Kaggle¶
train.yml (manual dispatch, inputs base_model, num_epochs, learning_rate and
push_only) renders kaggle/pipeline.py with those inputs, the commit and
HF_NAMESPACE, and pushes it as a private GPU kernel named
<KAGGLE_USERNAME>/fastapi-assistant-train (kaggle/kernel-metadata.json). It then
polls the kernel for up to 300 minutes and downloads its outputs. With
push_only: true it creates or updates the kernel without running it, which is the
first-time setup step (see the credentials below).
On the kernel, the pipeline installs the train extra with the GPU image's own CUDA
torch, torchvision and torchaudio pinned as pip constraints, so the install
cannot replace them, and then uninstalls the image's torchao, which is older than
peft accepts (#16).
| Setting | Value | Source |
|---|---|---|
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct (until #44: TinyLlama/TinyLlama-1.1B-Chat-v1.0) |
kaggle/pipeline.py CONFIG |
| LoRA | r 16, alpha 32, dropout 0.05, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
kaggle/pipeline.py, config.py |
| Optimisation | 3 epochs, learning rate 2e-4, batch 4, gradient accumulation 4 (effective batch 16, fits a T4 in fp16) | kaggle/pipeline.py |
| Sequence length | 512 tokens | config.py max_seq_length |
| Loss | Completion-only: prompt tokens are masked with label -100 | training.py, data.py |
| Prompt format | prompts.build_messages and encode_chat, the same format serving uses |
data.py, prompts.py |
| Version label | v<YYYY.MM.DD>-<short commit> |
kaggle/pipeline.py |
Credentials needed (all configured for this repository):
- the
KAGGLE_API_TOKENGitHub secret (or the legacyKAGGLE_KEY); - the Kaggle username as the
KAGGLE_USERNAMEGitHub variable (a secret of that name is also accepted); HF_NAMESPACEas a GitHub variable;HF_TOKENattached to the kernel as a Kaggle secret. On first setup, runtrain.ymlonce withpush_only: true, attach the secret to the new kernel on Kaggle, then run the workflow normally.
Without HF_TOKEN or a namespace, the kernel trains and evaluates but publishes
nothing, and train.yml stops.
5. Gate 1: beat the base model¶
On the GPU, the pipeline scores the base model (eval --no-adapter) and the
fine-tune on the 108-item eval set, then gates the fine-tune against the base
report with --max-regression set to minus min_improvement (default 0.005). The
fine-tune must therefore beat the base model by at least that margin; greedy decoding
makes the eval deterministic, so a tie is not an improvement. Otherwise the kernel
exits before merging or publishing.
Training never reads the eval file. data.load_raw_dataset splits a seeded
validation slice (VALIDATION_FRACTION, default 0.1) off the training file, and the
trainer selects its best checkpoint by loss on that slice.
6. Merge, GGUF export and quantisation¶
export mergeloads the base model in fp16, applies the adapter, and callsmerge_and_unload, writing a full model plus tokenizer.scripts/export_gguf.shclones llama.cpp at a pinned ref (b11971), converts the merged model to an f16 GGUF withconvert_hf_to_gguf.py, buildsllama-quantize(GGML_NATIVE=OFF), and quantizes toQ4_K_M(about 1 GB for the 1.5B-parameter Qwen model; 0.7 GB for the earlier 1.1B TinyLlama). The converter embeds the tokenizer's chat template (ChatML for Qwen), so serving uses the training prompt format without extra configuration.- The pipeline computes the SHA-256 of
model-Q4_K_M.gguf(hub.sha256_file) and writes it into the release manifest.
7. Publishing to Hugging Face¶
export push uploads the adapter to <HF_NAMESPACE>/fastapi-assistant-lora and the
GGUF to <HF_NAMESPACE>/fastapi-assistant-gguf, with a generated model card
(write_model_card: base model, LoRA settings, eval summary). Each upload commit is
tagged with the version label. The tag is created with exist_ok=False, so the
tooling fails rather than move an existing tag. The manifest pins
revision to that tag.
A candidate is published before Gate 2. One that fails Gate 2 stays on the Hub under
its tag but is never named by model-release.json, so it is never deployed.
8. Gate 2: production backend eval¶
eval.yml scores what will actually be served: the GGUF on llama.cpp, greedy
decoding, 256 new tokens, on the eval set.
| Case | Reference | Rule |
|---|---|---|
Called from train.yml with the candidate manifest |
The model in main's model-release.json, re-scored on the same runner and eval set |
Candidate composite must not be more than MAX_REGRESSION (0.02) below the reference |
PR that changes model-release.json |
Same as above | Same |
| PR that changes only prompts, engines, evaluation code or the eval set | The committed eval/baseline.json |
Same; a PR that changes the eval set without changing the model must also commit a new baseline |
evaluation.py defines the metrics: keyword recall over must_include, code
validity (every Python block parses), and ROUGE-L against the reference. The
composite is their mean over the items where each applies.
9. Release manifest¶
model-release.json is the only source of truth for deployed weights. The release
PR replaces it together with eval/baseline.json and eval/baseline.md.
| Field | Current value | Becomes |
|---|---|---|
model_version |
v2026.11.14-72d6b37 |
MODEL_VERSION (responses, metrics, usage), Cloud Run revision label, edge cache namespace |
base_model |
Qwen/Qwen2.5-Coder-1.5B-Instruct |
BASE_MODEL_NAME |
finetuned |
true |
FINETUNED_WEIGHTS, reported as uses_adapter by /health |
chat_format |
absent (the base release set zephyr) |
LLAMACPP_CHAT_FORMAT; absent for a fine-tune, which uses the GGUF's embedded template |
gguf.repo_id, gguf.filename, gguf.revision |
syed-waleed-ahmed/fastapi-assistant-gguf, model-Q4_K_M.gguf, the release tag |
GGUF_REPO_ID, GGUF_FILENAME, GGUF_REVISION |
gguf.sha256 |
64 hex characters | GGUF_SHA256, verified before load |
adapter |
syed-waleed-ahmed/fastapi-assistant-lora at the release tag |
ADAPTER_REPO_ID, ADAPTER_REVISION (transformers backend only) |
eval |
the Kaggle gate's summary (score 0.6716) | Eval summary of the release |
The translation is export.manifest_env, exposed as
fastapi-assistant export env model-release.json.
10. Rollout¶
- Merging the release PR pushes
model-release.jsontomain, which triggersrelease.yml. deploy-cloudrunrendersdeploy/cloudrun/service.yamlfrom the manifest and deploys a new revision labelled with the model version.- The new instance downloads the GGUF at the pinned revision, verifies its
SHA-256, and loads it. The startup probe holds traffic until
/readyanswers 200 (up to 240 s). If the hash does not match, the load fails and the revision never becomes ready. deploy-edgesets the Worker'sMODEL_VERSIONto the new version, which moves the shared answer cache to a new namespace so answers from the previous model are never served.
11. Rollback¶
Revert the release PR. Release redeploys with the previous manifest, and the
previous weights are still on the Hub under their tag, so nothing is retrained or
re-exported. The previous release's eval/baseline.json returns with the revert,
so the gate compares future candidates against the restored model.
12. Known gaps¶
- Two fine-tuned releases exist. The first did not improve keyword recall on its base model; the Qwen release raised it to 0.6852, but its answers are still often wrong, and retrieval has not been measured with either (ROADMAP.md).
- Fixed during this review.
train.ymlpasses the commit SHA asref, butkaggle/pipeline.pyused to clone withgit clone --branch <ref>, which rejects a commit hash. The pipeline now runsgit init,git fetch --depth 1 origin <ref>andgit checkout FETCH_HEAD. That form was checked to work against GitHub with both a commit SHA and a branch name. - Fixed after the first runs. The Kaggle CLI 2.2.5 imports
sixwithout declaring it, sotrain.ymlinstalls it (#15). After a Dependabot floor bump, installing thetrainextra replaced the image's CUDAtorchand broke itstorchvision; then the image'storchaomadepeftrefuse to apply LoRA. The pipeline now pins the preinstalledtorch,torchvisionandtorchaudio, uninstallstorchao, and thetorchfloor is back to>=2.2.0(#16). The rest of the pipeline is untested end to end on Kaggle until a run completes.