Contributing¶
Thank you for helping improve the FastAPI Niche Assistant. This guide explains how to set up a development environment, which checks a change must pass, the rules that protect the evaluation, and how to propose a new model release.
By taking part you agree to follow the Code of Conduct. Report security vulnerabilities privately as described in SECURITY.md, never in a public issue.
Before you start¶
- For anything larger than a small fix, open an issue first so the approach can be agreed before you write code.
- Read docs/ARCHITECTURE.md for the design and the README for the project layout.
- Keep each pull request to one logical change.
Development setup¶
The quickest route is the dev container: open the repository in
GitHub Codespaces,
or locally with the VS Code Dev Containers extension. It installs Python 3.11 with
the CPU-only torch build, Node.js 22, every dependency the test suites need and the
pre-commit hooks (see .devcontainer/post-create.sh). Then skip to Checks.
To set up your own machine instead, you need:
- Python 3.10, 3.11 or 3.12 (CI tests all three; the production image uses 3.11);
- Node.js 22 or later, for the edge Worker and the web UI;
- Git; Docker is optional (
docker compose up --buildruns the service with Postgres and pgvector).
git clone https://github.com/syed-waleed-ahmed/Niche-Finetuned-Model.git
cd Niche-Finetuned-Model
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
# Optional but recommended without a GPU: the CPU-only torch build is much smaller
# than the default CUDA build. CI installs it this way.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev,train,postgres]"
The extras in pyproject.toml:
| Extra | Installs | Needed for |
|---|---|---|
dev |
pytest, pytest-cov, ruff, mypy, openai | tests and checks (the OpenAI SDK runs examples/openai_sdk.py in the tests) |
train |
torch extra plus datasets |
fine-tuning |
torch |
torch, transformers, peft, accelerate, sentencepiece, protobuf | the transformers backend, merging, export |
postgres |
psycopg and its pool | per-user keys, usage, pgvector |
llamacpp |
llama-cpp-python | the production GGUF backend (needs a C/C++ toolchain if no wheel is available) |
rag |
fastembed | retrieval over the FastAPI docs |
sentry |
sentry-sdk | error reporting |
datagen |
anthropic | drafting training data with Claude |
serve |
llamacpp, postgres, rag, sentry |
everything the production image runs |
The test suite needs no model, network access, or database. The Makefile wraps the
common commands (make test runs pytest; make lint runs ruff and mypy); on Windows
without make, run the commands below directly.
Checks¶
Every pull request must pass the same checks CI runs. Run them locally first:
# Python
pytest # unit, API, and examples tests
ruff check src tests examples # lint
ruff format --check src tests examples # formatting (fix with: ruff format src tests examples)
mypy # type check; configured in pyproject.toml [tool.mypy]
fastapi-assistant data validate # dataset schema, code, duplicates, leakage
# Edge Worker
cd deploy/cloudflare-worker && npm ci && npm run typecheck && npm test && cd ../..
# Web UI
cd web && npm ci && npm run typecheck && npm test && npm run build && cd ..
Optional suites:
TEST_DATABASE_URL=postgresql://... pytest -m postgresruns the Postgres and pgvector integration tests (CI runs them againstpgvector/pgvector:pg16).pytest -m slowdownloads a real model.
CI also builds the container image, boots it, answers a real question, scans the
image with Trivy, scans the full git history with gitleaks, and runs CodeQL. The
Security workflow runs pip-audit on requirements/serve.txt, npm audit for the
Worker and the web UI, and GitHub dependency review.
Pre-commit hooks¶
The repository ships a pre-commit configuration with ruff (lint and format), basic file hygiene checks, and gitleaks:
pip install pre-commit
pre-commit install # run the hooks on every commit
pre-commit run --all-files # run them once over the whole tree
The first run takes longer: pre-commit builds the hook environments, including
gitleaks. The whitespace fixers deliberately skip data/*.jsonl, eval/ and
graphify-out/.
Dependencies¶
- Python dependency ranges live in
pyproject.toml. The production image installs fromrequirements/serve.txt, a hash-pinned lock. If you change theserveextras (or anything they pull in), regenerate the lock withmake lock(needs uv) and commit it in the same pull request. - npm projects (
web/,deploy/cloudflare-worker/) commit theirpackage-lock.json; install withnpm ci. - Dependabot opens update pull requests weekly; prefer those to manual bumps.
- New dependencies need a reason in the pull request and a license compatible with MIT distribution. Record new third-party components in docs/legal/LICENSE_AUDIT.md.
Changing the dataset¶
Training rows (data/fastapi_qa_train.jsonl) have instruction, input and
output. Eval rows (data/fastapi_qa_eval.jsonl) also have id, category and
must_include. Answers follow one style: a one or two sentence lead, one minimal
runnable example in a python fence, then at most two short sentences; at most 1,400
characters; current idioms (Pydantic v2, Annotated[...] parameters, lifespan
instead of on_event).
To add training data:
- Write rows by hand, or draft them with Claude from the official docs:
pip install -e ".[datagen]", thenfastapi-assistant data generate --out drafts.jsonl --pages 20. This needs your own Anthropic credentials and makes one API request per documentation section. - Review every drafted row yourself. Fix or delete anything that is wrong, unclear, or not supported by the documentation.
- Merge reviewed rows with
fastapi-assistant data merge drafts.jsonl --split train. - Run
fastapi-assistant data validateandfastapi-assistant data stats. - In the pull request, say how the rows were produced (hand-written, or drafted and reviewed) and what you checked.
Only add content you have the right to contribute. The FastAPI documentation is MIT-licensed; do not paste text from sources with other terms.
Evaluation integrity¶
The held-out eval set is the release gate: a model ships only if it scores well on it. A gate is only meaningful if nobody optimises against it. These rules are not negotiable:
- Never edit
data/fastapi_qa_eval.jsonlto improve a score. Do not rewrite, remove, or relabel items, references, ormust_includeidentifiers because a model gets them wrong. A genuine error in an eval item (for example a wrong reference answer) is fixed in a separate pull request that changes nothing else, explains the error with evidence, and records the new baseline (see rule 2). - Never edit the eval baselines by hand.
eval/baseline.json,eval/baseline.md,eval/baseline-rag.jsonandeval/baseline-rag.mdare written only byfastapi-assistant eval. The training workflow's release pull request replaceseval/baseline.jsonandeval/baseline.mdwith the report produced by the eval workflow. When the eval set changes, the eval workflow fails unlesseval/baseline.jsonchanges in the same pull request; regenerate it withfastapi-assistant evalon the production backend (load the settings withfastapi-assistant export env model-release.json), never by hand. - Never train or tune on the eval set. Do not add eval items, or paraphrases of
them, to the training data, and do not choose prompts, hyperparameters, or data by
repeatedly scoring against the eval set. The leakage check in
fastapi-assistant data validatecatches near-duplicates, not paraphrases, so this depends on you. Training itself never reads the eval file: checkpoint selection uses a seeded validation slice of the training file (VALIDATION_FRACTION, default 0.1; seesrc/fastapi_assistant/data.py). - Report results honestly. Paste the eval report produced by the tool. Do not round in your favour, drop categories, or quote a subset of items.
Changes to src/fastapi_assistant/prompts.py, src/fastapi_assistant/engines/ or
src/fastapi_assistant/evaluation.py trigger the eval workflow, which scores the
model against eval/baseline.json and fails on a regression of more than 0.02 in
the composite score.
Proposing a model release¶
The deployed model is whatever model-release.json pins. A release changes that file
and nothing about the eval set.
Maintainers run the "Train on Kaggle" workflow (.github/workflows/train.yml).
It needs the KAGGLE_API_TOKEN secret (or the legacy KAGGLE_KEY), the Kaggle
username as the KAGGLE_USERNAME variable (a secret is also accepted), the
HF_NAMESPACE variable, and an HF_TOKEN secret attached to the Kaggle kernel. On
first setup, run the workflow once with push_only: true to create the kernel, attach
HF_TOKEN to it on Kaggle, then run the workflow normally. It trains on a free
Kaggle GPU, requires the fine-tune to beat the base model's composite score by at
least 0.005 (min_improvement), merges and quantizes it, publishes the adapter and the GGUF to the
Hugging Face Hub under a new tag, scores the GGUF with the production backend, and
opens a pull request that updates model-release.json and eval/baseline.json.
Review the eval table in that pull request; merging it deploys the model, and
reverting it rolls back.
Contributors without access to the repository secrets:
- Open an issue describing the proposed change (data, hyperparameters, or base model) and the expected effect.
- Run
kaggle/pipeline.pyin your own Kaggle account, withrepoandrefin itsCONFIGblock pointing at your fork and branch,hf_namespaceset to your Hugging Face namespace, and your ownHF_TOKENattached as a Kaggle secret. It writesmodel-release.json,eval-base.jsonandeval-finetuned.json. - Make sure the published weights are public, carry the Apache-2.0 license of the base
model with attribution, and are pinned to the release tag with the GGUF's SHA-256
in
model-release.json. - Open a pull request that changes only
model-release.json, and paste both eval reports into the description. The eval workflow scores your model and the current one on the same eval set with the production backend and uploads the report as a workflow artifact. - Do not edit
eval/baseline.json: if the release is accepted, the maintainer replaces it with the report produced by the eval workflow.
A release must pass both gates, use the shared prompt format in prompts.py, and
come with an updated MODEL_CARD.md.
Commits and pull requests¶
- Branch from
mainwith a short descriptive name, for examplefeat/...,fix/...ordocs/..., and targetmain. - Write commit subjects in the imperative mood and sentence case, about 72 characters at most, with no trailing period (for example "Add a non-interactive mode to the secrets script"). Use the body to explain what changed and why.
- Fill in the pull request template. Link the issue (
Fixes #123). - Update the documentation the change affects (README,
docs/,.env.example, the model card) and add a line to theUnreleasedsection of CHANGELOG.md. - Preview the docs site:
pip install -r requirements/docs.txt && mkdocs serve. - Never commit secrets, API keys,
.envfiles, model weights, or generatedoutputs/. CI scans the full history with gitleaks; a leaked secret must be rotated, not only deleted. - All CI checks must pass before review.
License¶
By contributing, you agree that your contributions are licensed under the MIT License that covers this project.