Skip to content

FastAPI Niche Assistant

The FastAPI Niche Assistant is a small language model specialised in the FastAPI web framework, served as an OpenAI-compatible API with streaming, retrieval over the official documentation, per-user API keys, and an edge layer for rate limiting, caching and origin failover. It runs on free-tier infrastructure, and its release pipeline ships a new model only if it beats the current one on a held-out evaluation.

Try the live demo Source on GitHub

Status

The live service runs v2026.11.14-72d6b37, a LoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct quantized to GGUF. It scores 0.666 on the held-out set against 0.546 for the previous release, a TinyLlama-1.1B fine-tune. It is a small model and often wrong, so check its answers. The model card has the evaluation and the implementation status page has what is live.

Start here

  • Quickstart


    Install the package, fetch the released model and serve the API on your own machine in a few commands.

    Quickstart on GitHub

  • Architecture


    How a request travels from the edge Worker to the model, what runs where, and the decisions behind the design.

    Architecture overview

  • Operations runbook


    Health checks, logs, metrics and the routine tasks: rotating secrets, deploying code and model releases.

    Runbook

  • Security


    How to report a vulnerability privately, which versions receive fixes, and the risks that are accepted.

    Security policy

  • Model card


    The served model, its training data, evaluation results, intended use and known limitations.

    Model card

  • Contributing


    Development setup, the checks every pull request must pass, and the rules that protect the evaluation.

    Contributing guide

The web UI

The web UI welcome screen: the fine-tuned model is ready, the public demo needs no key, with example questions and the settings panel
Ask a FastAPI question and get a streamed answer; with documentation retrieval on, it cites the passages it was grounded on. The badge in the header shows which model is serving.