In-House LLM Platform (Altong)
Built the in-house LLM stack for a Korean Q&A platform: a fine-tuned Qwen3-8B classifier that beats gpt-4o 73.1% to 44.0% on the same 2,821-example holdout at p95 678ms and $0 per call, plus a streaming KO/EN/RU/UZ chat product (FastAPI + SSE, Next.js client) running entirely on company-owned hardware.
The problem
The company was paying per token to external APIs for background AI work (moderation, tagging, translation drafts, summaries, category suggestion) on a Korean-primary platform with EN/RU/UZ users. Nobody had measured whether a paid frontier model was actually good at the platform's own labelling conventions, and nobody had measured what the calls cost. There was no GPU budget approved — any argument for one had to be made with numbers produced on hardware already owned.
The solution
Two tracks. First, a chat product: a FastAPI backend streaming SSE with tool calling (web search, Wikipedia, image generation and editing, escalate-to-human), per-browser conversation and user-profile memory, effort routing that escalates hard questions to a stronger model, a live monitoring dashboard, Postgres persistence under Alembic, and a Next.js 16 client with four-language i18n, markdown rendering, image attachments and a conversation sidebar. Second, the local-model program: an Ollama-served MoE model behind a role registry so the model name lives in exactly one file, a job worker under launchd, RAG over the platform's own archive, and a LoRA fine-tuning pipeline on MLX with dataset curation, holdout evaluation, abstention calibration and latency benchmarking. Every claim was measured before it was reported — including the ones that came out negative.
Overview
Two repositories on this machine plus a third at `~/dev/ALTONG_LLM` form one product line, in two distinct phases. Phase one (2026-07-21, 60 commits across two repos in a single day) is a customer-facing chat assistant: `ALTONG_LLM`, a 3,644-line FastAPI backend streaming SSE with typed events, native tool calling (Tavily web search, Wikipedia, OpenAI image generation and editing, escalate-to-human), language detection that answers in the language written rather than the language selected, effort routing with model self-escalation, browser-keyed conversation and user-profile memory on Postgres with Alembic migrations, and a `/v1/monitor` dashboard — all under `mypy --strict`, ruff and a unit + integration pytest suite. `LLM-frontend` is its client: Next.js 16 Pages Router, React 19, strict TypeScript, SCSS Modules, 4,312 lines across a Screen → Section → Component structure, with a four-language KO/EN/RU/UZ message catalogue, a hand-written SSE parser (EventSource can't POST), streaming elapsed-time and phase indicators, image attach/lightbox/download, and a same-origin proxy so the backend URL never reaches the browser.
Phase two is `LLM_MODEL` — 114 commits between 2026-07-29 and 2026-08-21, and the part that carries the case study. It starts with an office PC running Qwen3.6-35B-A3B on CPU behind an outbound-only polling worker (no inbound ports opened), then deliberately retires that machine and moves the whole stack to an M3 Pro MacBook: the same fixed 16-case eval scores 16/16 at 3.1s per case on the Mac against 9.5–12s on the PC. The architecture rule is that nothing above the model layer knows which model it is talking to — prompts are text files, tools sit behind one dispatcher, and `models.py` is the only file that names a model, so swapping models costs an afternoon. The move surfaced three defects that had been invisible in production, including a macOS trust-store failure that made every web search fail silently while the model answered from 2023 memory in the same confident voice it uses for sourced answers.
The headline result is a category classifier. A LoRA fine-tune of Qwen3-8B-4bit was measured against gpt-4o on an identical 2,821-example holdout, using the prompt copied character-for-character out of the company's Java backend and category IDs verified against the production database: 73.1% balanced accuracy against 44.0%, at p50 533ms / p95 678ms against a 1,400ms internal budget, for $0 versus a measured $0.672 per thousand calls. Raw model output was still worse than the do-nothing baseline on real traffic weighting (42.0% against 68.3%, because 68.3% of live questions carry one catch-all label), so a top-2 gap policy tuned on one half and measured on the other lifted it to 74.6% — +6.3pp with a bootstrap 5th-percentile floor of +4.5pp, shipping through the existing single-`categoryId` contract with no frontend change. An earlier 32B adapter scored higher on accuracy but was rejected on latency (p95 2,694ms) — accuracy alone was never the bar.
The discipline is more interesting than the win. A parallel translation head-to-head — 250 examples, five directions, three blind LLM judges with rotated slots over two passes — was reported as a loss: 18.3% win rate against gpt-4o-mini, absolute quality 2.91 versus 3.99, with the root cause traced to the training data (39.2% of target rows carried truncated text or HTML residue, because the dataset was built from a defective production cache) rather than to the checkpoint, which was explicitly tested and cleared. Three earlier chrF numbers were retracted as circular, since the reference was itself the paid model's output. Along the way, a usage-metering audit found `gpt-4o-mini` missing from the company's model catalogue: four features configured for mini were silently falling through to gpt-4o at 16.7x the price, with no warning logged. The fix was written, compiled, verified, then reverted — reported as a document for the owning team rather than merged into someone else's branch.
Key features
- Fine-tuned Qwen3-8B category classifier (LoRA on MLX) beating gpt-4o 73.1% to 44.0% on an identical 2,821-example holdout with the production prompt and DB-verified label mapping
- Decision-policy layer (top-2 gap gate, tuned and tested on disjoint halves) lifting real-traffic accuracy to 74.6% from a 68.3% do-nothing baseline, with no frontend contract change
- Production classify service: hard 1,200ms deadline, request queue, health endpoint with rolling latency window, shadow mode, and a warm-always requirement documented from a measured 45s cold start
- Streaming FastAPI chat backend: SSE with typed events, native tool calling (web search, Wikipedia, image generation/editing, escalate), effort routing with model self-escalation, per-browser conversation and profile memory
- Next.js 16 chat client: hand-written SSE parser, four-language KO/EN/RU/UZ catalogue, markdown rendering, image attach/lightbox/download, conversation sidebar, theme resolved before paint
- Model-agnostic worker architecture — prompts as files, tools behind one dispatcher, model named in exactly one module — with a fixed 16-case eval and per-machine baselines for swap decisions
- Usage metering with PII whitelisting and per-call cost attribution, which surfaced a silent 16.7x overbilling in the company's model catalogue
- LLM-as-judge evaluation harness: three judges, blind task files, cyclic slot rotation, per-judge stability measured (Opus 249/250, Haiku 68%)
- Overnight and weekend autonomous training runners with thermal and thrash guards, resume-from-checkpoint, live dashboard, and a morning cleanup watchdog