Moosa Memon
← Index of work

A fine-tune that had to beat an honest baseline first

ForgeTune: fine-tunes a small open model to turn unstructured PSX company announcements into schema-valid JSON, on a hand-corrected dataset, evaluated against a prompted baseline instead of a straw man.

Type
ML Systems
Stack
LoRA / Small open models / Pydantic / Evaluation / Docker
Status
Complete: train, align, eval and serve stages

Problem

The model is the easy part. What decides whether a fine-tuning project means anything is three things: a JSON contract worth training against, a dataset that has been hand-corrected rather than copied from a teacher model, and an eval that compares the fine-tune against an honest baseline. Most fine-tuning portfolio projects fail all three.

System

Stock-exchange announcements (dividends, board meetings, financial results) come in as messy prose. A strict Pydantic schema defines the target. The training set was generated, then hand-corrected, because a model trained on a teacher’s mistakes learns them faithfully. The eval scores schema validity and field accuracy for the fine-tuned model and for a well-prompted larger model on the same held-out set.

Worth knowing

Why fine-tune this and not just prompt it is the first question anyone asks, and the project answers it directly. Fine-tuning earns its keep for rigid output-format consistency at scale, lower latency and per-call cost on a narrow high-volume task, domain conventions that are learned from examples more reliably than described, and on-prem deployment. It does not fix knowledge gaps; that’s what RAG is for. A model that doesn’t know a fact won’t learn it from a LoRA adapter trained on 800 extractions.

That framing is also how I’d scope it for a client: fine-tuning is a cost and consistency decision, not an intelligence upgrade.

Have a workflow that looks like this?

Most of these start as a messy, manual process someone got tired of. Twenty minutes is usually enough to sketch how I'd approach yours.