I sent Crabby a zip this week. Nine hundred and twenty-five training examples, forty-nine for validation, two folders named train and val, and a quiet ambition: teach a small model how I actually talk so I can stop repeating myself in long prompts.
The whole thing started as an experiment in a side conversation. I had been noticing that every time I gave a longer brief, the assistant would either over-explain the obvious parts or skip the part I actually cared about. Not a model-quality problem. A me problem. My prompts are inconsistent. Some days I write terse. Other days I over-write. The model has no idea which Tun it's getting.
Fine-tuning a tiny base model on a curated set of my own exchanges fixes that, in theory, by baking the variation into weights instead of asking for it in every prompt. The piece I keep coming back to, though, is not the model. It is the dataset. Everything else in the pipeline is a parameter. The dataset is the product.
What I actually put in the zip
I won't share the raw files — those are private conversations — but the shape is worth describing. Each example is a JSONL line with a messages array: a system prompt that establishes voice and a few hard rules, one user turn that mirrors the kind of thing I would actually say, and one assistant turn I rewrote by hand to match the response style I want. The validation split is forty-nine lines from a different week than the training set, so the model can't memorize the test.
Numbers I landed on, roughly:
- 925 training examples, 49 validation examples
- Roughly a 95/5 split, which is on the small end but adequate for a nano fine-tune
- Average prompt length around 280 tokens; average completion around 410
- Total training tokens somewhere just north of 600k after tokenization
That last number matters because OpenAI prices fine-tuning by training tokens, and gpt-4.1-nano is the cheap option. The math worked out to a single-digit-dollar training run, which is the right price for an experiment that might not survive contact with reality.
Why nano, and why now
GPT-4.1 nano is the smallest model in OpenAI's GPT-4.1 family. The numbers that caught my eye: a 1M-token context window, MMLU around 80.1%, GPQA around 50.3%, and the latency is the lowest of the GPT-4.1 line. The fine-tuning API for it went generally available to all paid tiers back in May, and DPO on the same family followed in early June.
For a voice fine-tune on a personal agent, that profile is exactly right. I am not trying to teach the model new facts. I am trying to teach it which facts to lead with, which metaphors to reach for, which phrasings to avoid. That is a style problem, and small models fine-tune well on style problems — there is genuinely less knowledge to overwrite.
Two timing wrinkles forced my hand, though. OpenAI announced in early May that the fine-tuning platform is winding down — existing customers can still run training jobs through January 6, 2027, but no new sign-ups, and the recommended replacement for older fine-tunes is the GPT-5 family, which is not currently fine-tunable. The nano model I'm using has a scheduled shutdown of October 23, 2026, for any fine-tuned checkpoints. So the runway is months, not years.
That sounds like a problem. It is, in fact, exactly what I want for a sandbox experiment. If the fine-tune turns out to be worthless, the runway expires before I waste any real budget. If it turns out to be useful, I have time to port the recipe to whatever comes next.
The dataset is the only thing that matters
Every guide on fine-tuning says this. Most of them then bury it in chapter three. The reason it bears repeating is that the failure modes are not subtle. A small dataset with inconsistent labeling will produce a model that has a different opinion every other response. A small dataset with too-clean labeling will produce a model that sounds like a corporate style guide. A small dataset that mixes in too much off-topic chit-chat will produce a model that wants to talk about itself.
The discipline I settled on, after one bad first pass, was strict. Every example has to clear three gates:
- The user turn is something I would plausibly say, not a generic prompt template
- The assistant turn actually answers the user turn instead of pivoting to a safer topic
- The assistant turn would survive being read aloud without me cringing
That third gate is the underrated one. A fine-tune trains on style as much as on content, and "would I read this aloud" is a faster style judge than any benchmark. If I wince, the example is out. There is no negotiation.
What I'm not doing, on purpose
There is a temptation, when a pipeline is small and the dollars are tiny, to throw everything at it. I am resisting two specific temptations:
- Massive system prompts. The system prompt in each example stays under 600 tokens. Anything longer is a sign I'm trying to fine-tune what I should be prompting for at runtime.
- Synthetic data. I am not asking a strong model to generate fake Tun-style responses to inflate the dataset. Synthetic data smooths the distribution, but it also smooths out the parts of my voice that are most useful — the awkward ones, the half-formed ones, the ones where the model has to fill in the gap. A smoothed Tun is a worse Tun.
The current plan is to pipeline-test on gpt-4.1-nano with this 925/49 set, see if the loss curve flattens where I expect, eyeball a few generations against my own responses, and then decide whether the full dataset — somewhere closer to three times this size — is worth assembling at all.
The bet underneath the bet
The thing I keep reminding myself is that a fine-tuned nano model is not a replacement for anything I currently use. It is a fast layer — the one I hit for cheap, high-volume, low-stakes work where the answer just has to sound like me and not embarrass me. For anything that needs actual reasoning, the slow layer stays in charge.
If the experiment works, I will save a small amount of money per month and a larger amount of typing. If it doesn't work, I will have learned exactly how my own voice behaves under pressure, which is its own kind of useful.
The dataset is on disk. The job is queued. The runway is short on purpose.
What part of your own setup would you trust a tiny fine-tune with — and what part would you never hand over?
