Prompting vs fine-tuning: which should you use?
A player asked whether their assistant needed a better prompt, fine-tuning, or an entirely new model. This is the AI version of requesting a new game engine because one quest marker is in the wrong place. Sometimes the engine is the problem. Usually the instruction, map, or test is missing.
The short answer: prompt first, tune later
Use prompting first. Add retrieval when the model needs private, specialised, or changing facts. Consider fine-tuning only when a good prompt still fails at the same measurable behaviour across many representative examples. Training a foundation model from scratch is a different scale of project and is almost never the next step for an ordinary product team.
That order is not merely the cautious answer. Google’s current tuning guidance recommends finding the best prompt first, then tuning if recurrent errors remain. OpenAI’s model-optimisation workflow similarly starts with evaluations and prompting before fine-tuning. Providers, models, and tuning availability change, but the decision logic travels well.
Prompting tells a model what to do now. Retrieval gives it facts for this answer. Fine-tuning changes how it tends to behave.
What prompting actually changes
A prompt supplies instructions, context, examples, and an output format at request time. It does not rewrite the model’s learned weights. This makes prompting the fastest and most reversible lever: you can edit one sentence, run the same test set again, and roll back without training or deploying a new model.
Start by turning the vague request into a small specification:
- Task: what must the model do?
- Inputs: what information may it use?
- Constraints: what must it avoid or admit it cannot know?
- Output: what exact structure should it return?
- Examples: what do two or three good answers look like?
Few-shot examples are especially useful when a format is easier to show than describe. My earlier field note on getting more out of ChatGPT covers the practical prompting habits. The important upgrade here is to test that prompt on a fixed set of realistic cases instead of judging it by one impressive demo.
Use retrieval when the problem is knowledge
Fine-tuning is often prescribed for the wrong symptom. If the assistant does not know today’s refund policy, a customer’s account state, or the latest product manual, teaching it a behavioural pattern will not keep those facts current. Retrieve the relevant source at request time and place it in the model’s context.
OpenAI describes retrieval as semantic search over your data, with the results available for a model to synthesise. Google’s generative-AI glossary makes the useful distinction: retrieval-augmented generation grounds output in information fetched after training. That is why retrieval is the better fit for facts that change, documents that need citations, or information that differs by user.
Use retrieval when the sentence “the answer is already written somewhere” is true. Use fine-tuning when the sentence “we can show many examples of how the answer should be produced” is true. You may need both: retrieval supplies the policy; a tuned model follows a specialised response pattern.
What fine-tuning changes
Fine-tuning starts with an existing model and trains it on example inputs paired with desired outputs. The process adjusts model parameters so the desired behaviour becomes more likely without repeating a long set of demonstrations in every prompt.
Good candidates are narrow, repeated tasks with an objective definition of success:
- returning a specialised format that prompts repeatedly miss;
- classifying or extracting information using a stable house taxonomy;
- following a distinctive response policy across high request volume;
- correcting a recurring instruction-following failure demonstrated in real traffic.
The training examples must resemble production inputs, including the awkward cases. Google emphasises representative, well-labelled data; OpenAI’s supervised fine-tuning guide likewise says to build realistic examples and establish evaluations before investing in tuning. A folder of hand-picked successes teaches the model the clean tutorial level while your users arrive through the swamp.
Fine-tuning is not a facts database
Do not fine-tune merely to insert a company handbook, product catalogue, or current price list. Learned information is harder to inspect, cite, update, and delete than a retrieved document. It may also be recalled incompletely. Put changing knowledge in a maintained source; reserve tuning for behaviour that remains useful across many questions.
Fine-tuning also does not remove the need for a prompt. The application still needs to state the current task, provide user-specific context, and define boundaries. Nor does tuning guarantee identical output: generation remains probabilistic, as the field note on why AI gives different answers explains.
When training from scratch makes sense
Training from scratch means creating the base model rather than adapting one. It requires a model architecture, a very large and carefully prepared corpus, serious computing infrastructure, specialist researchers and engineers, safety work, and a plan to evaluate and serve the result. It can make sense for organisations building foundational technology, working in an underserved language or modality, or needing control that available models cannot provide.
For most teams, it is not an escalation from fine-tuning. It is a separate quest line. At Wistkey, the practical question is usually how to assemble reliable workflows around capable models—not how to recreate the models underneath them.
A five-step decision test
Before spending money on tuning, run this sequence:
- Define the failure. “It is bad” cannot be measured. “It omits one required field in 18% of support summaries” can.
- Build a small evaluation set. Include normal, edge, and adversarial cases drawn from the work you expect.
- Improve the prompt. Clarify instructions, add representative examples, constrain the output, and retest.
- Separate facts from behaviour. If the missing piece is knowledge, add retrieval or a tool. If it is a repeated response pattern, tuning may fit.
- Tune only with a baseline. Compare the tuned model with the best prompted base model on held-out cases, including cost and latency. Keep it only if the measured gain matters.
The practical answer
Choose the least permanent intervention that solves the measured problem. Prompting is cheap to change. Retrieval keeps facts inspectable and current. Fine-tuning can make a repeated behaviour more reliable or efficient when you have enough good examples. Training from scratch belongs to teams whose actual product is the model—or whose requirements cannot be met any other way.
I have marked the quest accordingly: fix the instruction, then the map, then the character build. Replacing the entire game engine remains available, but it no longer fits in the “quick settings change” ticket.
