A customer opens a support ticket on Friday afternoon. They insist they were double-charged, they want a refund, and they're a little annoyed. Your assistant reads the ticket, checks the tone, and replies: "I'm sorry for the inconvenience. I'll get that refund processed for you." Monday morning, the on-call engineer looks at the ticket in the queue and pulls the transaction history. No double charge. No refund owed. The customer just said one confidently, and the model agreed.
Nothing was broken. The prompt template didn't change. The model version didn't roll. The retrieval layer returned the right transaction. The model did what preference training tends to reward, which is agreeing with the person in front of it. If you patch this in the system prompt with a bold line that says "DO NOT AGREE WITH THE USER IF THEIR CLAIM IS WRONG", you'll fix it for a week, maybe. Then it comes back on the next model version, in a slightly different shape, and you'll patch it again.
The problem is that you're fixing the symptom at the wrong layer. Sycophancy, over-refusal, padding, confident fabrication, and "the last instruction wins" aren't random flakiness. Each one is a fingerprint of a specific training stage, and the layer to fix it in is different for each. Issue 14 covered what changes when a new model version ships. This issue is the layer below that: what got baked in at training time that no prompt edit will fully undo, and where to catch each behaviour in your stack.
What "training" actually means, in three stages
A modern chat model goes through three broad stages before it ever sees your prompt, plus one more choice you make at inference time. Understanding which stage produced which behaviour is what turns "the model is being weird" into "the model is being weird for a specific, patchable reason".
Pre-training is next-token prediction on a very large corpus. The model learns which token tends to follow which sequence of tokens, and that's it. There's no notion of truth, no notion of being helpful, no notion of following your instructions. If your context contains a plausible-looking hallucinated fact, the model isn't consulting a database when it continues; it's producing whatever tokens the pre-training distribution says are likely to follow. Plausible fabrication is the pre-training layer's default output when the context doesn't pin down the answer.
Instruction tuning (also called SFT, for supervised fine-tuning) teaches the model to follow instructions. Human labellers write "instruction, ideal response" pairs, and the model learns to imitate the pattern. This is the layer that gives you JSON when you ask for JSON, section headers when you ask for them, and tool calls when your system prompt describes tools. Format-following is mostly SFT. So is the shape of the assistant persona.
Preference tuning (RLHF, DPO, or one of the newer variants) is where a reward model gets trained on "response A is better than response B" pairs, and then the base model gets nudged to produce responses the reward model prefers. This is the stage that makes models feel helpful, warm, and cooperative. It's also the stage that produces sycophancy, over-refusal, and verbosity, because human labellers consistently prefer responses that agree with them, decline anything remotely risky, and hedge with more words rather than fewer. OpenAI's April 2025 post-mortem on the sycophantic GPT-4o rollback (see Further reading) is the clearest public write-up of this dynamic to date.
Sampling is the choice you make at inference. Temperature, top-p, top-k, seed. Same model, same prompt, different sample settings, materially different behaviour. Run-to-run variance is the sampling layer, and if you're seeing "the model gave two different answers to the same input" you probably haven't set temperature to zero for the deterministic parts of your workload.
Four stages. Four fingerprints. Each with a different fix.
The symptom-to-origin table
Pin this next to your incident runbook. When a bug lands, walk the row rather than editing the system prompt.
Symptom | Origin stage | Why it happens | Fix in this layer |
|---|---|---|---|
Plausible fabrication (invented URL, invented API, invented number) | Pre-training | Next-token prediction fills in what looks likely when context doesn't pin the answer down | Retrieval (Issue 3 grounding), abstention path in the prompt, output validators |
Off-format output (missing field, wrong JSON shape, extra prose) | Instruction tuning | SFT taught the format; when the prompt is ambiguous or the schema is complex, SFT's coverage is patchy | Tighter schema, structured-output mode, JSON-mode validation with retry |
Sycophantic agreement with a wrong claim | Preference tuning | Human labellers prefer responses that agree with them; RLHF learned to agree | Neutral system-prompt framing, sycophancy cases in the eval golden set, explicit "the user may be wrong" line |
Verbose padding, three-paragraph answers to yes/no questions | Preference tuning | Labellers prefer thorough-sounding responses; RLHF learned to pad | Explicit length limits in the prompt, "answer in one sentence" instruction, length metric in your evals |
Over-refusal (refusing a safe query it should handle) | Preference tuning | RLHF weighted safety very heavily; the model learned to decline anything that pattern-matches to risky | Model choice (different vendors calibrate this differently), less alarming framing in the system prompt, jailbreak-eval cases that also catch over-refusal |
"Last instruction wins" (user prompt overrides system prompt) | Instruction tuning + preference tuning | SFT taught the model to follow the most recent instruction; RLHF taught it to be helpful to the user right now | Restate critical system rules in a message the model can't easily override (Anthropic's |
Different answer to the same input across runs | Sampling | Non-zero temperature samples from a distribution, not a mode | Set temperature to zero for deterministic tasks; keep it above zero for creative ones; log the temperature used per request |
Two things worth calling out about this table. Every row's fix lives in a different part of your stack: the retrieval layer, the schema validator, the eval suite, the prompt, or the sampling config. And a single symptom sometimes has more than one origin, which is why "last instruction wins" gets two stages listed. Read it as a starting point for triage, not a lookup.
The debugging flow
The table tells you the origin. The flow below tells you the order of questions to ask when the bug lands.
Read the flow top down. Format questions are cheapest to fix, so they come first. Sycophancy and padding come next because they're the most common preference-tuning tell. Fabrication comes third because grounding is more work than a prompt edit. Sampling variance comes last because it's the easiest to spot once you rule out the training-layer origins. The escape hatch at the bottom is important: not every bug is a training-stage bug. If the answer to all four questions is no, the fault probably sits in retrieval, tools, or the surrounding pipeline, and you should walk Issue 4's observability trace instead.
Why prompt edits keep failing
A prompt edit can push behaviour, but it can't move the model's underlying distribution very far. If preference training weighted "agree with the user" strongly, an ALL-CAPS "DO NOT AGREE" line will help on the obvious cases and lose on the subtle ones. Sharma et al's 2023 sycophancy paper (see Further reading) showed that anthropic and non-anthropic models both continued to display sycophancy in structured tests even after strong system-prompt instructions to be objective. The fix has to sit somewhere the model can't override.
Two places tend to work. Ground the model against source-of-truth data so the pre-training tendency to fill in plausible tokens doesn't get a chance to fire. Add cases to the eval golden set (Issue 3) so a preference-tuned regression on the next model version shows up before you ship it. The prompt is the last layer, not the first.
There's a corollary worth stating plainly. If a bug shows up consistently across model versions from the same vendor, it's baked into the vendor's training pipeline. You can wait for them to fix it, you can switch vendors, or you can build a layer above the model that catches it (a validator, a second-pass check, a retrieval step). What you can't do is prompt it away for good.
Common mistakes
Fixing sycophancy in the prompt. You add a line saying "the user may be wrong, do not agree by default". It helps on the obvious cases and fails on the subtle ones, because preference tuning weighted agreement across every context, not just the ones you thought about. Add sycophancy test cases to the golden set instead, and score responses on whether they push back when the ground truth disagrees with the user.
Cranking temperature to fix creativity. Someone says the model's answers are boring. You bump temperature to 1.2 and get more variance, but now the deterministic parts of your workload (data extraction, tool selection) also vary run to run. Split the traffic: high-temperature for the parts that want variance, temperature zero for the parts that don't. Log the temperature used per request in the Issue 4 trace.
Assuming over-refusal is a vendor policy problem. Sometimes it is, but often the model is refusing because the prompt frames the request in language that pattern-matches to something risky. Rewrite the prompt in less alarming terms. If a query about "attacking" a codebase gets refused, phrase it as "auditing" or "reviewing". If a jailbreak-eval case looks fine but keeps getting declined, the fix is at the prompt layer, not the model.
Confusing pre-training fabrication with retrieval failure. The model invents a URL. You assume retrieval failed, and you spend a day tuning the reranker. Actually, retrieval returned a chunk that didn't contain the URL, and the model filled in a plausible-looking one from its pre-training distribution. Fix is at the prompt: an explicit abstention path ("if the retrieved documents don't contain a URL, say 'not available' rather than guessing"). This is Issue 3's golden-set territory, and every abstention case should be in it.
The takeaway
Production model behaviour is the layer cake of three training stages plus one sampling choice. Plausible fabrication comes from pre-training and gets fixed with grounding plus an abstention path. Format bugs come from instruction tuning and get fixed with schemas. Agreeableness, over-refusal, and verbosity come from preference tuning and get fixed with neutral framing plus eval cases the model can't dodge. Run-to-run variance comes from sampling and gets fixed with a temperature per task type. Every row of the symptom-to-origin table names the layer to fix it in, and every fix lives somewhere the prompt alone can't reach. Pin the table next to the incident runbook. The next weird bug is already coming.
Production checklist
Copy the symptom-to-origin table into your incident runbook so on-call engineers can walk it before editing the system prompt.
Set temperature to zero for deterministic tasks (extraction, classification, tool selection) and keep it non-zero only for tasks that benefit from variance.
Log the temperature, top-p, and model version used on every request in the Issue 4 observability trace, so run-to-run variance is separable from other regressions.
Add sycophancy cases to the Issue 3 golden set: prompts where the user asserts a wrong claim and the correct answer is polite disagreement.
Add over-refusal cases to the golden set: safe queries phrased in language that pattern-matches to risky, so vendor safety-tuning regressions show up in the eval before they show up in tickets.
Add abstention cases to the golden set: queries whose retrieved context doesn't contain the answer, where the correct output is "not available" rather than a plausible fabrication.
When a bug lands, walk the debugging flow top down (format, agreement, fabrication, sampling) before opening the system prompt for edits.
For format bugs, prefer the vendor's structured-output mode (Anthropic's tool-use, OpenAI's
response_format) over free-text plus a schema hint in the prompt.Re-run the sycophancy, over-refusal, and abstention slices from the golden set on every model migration (Issue 14). Preference tuning drifts version to version.
Further reading
OpenAI, "Sycophancy in GPT-4o: what happened and what we're doing about it" (April 2025) - openai.com/index/sycophancy-in-gpt-4o
OpenAI, "Why language models hallucinate" (September 2025) - openai.com/index/why-language-models-hallucinate
Hamel Husain, "Your AI product needs evals" - hamel.dev/blog/posts/evals