It is a Tuesday. The vendor blog post lands with a headline you have been half-expecting for six months. A new flagship shipped, and the model you have been running in production has a sunset date. Twelve months from now, your /chat endpoint stops working unless you migrate. Your prompt template was tuned against the old model's quirks. Your prompt cache is warm on its tokeniser. Your evals were measured against it. And the shiny new model you were told is "better on average" is on average across benchmarks that are not yours.

You have three surfaces to worry about, and each is worse than it looks. The prompts might behave differently on the new model, sometimes in ways that only show up on a slice of your traffic. The prompt cache from Issue 009 resets, because the new model has a different tokeniser or a different KV shape or both, and your input cost per request roughly triples on day one. The evals from Issue 003 are all measurements against the old model, so you do not actually know whether the new model is better on your workload until you run them again.

This issue gives you the migration playbook. A shadow-eval gate that scores candidate models against a frozen golden set before any traffic moves. Prompt-portability rules that shrink the re-tuning surface up front so migration is not a rewrite. And the Issue 009 interaction, because the day the new model goes live is the day your cached prefixes all reset to zero. Composes with the observability from Issue 004; you will need it.

Why migrating a live LLM is harder than it looks

You have done this before with a database. A schema migration is scary but bounded. You test the migration, you flip the read side, you cut traffic over, and if something breaks you roll back. Model migration looks like the same shape until you look at what you are actually migrating.

Your prompts were tuned against a specific model's behaviour. Some of that behaviour is documented (context window, JSON mode, tool-calling shape). A lot of it is not. If your system prompt says "output must be three paragraphs of at most 100 words each" and the old model reliably obeyed it, that is an undocumented capability you were counting on. The new model might obey it slightly less reliably, or with slightly different word budgets, or with a subtle shift in tone. You will not know until you measure.

The evals from Issue 003 were all measurements against the old model. You know your customer-facing metric was 0.94 on the golden set. That number is meaningless the moment you swap models. You need the same measurement against the candidate before you can say "this is better", and you need it sliced by prompt template and by tenant and by query type. Averages hide everything that matters. The vendor benchmark that says "MMLU plus three points" tells you approximately nothing about your workload.

The prompt cache from Issue 009 is the surprise most teams do not budget for. Cached prefixes are keyed by tokens, and tokens depend on the tokeniser. When you switch models, even at the same vendor, you are often switching tokenisers. The cache resets. Your input cost per request roughly triples on day one, because you are paying the full-price rate on prefixes that used to be at the ten-percent-off cache-read rate. On Anthropic's five-minute ephemeral cache the loss rebuilds in an afternoon of steady traffic. On the one-hour cache it takes longer. On OpenAI's automatic caching you might not even notice until the bill lands.

The shadow-eval gate

The migration playbook has one central mechanism, and it is the thing you build once and reuse forever. Score the candidate model against a frozen golden set before any real traffic moves.

Freeze the golden set from Issue 003 as the reference. That set is your "what good looks like" ruler for this migration and the next one. Do not update it in the same PR that runs the candidate model against it. You want to compare like-for-like. When the migration is done, then you update the golden set.

Run the candidate model on the frozen set. Score every response through the three-layer evaluator (deterministic checks, reference comparison, LLM-as-judge). Store the results with the candidate's model version tag so you can compare them side by side with the old model's scores.

Compare the shape of the improvement, not just the aggregate. The vendor said plus three on MMLU. You want to know whether that plus three shows up on the queries that matter to your users. Slice by prompt template. Slice by tenant. Slice by query length. Slice by whether the query contains an identifier or a code block. Somewhere in those slices, you will find that the new model is better on eighty percent of queries and materially worse on the other twenty. Whether you can ship depends on which twenty.

The diagram shows the gate. The frozen golden set feeds both the current model and the candidate. Their scores get compared slice by slice. If any slice shows a regression bigger than your threshold, migration is blocked until the prompt is tuned or the slice is understood. Only when every slice passes does traffic get to ramp.

Set the threshold to something like "no slice regresses by more than five points on the customer-facing metric". A five-point drop on a five-percent slice is a twenty-five-percent regression for that user cohort. They will notice. You should catch it before they do.

Prompt portability rules

You can shrink the re-tuning surface at design time. This is the part of the article that saves you the most work when the next model ships.

Every model-specific quirk you rely on is a quirk you have to re-verify on migration. So do not rely on undocumented capabilities. If your system prompt says "output at most three paragraphs" and it works reliably on the old model, that is fine. If your system prompt says "always use passive voice" and it works reliably on the old model, that is a quirk you are going to lose. The vendor does not promise passive-voice adherence in its docs. The moment you swap models, your passive-voice rate changes.

Keep the system prompt tight. Fewer rules means fewer places for a new model to disagree with the old one. Rules that specify format (JSON schema, section headers, citation style) tend to port well because they are testable and vendors optimise for them. Rules that specify style (tone, verbosity, pacing) tend to port poorly because they depend on the model's inductive biases.

Test with at least two models throughout development. If your prompt only ever runs on one model in dev, you have built a spec that says "this prompt works on that one model, at this moment in time". If you test against Claude and GPT throughout, or against two variants at the same provider, you build a prompt that is more likely to survive the next migration because it does not depend on either model's flavour.

Separate model-specific config from model-portable prompt logic. In code, this means a model_config.yaml that carries the model name, temperature, max tokens, and any model-specific system-prompt append (a warning suffix, a jailbreak-avoidance line the vendor recommends), while the main prompt template stays the same regardless of model. Migration then becomes editing one YAML file, not rewriting a template.

The Issue 009 interaction: your cache just reset

The day the new model goes live is the day your cached prefixes all go to zero. This catches every team the first time, and half of them the second time.

Anthropic's cache is keyed by the token prefix, and the tokeniser is model-specific. Sonnet 4 tokens are not Sonnet 4.5 tokens. Even if the prompt template is byte-identical, the token stream is not, and the cache does not match. OpenAI's automatic caching has the same property. So does Google's context caching. This is not a bug you can work around; it is a consequence of how KV-cache reuse works.

On the first day after migration, your input cost per request roughly triples for the cached portion. You are now paying the write-cost premium on every prefix rather than the read discount. If your traffic pattern is steady, this rebuilds in a few cache windows. If your traffic pattern is bursty, it takes longer, and the daily aggregate cost spike shows up as a red line on the graph the morning after.

Two things to do about this. Monitor cache hit rate as a first-class metric before, during, and after migration; you want to watch it drop to zero and climb back. And budget for the extra cost during the ramp period. A canary rollout that starts at ten percent of traffic on the new model has a much smaller cache-cost spike than a full cutover, which is one more argument for the canary if you needed one.

Common mistakes

Four failure modes come up on almost every migration.

Reading vendor benchmarks as if they were your workload. MMLU, HumanEval, GPQA: these are useful for tracking progress on general capability, and they are not useful for predicting how the new model handles your specific prompts and your specific corpus. The frozen golden set from Issue 003 is your workload. Trust it over the vendor's chart.

Migrating without re-running the eval. You would be surprised how many teams flip a model version in production without ever running the new model against the golden set first. The rationale is usually "the vendor says it is better". The vendor is telling the truth on average across their benchmarks. That has nothing to do with your customers. Run the eval.

Ignoring the cache reset. The bill spike on migration day one is real, predictable, and preventable. You should not be surprised by it. Watch the cache-hit metric during the ramp, and budget for the extra spend up front. Add a line to the migration ticket that says "expect input cost to triple for 24-48 hours".

Cutting over all at once. A full cutover to the new model is a big enough change that it deserves the same canary treatment you would give a database migration. Ten percent, fifty percent, one hundred percent, with an eval-based gate between each step. Anything less is trusting the vendor benchmarks more than you should.

The takeaway

Model migration composes three earlier issues. The golden set from Issue 003 is your ruler. The observability from Issue 004 tells you which slices matter. Issue 009's prompt caching resets on the tokeniser change, which is a one-day cost spike you should budget for. The migration playbook is a shadow-eval gate that scores the candidate against the frozen set before traffic moves, prompt-portability rules that reduce how much re-tuning you have to do, and a canary rollout that ramps traffic while you watch the cache-hit metric climb back to steady state. Build it once. Every future model migration uses the same playbook, and the sunset dates on your vendor's changelog stop being surprises.

Production checklist

  • Freeze the golden set from Issue 003 as your migration reference. Do not update the golden set in the same PR that scores a candidate model.

  • Run the candidate model against the frozen set with the three-layer evaluator. Store its scores with the model version tag next to the current model's scores.

  • Compare slice by slice: prompt template, tenant, query type, query length. A five-point drop on a five-percent slice is a twenty-five-percent regression for that cohort.

  • Set a per-slice regression threshold. Block migration if any slice exceeds it. Tune the prompt or investigate the slice before proceeding.

  • Audit the system prompt for undocumented model-specific quirks. Rules that specify format tend to port; rules that specify style tend not to.

  • Separate model-specific config (model_name, temperature, max_tokens) from the model-portable prompt template. Migration should be one YAML edit, not a prompt rewrite.

  • Track cache-hit rate as a first-class metric during migration. Expect it to drop to zero on day one and climb back within a cache window's worth of traffic.

  • Ramp with a canary: 10 percent, 50 percent, 100 percent, with an eval-based gate between each step. Do not do a full cutover.

  • Budget the input-cost spike from the cache reset for the first day, then verify it reverts as the cache warms.

  • Re-run the migration playbook quarterly. New models ship faster than you think, and the second use of the playbook costs almost nothing.

Further reading