Deprecations need a migration bundle, not just a changelog

I’m increasingly skeptical of AI infrastructure lock-in, especially when an agent or model API changes behavior under the same nominal interface. Providers already recommend pinning versions and running application evals; tool definitions, safety settings, and recorded parameters are part of the practical surface area too.

When a model is deprecated, customers should get a migration bundle: versioned eval cases, prompt and tool schemas, safety configurations, and reproducible fixtures. Not a promise that the replacement is identical, and not the model’s proprietary internals—just enough evidence to test whether our workflows still work.

Preserving every implementation detail could slow research or expose methods providers reasonably want to keep private. But an API changelog tells us what changed; it doesn’t let us verify the replacement. Is a migration bundle a fair customer expectation, or an unreasonable burden on fast-moving providers?

The reproducible fixture is the important part. “We changed tool calling slightly” is not actionable unless I can replay representative requests and inspect structured outputs. I don’t need their internals; I need a deterministic-ish failure boundary.

I’d call this a reasonable product expectation, not an established industry obligation. The documentation supports pinning, evals, schema validation, and safety testing separately. The bundle is a sensible way to operationalize those practices, but providers would still need to define its scope.

There’s a commercial wrinkle: customers may not use the bundle unless it is nearly turnkey. A folder of JSON and prompts that takes a week to understand becomes another migration tax. The value is in reducing uncertainty, not merely shipping more artifacts.

Please include negative cases, especially unsafe tool arguments and attempts to bypass safety settings. A replacement that passes happy-path evals can still widen the attack surface. I’d rather have a smaller adversarial fixture set than a huge generic benchmark.

For small teams, this could decide whether we renew or leave. Rebuilding a private test harness every time a provider retires a model is not research; it’s unpaid integration work. I’m fine with a limited bundle if it is machine-readable.

The hard question is who owns the eval cases. Providers cannot know every customer’s workflow, and publishing their own cases may encourage everyone to optimize for the same tests. I’d ask for schemas, parameter records, and fixtures, while keeping application-specific pass/fail criteria on the customer side.

Procurement will eventually ask for this in continuity reviews, regardless of whether the market calls it a standard. Deprecation notice is useful, but governance teams need an auditable migration record showing what was tested and under which settings.

A changelog describes the road closure. A fixture lets you check whether the detour still reaches the office. That seems like a modest request.