Migrating off OpenAI to open weights

Three lines, same SDK

Most teams assume migrating off OpenAI means rewriting every integration, retraining prompt templates, and swapping SDKs. It does not. The OpenAI Python client and its equivalents in TypeScript, Go, and Java all accept a configurable base_url. Point that at an OpenAI-compatible inference endpoint, change the model name, and your existing code keeps working. The API surface — chat completions, streaming, tool calls — is the same because your code talks to the same interface.

from openai import OpenAI

  # Before: calling a proprietary API
  # client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  # resp = client.chat.completions.create(
  #     model="gpt-4o",
  #     messages=[{"role": "user", "content": "Analyze this contract."}],
  # )

  # After: same SDK, different endpoint, zero data retention
  client = OpenAI(
      base_url="https://api.hyperinfer.ai/v1",
      api_key=os.environ["HYPERINFER_API_KEY"],
  )

  resp = client.chat.completions.create(
      model="deepseek-v4-pro",
      messages=[
          {"role": "user", "content": "Analyze this contract for unusual terms."},
      ],
  )
  print(resp.choices[0].message.content)

Phase 1 — Shadow and evaluate

Before routing a single production request to a new model, run your existing traffic through it in shadow mode. Send the same prompts to both the old API and the new one, log both responses, and compare them. Focus on three signals: output quality on your actual tasks (not generic benchmarks), latency at your concurrency level, and failure modes you did not anticipate (different refusal behavior, different instruction-following patterns). This phase costs you nothing in user experience and tells you everything about whether the model is a real replacement.

Phase 2 — Route a percentage

Once shadow evaluation looks clean, route a small fraction of live traffic — 5 percent is a good starting point — to the new model with a feature flag. Monitor output quality, latency, and error rates side by side. Keep the old model as a fallback on the same flag so you can revert in seconds if something breaks. Gradually increase the percentage as confidence grows. The key discipline: do not skip from shadow to full cutover. The intermediate step catches edge cases that never appeared in offline evaluation.

Phase 3 — Cut over and clean up

When the new model has served a meaningful share of traffic for long enough that your monitoring is quiet, flip the flag to 100 percent and decommission the old integration. Remove the old API keys, revoke access, and confirm that no code path still references the previous endpoint. The cleanup step matters because a forgotten fallback that silently reactivates is a compliance finding waiting to happen.

Abstract the client now, migrate later

Even if you are not planning to migrate today, wrap your LLM calls in a thin abstraction layer that accepts a configurable base URL and model name. That costs you an afternoon of engineering and pays off the moment your legal team asks whether your current provider logs your prompts or your CFO asks what the API bill is funding. Vendor lock-in is not a disaster — it is a tax you pay every month. A two-minute config change that frees you from it is worth building now.

Picking the right model and verifying parity

Not every open-weight model is a drop-in replacement for every workload. The model you pick matters. Choosing the right one for regulated use means evaluating capability, license, and context length against your actual tasks, not a leaderboard. Once you have a candidate, verify parity with a focused eval suite on your own data: prompt-response pairs from production, edge cases your team has hit before, and the tasks where your users are most sensitive to quality regressions. A model that scores two points lower on MMLU but matches your production quality is a better choice than one that wins benchmarks but fails your prompts.

• Shadow first. Run both models in parallel on real traffic. Do not trust benchmarks alone.
• Route gradually. Start at 5 percent, monitor, increase. Keep the fallback one flag flip away.
• Cut over cleanly. Remove old keys and endpoints. A forgotten fallback is a compliance risk.
• Abstract the client. A thin wrapper around the OpenAI SDK with configurable base_url costs an afternoon and eliminates lock-in.
• Verify on your data. Run your own eval suite against production prompts, not just public benchmarks.
• Pick the license first. MIT or Apache 2.0 models give you the most deployment freedom.

Migrating off a proprietary API is not a sprint. It is a phased engineering exercise that, done right, improves your cost, your privacy posture, and your negotiating position with every vendor. The OpenAI-compatible API surface means the technical lift is smaller than most teams assume. The real work is the evaluation discipline. Talk to an engineer if you want a scoping call for your migration.