Dev & EngARTICLE

Vercel's AI Gateway unifies private models and catalog models in a single API

Vercel's AI Gateway unifies private models and catalog models in a single API
Image: Carina Ferreira
Article · Dev & Eng
Reading
5 min
Code
2
In short

The AI Gateway's Private Inference feature lets you plug fine-tuned, self-hosted, or prerelease endpoints into the same authentication, SDKs, and observability used for Vercel's catalog models.

What changes: one gateway for everything

Until recently, teams that maintained fine-tuned, self-hosted, or prerelease models needed a parallel infrastructure: separate API keys, their own client code, and a logging pipeline isolated from the rest of the stack. That was the price of stepping outside a provider's ready-made catalog.

Vercel's AI Gateway now offers the Private Inference feature, which connects these in-house endpoints to the same authentication layer, SDKs and APIs, and observability used by catalog models (OpenAI, Anthropic, and so on). In practice, the model you trained or host becomes just another option for model in your call, alongside what's already available.

Private Inference dashboard in AI Gateway lists private models registered by group, such as Acme Inference and Research Cluster, along with their paths and API types
Private Inference dashboard in AI Gateway lists private models registered by group, such as Acme Inference and Research Cluster, along with their paths and API types. Reprodução: vercel.com.

In short: AI Gateway stops being just a router for third-party models and also becomes the single entry point for any model your company controls, whether bought or trained in-house.

How to connect your own endpoint

The flow described in Vercel's documentation follows three steps:

  1. Open the Private Inference page in the AI Gateway dashboard and register the endpoint as a "private provider," providing the base URL, the API format (Chat Completions, Responses, or Messages), and the credentials.
  2. Register the models that provider serves, giving each one an internal slug.
  3. Call the model by its private/<slug> ID anywhere you already use a catalog model today.

The same process can be done via the Vercel CLI, version 65.16.0 or later, which matters for teams automating infrastructure in CI/CD:

bash
# Connects your endpoint as a private provider
vercel ai-gateway private-inference providers create acme-vllm \
 --base-url https://vllm.example.com/v1

# Exposes the endpoint's model as private/support-fine-tune
vercel ai-gateway private-inference models create support-fine-tune \
 --provider acme-vllm \
 --api-model acme/support-llama-ft

Note that the endpoint can be anything compatible with the supported formats, including your own vLLM server, which covers the common scenario of teams that fine-tune models and host inference themselves.

From the command line to code: the practical example

Once registered, the private model behaves like any other in the AI SDK. Vercel's own example shows usage with streamText, including a fallback to a catalog model in case the private endpoint fails:

typescript
import { streamText } from 'ai';

const result = streamText({
 model: 'private/support-fine-tune',
 prompt: 'Draft a reply to a customer who was charged twice.',
 providerOptions: {
 gateway: {
 // Optional fallback to a catalog model
 models: ['anthropic/claude-sonnet-5.5'],
 },
 },
});

for await (const text of result.textStream) {
 process.stdout.write(text);
}

Switching from a private model to a catalog model (or vice versa) literally becomes just changing the value of model. This matters for teams that currently maintain separate code branches or wrappers just to switch between an internal fine-tuned model and an off-the-shelf model: the calling logic becomes the same.

Observability and fallback: what it actually solves

The most practical point of the announcement, for those running production workloads, isn't the connection itself, but what it avoids duplicating:

  • Unified logs: requests to private models appear alongside catalog requests in AI Gateway logs, trace drains, and custom reports, with the same volume, token, and time-to-first-token charts broken down by model.
  • Scoped by team: private models stay restricted to the Vercel team that registered them, don't appear in the public catalog, and credentials are stored encrypted.
  • Zero Data Retention per provider: you can enable, provider by provider, a zero-retention policy, something regulated teams (healthcare, financial) tend to require contractually.
  • Coding agents: private models become available to Claude Code, Codex, Cursor, and other agents already connected to AI Gateway, without separate reconfiguration.

This directly addresses the fragmented observability problem: previously, a fine-tuned model running outside the catalog created a blind spot in the dashboard, forcing the team to build its own telemetry or accept operating in the dark.

The trade-offs nobody announces in the changelog

Not everything here is a free gain. Putting your own endpoint behind AI Gateway means every call passes through a Vercel format-translation layer before reaching your server, which adds a network hop and one more dependency in the availability chain.

There's also a trust aspect: even with encrypted credentials and optional Zero Data Retention, traffic from your proprietary model passes through Vercel's infrastructure before reaching the endpoint you control. For teams with strict data sovereignty requirements, this can be a point of friction to check with the compliance team before migrating.

And there's the opportunity cost of lock-in: the more your application depends on the AI SDK and AI Gateway's call format to orchestrate your own models, the more expensive it becomes to switch gateways in the future. This isn't a problem exclusive to Vercel, but it's compounded when you also move the private part of your AI stack into that same router.

When it's not worth it

If your team only uses catalog models (GPT, Claude, Gemini via public API) and never intends to host or fine-tune anything, Private Inference simply doesn't apply: there's no gain in unifying what's already unified. The feature also matters less for those who don't use Vercel's AI SDK or AI Gateway today, because the central advantage is precisely reusing an observability and authentication layer that already exists in your stack.

It's worth a closer look if your company already maintains (or plans to maintain) fine-tuned models in production, especially those served by something like your own vLLM: in that case, trading duplicated logs and authentication for a single dashboard is a real reduction in operational overhead, not just a marketing promise. The full documentation on supported model types, requirements, and billing is on Vercel's site, and it's worth reading before migrating anything that's already in production.

Translated from the Brazilian Portuguese original · Read the original

Read also