Realtime protocol

The Opper Realtime API lets you build voice-to-voice applications. Your customer speaks, the model responds with audio in real time, and tool calls flow back and forth without you holding the connection together yourself. The same protocol works across OpenAI, xAI, and Gemini, so you switch upstream models by changing a string. It also drives live speech-to-text (audio in, streaming transcript out) for transcription-only models such as Mistral Voxtral and Google Gemini Transcribe Live, including end-of-turn detection for voice agents and 8 kHz phone audio (see Live transcription).

How it works

You open a WebSocket to wss://api.opper.ai/v3/realtime, send a session.start frame that picks the model and configures the agent, then stream microphone audio in and play assistant audio out as it arrives. Opper resolves the provider from the model id, handles the upstream handshake (including provider quirks like xAI’s client-secret exchange), meters usage as you go, and gives you one event vocabulary so your client code stays the same no matter which model you target.

┌──────────┐   audio.append    ┌──────────┐   provider-specific   ┌────────────┐
│  Client  │ ───────────────▶  │  Opper   │ ──────────────────▶   │  Upstream  │
│ (browser │                   │ Realtime │                       │  (OpenAI / │
│ or your  │ ◀───────────────  │  proxy   │ ◀──────────────────   │   xAI /    │
│ backend) │   audio.delta     │          │   normalized frames   │  Gemini)   │
└──────────┘                   └──────────┘                       └────────────┘

Just trying it out?

The Realtime quickstart is the fastest way to a working session. It has one runnable code block per environment (server or browser), and you’ll be live in under five minutes. For a production-shaped example with browser UI, microphone capture, audio playback, tool calls, and provider switching, see the brainstorm-time cookbook.

Authentication

There are two patterns, depending on whether your client runs server-side or in a browser.

Server-side: bearer token

Server-to-server clients (Node, Python, Go, anything that runs in your backend) connect directly with a project-scoped runtime API key in the Authorization header. The quickstart uses this pattern.

new WebSocket("wss://api.opper.ai/v3/realtime", {
  headers: { Authorization: `Bearer ${process.env.OPPER_API_KEY}` },
});

Management keys (op-mak-…) are rejected. Only project-scoped runtime keys can open realtime sessions.

Browser: ephemeral tickets

Browsers can’t set an Authorization header on the native WebSocket constructor. To open a realtime session directly from a browser, your backend mints a single-use ticket and the browser redeems it on the upgrade request. Step 1: server-side mint. Your backend hits POST /v3/realtime-sessions with its normal API key. The response carries a short-lived client_secret and an expires_at timestamp.

// On your backend (e.g. /api/realtime-ticket)
const resp = await fetch("https://api.opper.ai/v3/realtime-sessions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.OPPER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    config: {
      model: "openai/gpt-realtime-2",
      voice: "marin",
      instructions: "You are a concise voice assistant.",
      tools: [/* ... */],
    },
    ttl_seconds: 60,
  }),
});
const { client_secret, expires_at, ws_url } = await resp.json();
// Return client_secret + ws_url to the browser

Step 2: browser redeem. The browser opens the WebSocket carrying the ticket in the Sec-WebSocket-Protocol subprotocol header. Prefer this form. Credentials in the URL query string end up in access logs, browser history, and Referer headers, while the subprotocol header is request-only and stays out of those places.

// In the browser, recommended
const ws = new WebSocket(
  "wss://api.opper.ai/v3/realtime",
  [`opper-ticket.${clientSecret}`],
);

ws.onopen = () => {
  // The ticket already carries the bound config. For fields that weren't
  // pre-bound at mint time, send them now:
  ws.send(JSON.stringify({
    type: "session.start",
    config: { modalities: ["audio"] },
  }));
};

A query-parameter fallback is also accepted for environments that can’t set a subprotocol (uncommon: most clients support the second WebSocket constructor argument):

// Fallback only
const ws = new WebSocket(`wss://api.opper.ai/v3/realtime?ticket=${clientSecret}`);

Ticket properties:

Single-use. Replays return 401.
Default TTL 60 seconds, max 5 minutes.

Pre-binding for security

Any field you populate in config at mint time is locked for that ticket. The browser can’t override it on session.start, and it can’t change it later with session.update either. This is the main safety guarantee. A leaked ticket can only open the session your backend authorized. It can’t pivot to a more expensive model or a different system prompt, and it can’t unlock the field by updating it after start. At minimum, bind model. Tighter setups also bind instructions, tools, and voice. Fields left zero stay open for the browser to set in session.start. To force a boolean off (for example, to stop the browser from enabling output transcription), list the field in locked_fields. Without it, a zero value in config doesn’t lock anything, so the browser could still flip the bool on. A ticket with locked_fields but no config is also valid: the listed fields are force-set to their zero value regardless of what the browser sends. Unknown field names are rejected at mint time with 400 Bad Request.

// Forbid the browser from enabling output transcription
{
  config: {
    model: "openai/gpt-realtime-2",
    output_transcription: false,
  },
  locked_fields: ["output_transcription"],
}

// Server-side mint with strong binding. Browser cannot change any of these
{
  config: {
    model: "openai/gpt-realtime-2",
    voice: "marin",
    instructions: "...locked system prompt...",
    tools: [/* customer-approved tools only */],
    reasoning_effort: "low",
  },
  ttl_seconds: 30,
}

Preflight rejections

Before the WebSocket upgrade, Opper checks balance, spend caps, whether a card is on file, and concurrency caps. Failures come back as HTTP status codes rather than opaque WS closes, so you can react to them:

Status	Meaning
`401 Unauthorized`	Missing key, non-runtime key, or project-less key
`402 Payment Required`	Balance exhausted, spend cap hit, or no card on file
`429 Too Many Requests`	Concurrent-session cap reached for this project
`503 Service Unavailable`	Realtime endpoint not configured for the requested provider

Session lifecycle

A session goes through four phases. Most of the time you only care about the middle two. 1. Connect. WebSocket upgrade. A successful response is 101 Switching Protocols. After this the connection is open, but the agent isn’t running yet. 2. Configure (session.start). Send this once as the first message after upgrade. The config selects the model, voice, system instructions, tool list, VAD parameters, and optional transcription toggles. Opper validates the config against the resolved model’s capabilities and rejects anything unsupported before dialing the upstream. The same validation runs on every later session.update, so unsupported modalities, voice, or reasoning_effort values return an error mid-stream. 3. Interact. Stream audio in (audio.append + audio.commit if not using server VAD), send text turns (text.input), receive audio.delta / text.delta frames, handle tool.call events, return tool.result. 4. Close. Either side can close the WebSocket. On a client-initiated close, Opper performs a final billing flush and tears down the upstream connection. On a server-initiated close (caps, idle timeout, balance exhaustion), you’ll see a closing event (session.terminating) with a structured reason before the connection ends.

Configuring the session

Everything goes in config on the first session.start frame:

Field	Type	Notes
`model`	string	Required. Provider-prefixed model id, e.g. `openai/gpt-realtime-2`. See Per-provider notes.
`voice`	string	Provider-specific voice name. Optional, model falls back to its default.
`instructions`	string	System prompt for the agent.
`modalities`	string[]	Output modalities. OpenAI realtime accepts a single value (`["audio"]` or `["text"]`); xAI and Gemini accept either form. Default depends on model.
`turn_detection`	object	`{ type: "server_vad", threshold, prefix_padding_ms, silence_duration_ms }` for acoustic VAD, `{ type: "semantic_vad", eagerness }` for model-based turn detection (OpenAI realtime, and transcription models with `params.realtime.semantic_vad`), or `{ type: "none" }` to drive turns yourself. Defaults are server-side.
`input_audio_format`	string	`pcm16` (default). `g711_ulaw` and `g711_alaw` (8 kHz phone audio) on models whose `params.realtime.input_audio_formats` lists them. `session.started` reports the rate to send.
`tools`	object[]	Tools the model can call, each `{ name, description, parameters }` with a JSON Schema for `parameters`. Flat, without the `function` wrapper used in gateway tool calling.
`reasoning_effort`	string	`gpt-realtime-2` only. `minimal` / `low` / `medium` / `high` / `xhigh`.
`input_transcription`	bool	Surface `transcript.committed` events with the user’s speech transcribed. Off by default.
`input_transcription_model`	string	OpenAI only. Optional selector. Falls back to `gpt-4o-mini-transcribe`.
`output_transcription`	bool	Surface `text.delta` events with the assistant’s spoken words. Off by default.
`transcription_language_codes`	string[]	Live transcription. BCP-47 codes: one pins the language, several limit detection to them. Empty means auto-detect.
`transcription_custom_vocabulary`	string[]	Live transcription. Names and terms to bias recognition towards.
`transcription_mode`	string	Live transcription. `verbatim` keeps filler words, `smart` removes them.
`diarize`	bool	Live transcription. Label each final segment with its speaker, on models with `params.realtime.diarize`; refused elsewhere.
`parameters`	object	Provider settings Opper does not model, under the provider’s own names, e.g. `{ "max_speakers": 2 }`. Each model accepts its own keys, listed in `params.realtime.parameters` on `GET /v3/models`. Any other key fails `session.start`, naming the keys the model takes.

Event vocabulary

All events are JSON over text WebSocket frames.

Client → server

Event	Purpose
`session.start`	First frame. Opens the session with config.
`session.update`	Update the config mid-session (provider-dependent; Gemini ignores this).
`audio.append`	Append base64-encoded mic audio in the session’s `input_audio_format` (PCM16 by default) at the input sample rate from `session.started`.
`audio.commit`	Mark end of speech (only needed when not using server VAD). On live transcription models it flushes the pending utterance, so its final transcript arrives at once.
`audio.clear`	Discard buffered mic audio that hasn’t been committed yet.
`text.input`	Send a typed user message.
`image.input`	Send a still image as a user message. `image_url` accepts a base64 data URI (`data:image/png;base64,...`) on every vision-capable realtime model; OpenAI also accepts https URLs, while Gemini Live requires inline data and rejects https with a clear error. Optional `text` rides on the same message item so the model sees the question alongside the image. Optional `image_detail` (`"auto"` / `"low"` / `"high"`) is honored by OpenAI and ignored by Gemini. Only supported on realtime models whose capabilities include `vision` — the gateway responds with an `error` event otherwise. Image bytes are not stored in the session trace recording.
`response.create`	Force the model to respond now (when not using server VAD).
`response.cancel`	Cancel an in-flight response.
`tool.result`	Return a tool result the model asked for via `tool.call`.

Server → client

Event	Purpose
`session.started`	Confirmation that the upstream session is live. Carries `session_id`, `input_sample_rate`, `output_sample_rate`, `audio_format` (for example `input_sample_rate: 8000` and `audio_format: "g711_ulaw"` on a G.711 phone-audio session).
`audio.delta`	Assistant audio chunk (base64-encoded PCM16). Play as it arrives.
`text.delta`	Streaming assistant text. Only fires if `output_transcription` is on (or the model emits text natively). With `diarize` on a live transcription model, it carries a `speaker` label (`speaker_0`, `speaker_1`, …).
`transcript.partial`	Live transcription only. The whole hypothesis so far for the current segment, revised as the speaker goes on: replace your pending line with it, don’t append.
`transcript.committed`	User speech transcript, after VAD commits a turn (on voice models only if `input_transcription` is on), or a final segment on live transcription models. Carries `words` when the session asked for word timings (see Word timings).
`speech.started` / `speech.stopped`	VAD signals for the user’s microphone. Use these to interrupt playback when the user barges in.
`speech.eager_end` / `speech.resumed`	Model-based turn detection on live transcription models: the speaker has probably finished (start preparing a reply, but hold it), or carried on after all (drop it). `speech.stopped` confirms the end of the turn.
`response.started` / `response.completed`	Lifecycle markers for an assistant turn.
`tool.call`	Model wants to call a tool. You must reply with a matching `tool.result`.
`session.terminating`	Server is closing the session. Carries a structured reason (see below).
`session.ended`	Final frame before the upstream connection closes.
`error`	Provider or protocol error mid-session.

Termination reasons on session.terminating.error.code: session_timeout, idle_timeout, balance_exhausted, project_spend_cap_hit, org_spend_cap_hit, billing_not_supported.

Live transcription (speech-to-text)

The same WebSocket also powers transcription-only models: audio in, streaming text out, no audio response. Use it for live captions, dictation, call transcription, or as the ears of a voice agent that runs its own LLM and TTS.

Pick a model

List them with GET /v3/models?type=realtime. Transcription models have output_modalities of text only. Each model’s params.realtime says what it takes, so read it from there instead of hard-coding it:

Field	Meaning
`languages`	Language codes the model transcribes.
`input_audio_formats`	Audio formats `input_audio_format` accepts, e.g. `pcm16`, `g711_ulaw`, `g711_alaw`.
`diarize`	The model can label speakers (`diarize: true`).
`semantic_vad`	The model supports model-based turn detection (`turn_detection: { "type": "semantic_vad" }`).
`parameters`	The provider settings it accepts under `config.parameters`, each with `name`, `type`, `description` and, where fixed, `enum`.

Model	Hosted	Partials	Turn detection	Speaker labels
`mistral/voxtral-mini-transcribe-realtime-3-0`	EU	no (text deltas)	yes	yes
`mistral/voxtral-mini-transcribe-realtime-2602`	EU	no (text deltas)	no	no
`gemini/gemini-3.5-transcribe-live`	Google	yes	no	no

Start a session

Request text output and send the settings on session.start. Most transcription models fix their settings when the session connects, so a session.update that changes them is refused:

{
  "type": "session.start",
  "config": {
    "model": "mistral/voxtral-mini-transcribe-realtime-3-0",
    "modalities": ["text"],
    "transcription_language_codes": ["sv-SE"],
    "transcription_custom_vocabulary": ["Opper", "förmaksflimmer"],
    "diarize": true
  }
}

Then stream audio with audio.append exactly as you would for a voice session, and send audio.commit when your own VAD hears silence to get the final transcript without waiting.

Read the transcript

Depending on the model, the in-flight text arrives as transcript.partial (the whole hypothesis so far, which you replace) or as text.delta (fragments you append). Either way, transcript.committed finalizes each segment:

switch (ev.type) {
  case "transcript.partial":    // whole hypothesis so far: replace
    pending = ev.transcript;
    break;
  case "text.delta":            // fragment: append (carries ev.speaker with diarize)
    pending += ev.delta;
    break;
  case "transcript.committed":  // final segment
    lines.push(ev.transcript);
    pending = "";
    break;
}

There is no audio.delta (these models don’t speak), and voice, tools and the audio-output fields don’t apply. Requesting modalities: ["audio"] returns an error.

Turn detection for voice agents

On models with params.realtime.semantic_vad (such as Voxtral 3), turn_detection: { "type": "semantic_vad" } judges the end of a turn from what was said as well as from pauses:

speech.started: the user started a turn.
speech.eager_end: the user has probably finished. Start generating a reply, but hold it.
speech.resumed: they carried on after all. Drop the held reply. Some models send no speech.resumed and signal it with a new transcript.partial instead; treat either as “keep listening”.
transcript.committed then speech.stopped: the turn is over. Play the reply.

eagerness tunes how soon a turn ends where the model supports it; on Voxtral 3, "high" ends turns sooner. Models with their own thresholds list them in params.realtime.parameters.

Speaker labels

diarize: true labels each final segment with its speaker, sent as a text.delta with speaker (speaker_0, speaker_1, …) next to the transcript.committed. It needs a model with params.realtime.diarize (such as Voxtral 3), and can’t be combined with turn detection. A model that can cap the number of speakers lists the setting in params.realtime.parameters.

Phone audio

For audio from a phone line (Twilio, most SIP trunks), set input_audio_format to g711_ulaw or g711_alaw and send the raw 8 kHz mono bytes as they arrive, with no resampling. session.started confirms input_sample_rate: 8000, and billing counts the audio at that rate. Send G.711 only to a model whose params.realtime.input_audio_formats lists it; the others take 16 kHz PCM16.

Word timings

Models that can time each word expose it as a config.parameters setting (look for include_words and include_confidence in params.realtime.parameters). With it on, each transcript.committed carries words, in the same shape /v3/audio/transcriptions returns: start and end are seconds from the start of the session’s audio, and confidence (between 0 and 1) is present when asked for. Partials carry no words.

{
  "type": "transcript.committed",
  "transcript": "Book a table for two.",
  "words": [
    { "word": "Book", "start": 0.2, "end": 0.5, "confidence": 0.98 },
    { "word": "two.", "start": 1.1, "end": 1.5, "confidence": 0.95 }
  ]
}

Billing

Live transcription bills per minute of input audio (audio_input_per_minute on GET /v3/models); there is no output charge. A session that fails to start costs nothing.

Tool calls

Tools follow the same round trip as gateway tool calling: declare them in config.tools with a JSON Schema for parameters. When the model decides to call one, you get a tool.call event. Reply with tool.result and the model continues the conversation. Opper handles the provider wire format internally, so your handler is the same across upstreams.

case "tool.call":
  const result = await runTool(ev.tool_name, ev.tool_arguments);
  ws.send(JSON.stringify({
    type: "tool.result",
    tool_call_id: ev.tool_call_id,
    tool_result: result,
  }));
  break;

While a tool is executing, the upstream model is paused. You’ll hear a silence in the audio stream proportional to your tool’s latency. For tools that take more than a second or two, plan the UX around it with something like a “looking that up…” cue.

Per-provider notes

The protocol is the same across providers, but each upstream has quirks worth knowing.

OpenAI
xAI
Gemini
Mistral (transcription)

Models: openai/gpt-realtime-2, openai/gpt-realtime-1.5, openai/gpt-realtime, openai/gpt-realtime-mini.Voices: alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, cedar (latest models).Sample rate: 24 kHz symmetric (in/out).Reasoning effort: gpt-realtime-2 accepts reasoning_effort and bills reasoning tokens. Other models ignore it.Transcription: input transcription runs in a parallel pipeline upstream, so the user transcript can arrive after the assistant has started responding. Enabling input_transcription adds a separate per-minute bill line (input_transcription_ms) at the rate of the transcription model.Transcription is set-at-start. input_transcription and input_transcription_model cannot be toggled via session.update on OpenAI. Mid-session changes return an error. Request transcription at session.start or not at all. xAI and Gemini fold transcription into their per-minute audio rate and don’t have this restriction.

Models: xai/grok-voice-latest.Voices: ara, eve, leo, rex, sal.Sample rate: 24 kHz symmetric.Billing: flat per-minute rates for audio in / audio out. Input and output transcription are folded into the per-minute rate; toggling them on doesn’t cost extra.Connect latency: xAI requires a three-step handshake (client-secret exchange, WS dial, session.updated). Expect ~1s before session.started lands.

Models: gemini/gemini-3.1-flash-live-preview.Voices: Puck, Charon, Kore, Fenrir, Aoede, Leda, Orus, Zephyr.Sample rate: asymmetric. 16 kHz input, 24 kHz output. Both rates are reported in session.started. Read input_sample_rate for mic capture and output_sample_rate for playback.Tools: supported; translated to Gemini’s toolResponse format internally.Mid-turn close: Gemini reports usage only at turn boundaries. If you close mid-sentence, Opper waits briefly for the trailing usage frame before tearing down.

Models: mistral/voxtral-mini-transcribe-realtime-3-0 (Voxtral 3) and mistral/voxtral-mini-transcribe-realtime-2602 (Voxtral 2), EU-hosted; mistral:global/... twins run on Mistral’s global infrastructure. Transcription only. See Live transcription.Sample rate: 16 kHz input (PCM16). No audio output.Output: streaming text.delta plus a transcript.committed at each segment boundary. Set modalities: ["text"].Voxtral 3 settings: Mistral fixes them when the first audio arrives, so send them on session.start. transcription_language_codes with one code pins that language (sv-SE pins sv); with several, Voxtral detects the language. transcription_custom_vocabulary is Voxtral’s context bias: names and terms, or a sentence describing the audio. turn_detection: { "type": "semantic_vad" } turns on end-of-turn events; eagerness: "high" ends turns sooner, "auto" and "medium" use the defaults, "low" is refused. diarize: true labels each text delta with its speaker. Diarization and turn detection can’t be combined.Voxtral 2 takes none of these settings: it ignores the language and vocabulary hints, and refuses diarize and semantic_vad.Billing: per minute of input audio (audio_input_per_minute); there is no output-audio charge.

Session limits

Limit	Default	Trigger
Concurrent sessions per project	5	`429` at preflight
Max session duration	30 minutes	`session.terminating` with `session_timeout`
Idle timeout (no client traffic)	60 seconds	`session.terminating` with `idle_timeout`
Session-start grace period	10 seconds	WebSocket close if `session.start` never arrives

Billing

Realtime sessions bill incrementally as the conversation runs. Per-model rates are published on GET /v3/models. Look for audio_input, audio_output, audio_input_per_minute, audio_output_per_minute, and input_transcription_per_minute on the relevant model.

Tracing

Each session is listed in Logs with its usage and cost. With tracing on (set in Opper retention), its generation records also appear in the Trace explorer, filterable by model or session_id.

Get started

Platform

Build

Observe & Control

Apps

Tutorials

Tooling

Realtime protocol

How it works

Just trying it out?

Authentication

Server-side: bearer token

Browser: ephemeral tickets

Pre-binding for security

Preflight rejections

Session lifecycle

Configuring the session

Event vocabulary

Client → server

Server → client

Live transcription (speech-to-text)

Pick a model

Start a session

Read the transcript

Turn detection for voice agents

Speaker labels

Phone audio

Word timings

Billing

Tool calls

Per-provider notes

Session limits

Billing

Tracing

See also

​How it works

​Just trying it out?

​Authentication

​Server-side: bearer token

​Browser: ephemeral tickets

​Pre-binding for security

​Preflight rejections

​Session lifecycle

​Configuring the session

​Event vocabulary

​Client → server

​Server → client

​Live transcription (speech-to-text)

​Pick a model

​Start a session

​Read the transcript

​Turn detection for voice agents

​Speaker labels

​Phone audio

​Word timings

​Billing

​Tool calls

​Per-provider notes

​Session limits

​Billing

​Tracing

​See also

How it works

Just trying it out?

Authentication

Server-side: bearer token

Browser: ephemeral tickets

Pre-binding for security

Preflight rejections

Session lifecycle

Configuring the session

Event vocabulary

Client → server

Server → client

Live transcription (speech-to-text)

Pick a model

Start a session

Read the transcript

Turn detection for voice agents

Speaker labels

Phone audio

Word timings

Billing

Tool calls

Per-provider notes

Session limits

Billing

Tracing

See also