Audio

The gateway covers audio with three endpoints:

Text to speech: POST /v3/audio/speech turns text into spoken audio.
Speech to text: POST /v3/audio/transcriptions turns audio into a transcript.
Music and sound effects: POST /v3/audio/generations turns a prompt into a score, a song, a sound effect or a loop of ambience. See Music and sound effects.

All are provider-agnostic: model selects the provider, a small set of fields is normalized for you, and anything model-specific goes in parameters. You can also clone a custom voice from a short sample and reuse it. For live, two-way voice conversations, use Realtime instead.

Text to speech

Pass a model and the input text. The response carries the audio inline as base64. Add "store": true to also keep a copy in Files and get a file_id + presigned url.

curl -sX POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/tts-1",
    "input": "Hello from Opper.",
    "voice": "alloy",
    "format": "mp3"
  }'

import os, base64, requests

r = requests.post(
    "https://api.opper.ai/v3/audio/speech",
    headers={"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"},
    json={
        "model": "openai/tts-1",
        "input": "Hello from Opper.",
        "voice": "alloy",
        "format": "mp3",
    },
)
r.raise_for_status()
audio = r.json()["audio"]
open("speech.mp3", "wb").write(base64.b64decode(audio["b64_json"]))
print("saved speech.mp3")

Response

{
  "id": "spch_...",
  "model": "openai/tts-1",
  "created": 1716124800,
  "audio": {
    "b64_json": "SUQzBAAAAAA...",
    "mime_type": "audio/mpeg"
  },
  "usage": { "cost": 0.0009, "characters": 17 }
}

Field	What it does
`model`	Required. The TTS model, e.g. `openai/tts-1`.
`input`	Required. The text to speak.
`voice`	Provider voice; empty uses the model’s default. See Finding voices.
`format`	`mp3` (default), `wav`, `opus`, `aac`, `flac`, or `pcm`.
`speed`	`0.25`–`4.0` where the model supports it.
`store`	Persist the audio and return a `file_id` + `url`. Defaults to false; a stored file is permanent until deleted. Refused on zero-data-retention projects.
`parameters`	Opaque per-provider passthrough.

Speech is billed per input character.

Finding voices

GET /v3/audio/voices?model=<model> lists every voice a text-to-speech model accepts as voice: the provider’s built-in voices (type: preset) first, then your project’s cloned voices on the same provider (type: custom). Pass the id of any entry as voice. For ElevenLabs and Mistral, the built-in voices are read from the provider itself, so a voice they add shows up without any change on our side. Each comes with a name, gender, accent and languages, and ElevenLabs voices carry a preview_url you can listen to before you pick. For other providers, the list is the voices the model declares. meta.presets_source says which one you got (provider or catalog).

curl -s "https://api.opper.ai/v3/audio/voices?model=elevenlabs/eleven_v3&gender=female" \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  | jq '.data[] | {id, name, accent, languages, preview_url}'

{
  "id": "hpp4J3VqNfWAUOO0d1Us",
  "name": "Bella - Professional, Bright, Warm",
  "accent": "american",
  "languages": ["en"],
  "preview_url": "https://..."
}

Filter	What it does
`model`	Required for this listing. A text-to-speech model id.
`type`	`preset` for built-in voices only, `custom` for your clones only.
`language`	A language code such as `de` or `en-GB`. `en` also matches `en-US` and `en-GB`.
`gender`	As the provider labels it, for example `female` or `male`.
`limit`, `offset`	Paging. Up to 1000 per page.

The entry marked "default": true is the voice used when you leave voice out.

Provider	Notes
OpenAI, Gemini, xAI	A short fixed list. A voice outside it returns a `400`.
Azure Speech	The voice name selects both the speaker and the locale, for example `en-GB-SoniaNeural`, so `language=fr-CA` finds every French (Canada) voice. Standard voices are on `azure-speech/neural-tts`, HD voices on `azure-speech/neural-hd-tts`.
Mistral Voxtral	Presets such as `en_paul_neutral` and `gb_jane_neutral`, in several moods per speaker.
ElevenLabs	ElevenLabs’ built-in voices. Voices in your own ElevenLabs account are not available here, because Opper calls ElevenLabs with its own account: clone the voice instead.
Fish Audio	Not listed. Pass a Fish voice id as `voice`; names are not accepted. Browse voices with audio samples and ids in the Fish voice library. Set a voice explicitly for consistent results.

Voices for a specific language

Filter by language to find voices made for it. This lists every French (Canada) voice on Azure, and works the same for any locale such as de-AT, sv-SE or ja-JP:

curl -s "https://api.opper.ai/v3/audio/voices?model=azure-speech/neural-tts&language=fr-CA" \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  | jq -r '.data[].id'

Most multilingual models will also read text in another language with any voice, but with a foreign accent. For native-sounding speech, pick a voice made for that language or regional variety.

Expressiveness

A voice can sound flat by default. How you add expression depends on the provider:

ElevenLabs exposes voice controls through parameters.voice_settings: lower stability for a more dynamic, emotional read, higher style to exaggerate delivery, and use_speaker_boost. The elevenlabs/eleven_v3 model also understands inline audio tags in the input text — [excited], [whispers], [sarcastic], [laughs] — for fine-grained emotion.
Mistral Voxtral has no style knobs; it mirrors the delivery of the voice it speaks as. For a lively result, use — or clone — a lively-sounding voice.

curl -sX POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "elevenlabs/eleven_multilingual_v2",
    "input": "We did it — the results are in!",
    "voice": "voice_abc123",
    "parameters": { "voice_settings": { "stability": 0.3, "style": 0.8, "use_speaker_boost": true } }
  }'

curl -sX POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "elevenlabs/eleven_v3",
    "input": "[excited] We did it! [whispers] and no one even noticed.",
    "voice": "voice_abc123"
  }'

Audio tags count toward billed input characters, and eleven_v3 bills at a higher rate — great for a hero voice, more than you need for bulk narration. Use GET /v3/audio/models to discover the exact parameters a model accepts (its params.tts.parameters list).

Custom voices (cloning)

Clone a voice from a short reference sample and reuse it by an opaque id. POST /v3/audio/voices takes a model (which selects the provider to clone on), an audio reference — a file_id, https URL, or base64 data-URI of a few seconds of speech — and an optional name, languages, and gender. It returns a voice_... id you pass as voice on /v3/audio/speech, exactly like a built-in voice.

# 1. Clone — returns { "id": "voice_...", "provider": "mistral", ... }
curl -sX POST https://api.opper.ai/v3/audio/voices \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "mistral/voxtral-mini-tts-2603",
    "name": "my-voice",
    "audio": "file_abc123",
    "languages": ["de"]
  }'

# 2. Speak with it — pass the voice_... id as `voice`
curl -sX POST https://api.opper.ai/v3/audio/speech \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{ "model": "mistral/voxtral-mini-tts-2603", "input": "Hallo!", "voice": "voice_..." }'

import os, requests

base = "https://api.opper.ai"
h = {"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"}

voice = requests.post(f"{base}/v3/audio/voices", headers=h, json={
    "model": "mistral/voxtral-mini-tts-2603",
    "name": "my-voice",
    "audio": "file_abc123",   # or an https URL / data-URI
    "languages": ["de"],
}).json()

speech = requests.post(f"{base}/v3/audio/speech", headers=h, json={
    "model": "mistral/voxtral-mini-tts-2603",
    "input": "Hallo!",
    "voice": voice["id"],     # voice_...
}).json()

Response (create)

{
  "id": "voice_033Qq...",
  "object": "voice",
  "name": "my-voice",
  "provider": "mistral",
  "languages": ["de"],
  "created": 1716124800,
  "expires_at": 1718716800
}

Manage your voices — all scoped to your project:

Method	Path	What it does
`GET`	`/v3/audio/voices`	List your project’s cloned voices (newest first, paginated). Add `?model=` to also list that model’s built-in voices, see Finding voices.
`GET`	`/v3/audio/voices/{id}`	Fetch one voice.
`DELETE`	`/v3/audio/voices/{id}`	Remove it — also deletes it at the provider.

Field (create)	What it does
`model`	Required. Selects the provider to clone on (e.g. `mistral/voxtral-mini-tts-2603`, `elevenlabs/eleven_multilingual_v2`).
`audio`	Required. The reference sample: a `file_id`, https URL, or base64 data-URI.
`name`	A human label for the voice.
`languages`	Language hints, e.g. `["de"]`.
`gender`	Optional voice gender hint.
`parameters`	Opaque per-provider passthrough for create-voice fields we don’t normalize.

A clone carries the voice, not a language. Cloning captures timbre and accent; the output language comes from the text you synthesize with a multilingual model — so one clone can speak any supported language. Recording in the target language gives the most native accent.

A few things to know:

Provider binding. A voice is bound to the provider you cloned it on and only works with that provider’s TTS models. Using it with another provider’s model returns a 400.
Isolation & privacy. Voices are scoped to your project — never visible to or usable by another project — and the reference sample is not stored.
Expiry. Some providers remove clones after a while. When they do, expires_at is returned. Treat it as the provider’s estimate, not a deadline: it is the earliest date the provider may remove the voice, and a voice often keeps working well past it. A voice stays in your list for as long as the provider still has it. Once the provider has removed it, it drops out of the list and a speak returns a clear “no longer available” error. Create a new one from the same sample. Providers that don’t expire voices omit expires_at.
Sample length. A few seconds is enough for Mistral Voxtral (zero-shot). ElevenLabs works best with ~1–2 minutes of clean audio (avoid more than ~3 minutes); recording quality matters more than length.

Speech to text

Pass a model and an audio source: a file_id from a previous generation or upload, an https URL, or a data-URI (max 25 MB decoded synchronously; up to 100 MB in async mode).

A file_id lets you transcribe audio you already have on Opper, such as a clip you uploaded or speech you generated with "store": true, without re-sending the bytes. See Files for how files work, lifecycle, and storage quotas.

Transcribe a file_id

curl -sX POST https://api.opper.ai/v3/audio/transcriptions \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{ "model": "openai/whisper-1", "audio": "file_abc123" }'

curl -sX POST https://api.opper.ai/v3/audio/transcriptions \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/whisper-1",
    "audio": "https://example.com/meeting.mp3",
    "language": "en"
  }'

import os, requests

r = requests.post(
    "https://api.opper.ai/v3/audio/transcriptions",
    headers={"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"},
    json={
        "model": "openai/whisper-1",
        "audio": "https://example.com/meeting.mp3",
        "language": "en",
    },
)
r.raise_for_status()
print(r.json()["text"])

Response

{
  "id": "trsc_...",
  "model": "openai/whisper-1",
  "created": 1716124800,
  "text": "Full transcript here...",
  "language": "en",
  "duration": 42.5,
  "segments": [
    { "start": 0.0, "end": 3.2, "text": "Full transcript here..." }
  ],
  "usage": { "cost": 0.004, "seconds": 42.5 }
}

Field	What it does
`model`	Required. The transcription model, e.g. `openai/whisper-1`.
`audio`	Required. A `file_id`, https URL, or data-URI to transcribe.
`language`	ISO-639-1 hint (e.g. `en`); the provider validates it.
`prompt`	Context or vocabulary hint to bias the transcript.
`diarize`	Request speaker labels on segments. Rejected on models that don’t support diarization.
`stream`	Return the transcript live as Server-Sent Events. Supported where `params.stt.stream` is true; a 400 otherwise. Mutually exclusive with `async`. See below.
`async`	Run as a background job for long recordings — returns `202` with a status URL to poll instead of blocking. See below.
`parameters`	Opaque per-provider passthrough.

Transcription is billed per provider-reported audio duration. Segment and word timestamps are returned when the model provides them.

Live transcript (streaming)

Set stream: true to receive the transcript as it’s produced — as Server-Sent Events instead of one JSON body — for a live transcript while the audio is processed. The stream emits transcript.text.delta chunks, then a terminal transcript.text.done with the full transcript and usage, then data: [DONE]. Streaming is supported on models whose params.stt.stream is true (discover them via GET /v3/audio/models) — e.g. mistral/voxtral-mini-2602; other models return a 400. stream and async are mutually exclusive.

# -N keeps curl from buffering, so events print as they arrive
curl -N -sX POST https://api.opper.ai/v3/audio/transcriptions \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{ "model": "mistral/voxtral-mini-2602", "audio": "file_abc123", "stream": true }'

import os, json, requests

with requests.post(
    "https://api.opper.ai/v3/audio/transcriptions",
    headers={"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"},
    json={"model": "mistral/voxtral-mini-2602", "audio": "file_abc123", "stream": True},
    stream=True,
) as r:
    for line in r.iter_lines():
        if not line or not line.startswith(b"data: "):
            continue
        data = line[len(b"data: "):]
        if data == b"[DONE]":
            break
        event = json.loads(data)
        if event["type"] == "transcript.text.delta":
            print(event["delta"], end="", flush=True)
        elif event["type"] == "transcript.text.done":
            print("\n\n", event["usage"])

Event stream

data: {"type":"transcript.text.delta","delta":"Good morning and "}
data: {"type":"transcript.text.delta","delta":"welcome!"}
data: {"type":"transcript.text.done","id":"trsc_...","text":"Good morning and welcome!","language":"en","duration":101,"usage":{"cost":0.00505,"seconds":101}}
data: [DONE]

Billing is identical to the synchronous call (per provider-reported audio duration). For long recordings that don’t need a live transcript, use async instead; for two-way live voice, use Realtime.

Long recordings (async)

A synchronous request is a good fit up to roughly an hour of audio. For longer recordings, set async: true: the request returns immediately with a 202 and a status_url. Poll it until the job is completed, then fetch the transcript from the returned url. Async accepts the same audio sources as the synchronous call (file_id, https URL, or data-URI). Async also raises the decoded-audio limit from 25 MB to 100 MB (the synchronous cap stays 25 MB). To reach the full 100 MB, pass a file_id or https URL. A request body is limited to 100 MiB, and base64 adds about a third, so an inline data-URI carries at most about 75 MB of audio. The 100 MB limit lines up with the Files per-file limit, so a stored file_id transcribes in full. An inline data-URI in async mode is held in Opper’s storage in Stockholm until the job starts, then deleted. The transcript is kept for up to 30 days; each status call returns a fresh download link valid for 1 hour.

Submit + poll

# 1. Submit — returns { "id": "gen_...", "status_url": ".../v3/artifacts/{id}/status" }
curl -sX POST https://api.opper.ai/v3/audio/transcriptions \
  -H "Authorization: Bearer $OPPER_API_KEY" -H "Content-Type: application/json" \
  -d '{ "model": "mistral/voxtral-mini-2602", "audio": "file_abc123", "diarize": true, "async": true }'

# 2. Poll — 202 while processing, then 200 with a presigned url to the transcript JSON
curl -s https://api.opper.ai/v3/artifacts/{id}/status \
  -H "Authorization: Bearer $OPPER_API_KEY"

mistral/voxtral-mini-2602 is an EU-hosted batch transcription model (speaker diarization, word timestamps, up to 3 h per request) — a good fit for the async flow.

Music and sound effects

POST /v3/audio/generations generates audio from a prompt with any model tagged music or sound. It runs synchronously and returns the audio inline as base64, like speech. The same request also answers on POST /v3/audio/music.

A sound effect

Describe one action per request, as the material and distance you want to hear, and set duration_seconds to the length of that action.

curl -sX POST https://api.opper.ai/v3/audio/generations \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "elevenlabs/eleven_text_to_sound_v2",
    "prompt": "a single hammer strike on an iron anvil, close, dry",
    "duration_seconds": 1
  }'

import os, base64, requests

r = requests.post(
    "https://api.opper.ai/v3/audio/generations",
    headers={"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"},
    json={
        "model": "elevenlabs/eleven_text_to_sound_v2",
        "prompt": "a single hammer strike on an iron anvil, close, dry",
        "duration_seconds": 1,
    },
)
r.raise_for_status()
open("anvil.mp3", "wb").write(base64.b64decode(r.json()["audio"]["b64_json"]))

Response

{
  "id": "snd_...",
  "model": "elevenlabs/eleven_text_to_sound_v2",
  "created": 1790781677,
  "audio": { "b64_json": "SUQzBAAAAAA...", "mime_type": "audio/mpeg" },
  "usage": { "cost": 0.002, "duration_seconds": 1 }
}

For ambience and hums, add "loop": true on a model with the audio_loop capability: the result repeats without a seam, so you can tile it under a whole scene.

Music with timed sections

A plain prompt with duration_seconds gives you a track. When the music has to fit a picture, send sections instead: a timed plan where each section has its own duration_ms, prompt and styles. The sections set the total length. This needs a model with the audio_sections capability.

curl -sX POST https://api.opper.ai/v3/audio/generations \
  -H "Authorization: Bearer $OPPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "elevenlabs/music_v2_5",
    "sections": [
      {
        "duration_ms": 4000,
        "prompt": "120 BPM, playful marimba intro, sound from the first beat",
        "styles": ["marimba", "instrumental"]
      },
      {
        "duration_ms": 4000,
        "prompt": "the same instruments, fuller, ending on a resolved chord",
        "styles": ["marimba", "pizzicato strings", "instrumental"]
      }
    ]
  }'

import os, base64, requests

r = requests.post(
    "https://api.opper.ai/v3/audio/generations",
    headers={"Authorization": f"Bearer {os.environ['OPPER_API_KEY']}"},
    json={
        "model": "elevenlabs/music_v2_5",
        "sections": [
            {
                "duration_ms": 4000,
                "prompt": "120 BPM, playful marimba intro, sound from the first beat",
                "styles": ["marimba", "instrumental"],
            },
            {
                "duration_ms": 4000,
                "prompt": "the same instruments, fuller, ending on a resolved chord",
                "styles": ["marimba", "pizzicato strings", "instrumental"],
            },
        ],
    },
)
r.raise_for_status()
open("score.mp3", "wb").write(base64.b64decode(r.json()["audio"]["b64_json"]))

A section’s prompt describes the music and is never sung. Words to be sung go only in that section’s lyrics; leave it out for instrumental music. negative_styles lists what a section should avoid, for example ["vocals"]. Some models take either a top-level prompt or sections, and put instrumental in the section styles rather than the top-level field; they answer 400 saying so. On ElevenLabs music, seed works only with sections: a prompt with a seed answers 400.

Field	What it does
`model`	Required. A `music` or `sound` model from Discover models.
`prompt`	What to generate. Required unless `sections` are sent.
`duration_seconds`	Requested length, up to 600. Omit it to let the model choose; not sent with `sections`.
`sections`	A timed plan: `duration_ms`, `prompt`, `styles`, `negative_styles`, `lyrics` per section. Needs `audio_sections`.
`loop`	Seamless repeat for ambience. Needs `audio_loop`.
`instrumental`	Music only: no vocals.
`format`	`mp3`, `wav`, `opus` or `pcm`. Omit it for the model’s native format.
`seed`	For more repeatable output, where the model supports it. ElevenLabs music takes it only with `sections`, and ElevenLabs sound effects not at all.
`async`	Run in the background and poll a `status_url` (202), for long tracks. Works like long transcriptions.
`store`	Also save the result to Files and return a `file_id`.
`parameters`	Provider fields, forwarded as-is in the provider’s own names.

A value a model cannot honor (a length, a format, sections or loop on a model without the capability) answers 400 naming the limit, rather than being ignored. parameters is never checked by Opper; the provider validates it. Music and sound effects are billed per minute of audio or per generation, depending on the model; the model entry’s pricing says which.

Getting usable results

One action per sound effect. “A hammer tap” and “a saw stroke” are two requests; mix them yourself.
Put section boundaries on the tempo’s phrase grid. At 120 BPM, multiples of 2 or 4 seconds. Boundaries between phrases get moved to the nearest phrase.
Keep one instrument palette across sections and vary the intensity. Changing instruments between sections sounds like a different track.
Do silences, drops and breakdowns in your mix, not in the prompt. Models often play through a section that asks for “drums out”.
Measure what you got (length, loudness per second) before cutting picture to it.

Discover models

GET /v3/audio/models lists the audio models available, tagged tts, stt, music or sound, with their voices, formats and capabilities. Filter with ?type=music or ?type=sound, or by capability with ?capability=audio_sections or ?capability=audio_loop. Each model’s params.tts / params.stt block also carries a parameters list of the native passthrough keys it accepts (e.g. ElevenLabs’ exact output_format preset, or a language_code), so you can discover the model-specific knobs for parameters rather than guessing:

# all audio models, or filter by type
curl -s "https://api.opper.ai/v3/audio/models?type=tts" \
  -H "Authorization: Bearer $OPPER_API_KEY"

What’s next

Realtime voice

Live, two-way voice over WebSocket.

Models

Which models do speech, transcription, music and sound effects.

Observe & Control

Inspect calls and manage provider, region, and spend rules.

Video

Generate video from a prompt or image.

Get started

Platform

Build

Observe & Control

Apps

Tutorials

Tooling

Text to speech

Finding voices

Voices for a specific language

Expressiveness

Custom voices (cloning)

Speech to text

Live transcript (streaming)

Long recordings (async)

Music and sound effects

A sound effect

Music with timed sections

Discover models

What’s next

Realtime voice

Models

Observe & Control

Video

​Text to speech

​Finding voices

​Voices for a specific language

​Expressiveness

​Custom voices (cloning)

​Speech to text

​Live transcript (streaming)

​Long recordings (async)

​Music and sound effects

​A sound effect

​Music with timed sections

​Discover models

​What’s next

Realtime voice

Models

Observe & Control

Video

Text to speech

Finding voices

Voices for a specific language

Expressiveness

Custom voices (cloning)

Speech to text

Live transcript (streaming)

Long recordings (async)

Music and sound effects

A sound effect

Music with timed sections

Discover models

What’s next