> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getpatter.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ultravox Realtime

> Ultravox managed-agent realtime engine — pure WebSocket, no vendor SDK.

# Ultravox Realtime

`UltravoxRealtimeAdapter` bridges a bidirectional audio stream to an [Ultravox](https://www.ultravox.ai/) managed-agent call. The transport is plain WebSocket plus one REST call to create the session — no vendor SDK is required.

It speaks the same `connect / send_audio / receive_events / close` surface as `OpenAIRealtimeAdapter`, so you can swap engines without touching the call handler.

## Install

<CodeGroup>
  ```bash Python theme={null}
  pip install "getpatter[ultravox]"
  ```

  ```bash TypeScript theme={null}
  npm install getpatter
  ```
</CodeGroup>

The TypeScript adapter has no extra peer dependency — it uses the bundled `ws` package.

Set `ULTRAVOX_API_KEY` in your environment.

## Constructor

<CodeGroup>
  ```python Python theme={null}
  from getpatter.providers.ultravox_realtime import (
      UltravoxRealtimeAdapter,
      UltravoxModel,
      UltravoxSampleRate,
  )

  adapter = UltravoxRealtimeAdapter(
      api_key="",                                   # ULTRAVOX_API_KEY when wired
      model=UltravoxModel.FIXIE_AI_ULTRAVOX,        # default
      voice="",                                     # ultravox voice id (optional)
      instructions="You are a helpful, concise voice assistant.",
      language="en",
      sample_rate=UltravoxSampleRate.HZ_16000,      # 8k / 16k / 24k / 48k
      first_message="",                             # if set, agent speaks first
  )
  ```

  ```typescript TypeScript theme={null}
  import { UltravoxRealtimeAdapter } from "getpatter";

  const adapter = new UltravoxRealtimeAdapter(process.env.ULTRAVOX_API_KEY!, {
    model: "fixie-ai/ultravox",
    voice: "",
    instructions: "You are a helpful, concise voice assistant.",
    language: "en",
    sampleRate: 16000,
    firstMessage: "",
  });
  ```
</CodeGroup>

## Usage

Pass the adapter as the `engine` on `phone.agent(...)`:

<CodeGroup>
  ```python Python theme={null}
  import asyncio
  from getpatter import Patter, Twilio
  from getpatter.providers.ultravox_realtime import UltravoxRealtimeAdapter

  phone = Patter(carrier=Twilio(), phone_number="+15550001234")

  agent = phone.agent(
      engine=UltravoxRealtimeAdapter(api_key=""),
      system_prompt="You are a helpful assistant.",
      first_message="Hi! How can I help today?",
  )

  asyncio.run(phone.serve(agent))
  ```

  ```typescript TypeScript theme={null}
  import { Patter, Twilio, UltravoxRealtimeAdapter } from "getpatter";

  const phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234" });

  const agent = phone.agent({
    engine: new UltravoxRealtimeAdapter(process.env.ULTRAVOX_API_KEY!),
    systemPrompt: "You are a helpful assistant.",
    firstMessage: "Hi! How can I help today?",
  });

  await phone.serve({ agent });
  ```
</CodeGroup>

Tools work via `phone.agent(tools=[Tool(...)])` / `tools: [...]`. Patter translates the OpenAI-style JSON schema into Ultravox `dynamicParameters` automatically.

## How it works

1. The adapter `POST`s to `https://api.ultravox.ai/api/calls` to create a call. The response includes a single-use `joinUrl`.
2. The adapter opens that URL as a WebSocket. From here:
   * **Binary frames** are PCM16 mono audio (both directions).
   * **Text frames** are JSON control events (transcripts, tool invocations, state).

`firstSpeaker` and `initialMessages` are mutually exclusive on the Ultravox API. When `first_message` / `firstMessage` is set, Patter sends an `initialMessages` agent turn; otherwise the user speaks first.

## Sample rates

`UltravoxSampleRate` accepts 8000, 16000, 24000, or 48000 Hz. Default is 16000 — what Patter's pipeline-mode audio bus uses internally.

## When to use Ultravox vs alternatives

| Use Ultravox when…                                                                             | Use [OpenAI Realtime](/python-sdk/providers/openai-realtime) when… | Use [Gemini Live](/python-sdk/providers/gemini-live) when… |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------- |
| You want a managed agent with its own voice library and no vendor SDK in your dependency tree. | You need the broadest tool-calling ecosystem and reasoning tiers.  | You want native-audio Gemini voices and 1M+ context.       |

## Notes

* The adapter implements `cancel_response()` / `cancelResponse()` by sending `playback_clear_buffer`, which interrupts the agent's current turn for clean barge-in.
* The Python adapter uses `aiohttp`; the TS adapter uses `ws` plus the platform `fetch`.
* The receive loop yields `("audio", bytes)` for agent audio, `("transcript_input", str)` / `("transcript_output", str)` for transcripts, and `("function_call", {...})` for tool calls.

## What's Next

<CardGroup cols={2}>
  <Card title="Engines" icon="bolt" href="/python-sdk/engines">All engines side by side.</Card>
  <Card title="OpenAI Realtime" icon="bolt" href="/python-sdk/providers/openai-realtime">The default engine.</Card>
  <Card title="Gemini Live" icon="google" href="/python-sdk/providers/gemini-live">Google's native-audio realtime API.</Card>
  <Card title="Tools" icon="screwdriver-wrench" href="/python-sdk/tools">Function calling inside a realtime session.</Card>
</CardGroup>