> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getpatter.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ultravox Realtime

> Ultravox managed-agent realtime engine — pure WebSocket, no vendor SDK.

# Ultravox Realtime

`UltravoxRealtimeAdapter` bridges a bidirectional audio stream to an [Ultravox](https://www.ultravox.ai/) managed-agent call. The transport is plain WebSocket plus one REST call to create the session — no vendor SDK is required.

It speaks the same `connect / sendAudio / onEvent / close` surface as `OpenAIRealtimeAdapter`, so you can swap engines without touching the call handler.

## Install

<CodeGroup>
  ```bash TypeScript theme={null}
  npm install getpatter
  ```

  ```bash Python theme={null}
  pip install "getpatter[ultravox]"
  ```
</CodeGroup>

The TypeScript adapter has no extra peer dependency — it uses the bundled `ws` package.

Set `ULTRAVOX_API_KEY` in your environment.

## Constructor

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { UltravoxRealtimeAdapter } from "getpatter";

  const adapter = new UltravoxRealtimeAdapter(process.env.ULTRAVOX_API_KEY!, {
    model: "fixie-ai/ultravox",                  // default
    voice: "",                                   // ultravox voice id (optional)
    instructions: "You are a helpful, concise voice assistant.",
    language: "en",
    sampleRate: 16000,                           // 8k / 16k / 24k / 48k
    firstMessage: "",                            // if set, agent speaks first
  });
  ```

  ```python Python theme={null}
  from getpatter.providers.ultravox_realtime import UltravoxRealtimeAdapter

  adapter = UltravoxRealtimeAdapter(
      api_key="",
      model="fixie-ai/ultravox",
      voice="",
      instructions="You are a helpful, concise voice assistant.",
      language="en",
      sample_rate=16000,
      first_message="",
  )
  ```
</CodeGroup>

## Usage

Pass the adapter as the `engine` on `phone.agent({...})`:

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { Patter, Twilio, UltravoxRealtimeAdapter } from "getpatter";

  const phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234" });

  const agent = phone.agent({
    engine: new UltravoxRealtimeAdapter(process.env.ULTRAVOX_API_KEY!),
    systemPrompt: "You are a helpful assistant.",
    firstMessage: "Hi! How can I help today?",
  });

  await phone.serve({ agent });
  ```

  ```python Python theme={null}
  import asyncio
  from getpatter import Patter, Twilio
  from getpatter.providers.ultravox_realtime import UltravoxRealtimeAdapter

  phone = Patter(carrier=Twilio(), phone_number="+15550001234")

  agent = phone.agent(
      engine=UltravoxRealtimeAdapter(api_key=""),
      system_prompt="You are a helpful assistant.",
      first_message="Hi! How can I help today?",
  )

  asyncio.run(phone.serve(agent))
  ```
</CodeGroup>

Tools work via `phone.agent({ tools: [...] })`. Patter translates the OpenAI-style JSON schema into Ultravox `dynamicParameters` automatically.

## How it works

1. The adapter `POST`s to `https://api.ultravox.ai/api/calls` to create a call. The response includes a single-use `joinUrl`.
2. The adapter opens that URL as a WebSocket. From here:
   * **Binary frames** are PCM16 mono audio (both directions).
   * **Text frames** are JSON control events (transcripts, tool invocations, state).

`firstSpeaker` and `initialMessages` are mutually exclusive on the Ultravox API. When `firstMessage` is set, Patter sends an `initialMessages` agent turn; otherwise the user speaks first.

## Sample rates

`sampleRate` accepts 8000, 16000, 24000, or 48000 Hz. Default is 16000 — what Patter's pipeline-mode audio bus uses internally.

## When to use Ultravox vs alternatives

| Use Ultravox when…                                                                             | Use [OpenAI Realtime](/typescript-sdk/providers/openai-realtime) when… | Use [Gemini Live](/typescript-sdk/providers/gemini-live) when… |
| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------- |
| You want a managed agent with its own voice library and no vendor SDK in your dependency tree. | You need the broadest tool-calling ecosystem and reasoning tiers.      | You want native-audio Gemini voices and 1M+ context.           |

## Notes

* The adapter implements `cancelResponse()` by sending `playback_clear_buffer`, which interrupts the agent's current turn for clean barge-in.
* The TS adapter uses `ws` plus the platform `fetch`; the Python adapter uses `aiohttp`.
* Register handlers via `adapter.onEvent((type, data) => ...)`. Event types: `audio`, `transcript_input`, `transcript_output`, `function_call`, `speech_started`, `response_done`, `error`.

## What's Next

<CardGroup cols={2}>
  <Card title="Engines" icon="bolt" href="/typescript-sdk/engines">All engines side by side.</Card>
  <Card title="OpenAI Realtime" icon="bolt" href="/typescript-sdk/providers/openai-realtime">The default engine.</Card>
  <Card title="Gemini Live" icon="google" href="/typescript-sdk/providers/gemini-live">Google's native-audio realtime API.</Card>
  <Card title="Tools" icon="screwdriver-wrench" href="/typescript-sdk/tools">Function calling inside a realtime session.</Card>
</CardGroup>