xAI Realtime

XaiRealtime selects xAI’s Grok Voice Agent API (wss://api.x.ai/v1/realtime) as an end-to-end speech-to-speech engine. Pass it as the engine on phone.agent({...}) and Patter wires the audio stream straight through — no separate STT or TTS. xAI’s Voice Agent API is OpenAI-Realtime-GA-compatible: the same session.update / response.create / streaming-event flow works after swapping the base URL, the API key, and the model. Under the hood XaiRealtime reuses Patter’s GA realtime adapter and grafts on the xAI-specific session knobs (reasoning effort, language hint, keyterms, output speed, pronunciation replacements, session resumption, and server-side tools), so every feature gate in the call handler — barge-in, tool calling, heartbeat, transcode — fires for xAI with no per-provider branches.

Beta. The engine is validated against the xAI Voice Agent API spec; it has not yet been exercised against a live phone call.

Install

npm install getpatter

pip install "getpatter[xai]"

Set XAI_API_KEY in your environment (or pass apiKey).

Constructor

import { XaiRealtime } from "getpatter";

const engine = new XaiRealtime({
  model: "grok-voice-latest",       // default — always the newest voice model
  voice: "eve",                     // default
  reasoningEffort: "none",          // "high" (default) enables reasoning; "none" disables it
  languageHint: "en",               // bias input transcription (BCP-47)
  speed: 1.0,                       // assistant playback speed [0.7, 1.5]
});

from getpatter import XaiRealtime

engine = XaiRealtime(
    model="grok-voice-latest",       # default — always the newest voice model
    voice="eve",                     # default
    reasoning_effort="none",         # "high" (default) enables reasoning; "none" disables it
    language_hint="en",              # bias input transcription (BCP-47)
    speed=1.0,                       # assistant playback speed [0.7, 1.5]
)

Usage

Pass the engine as the engine on phone.agent({...}):

// npx tsx example.ts
import { Patter, Twilio, XaiRealtime } from "getpatter";

const phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234" });

const agent = phone.agent({
  engine: new XaiRealtime({ voice: "eve" }),               // XAI_API_KEY from env
  systemPrompt: "You are a friendly receptionist.",
  firstMessage: "Hello! How can I help?",
});

await phone.serve({ agent });

import asyncio
from getpatter import Patter, Twilio, XaiRealtime

phone = Patter(carrier=Twilio(), phone_number="+15550001234")

agent = phone.agent(
    engine=XaiRealtime(voice="eve"),                       # XAI_API_KEY from env
    system_prompt="You are a friendly receptionist.",
    first_message="Hello! How can I help?",
)

asyncio.run(phone.serve(agent))

Models

Pass a versioned name to pin a specific release; grok-voice-latest always points at the newest voice model.

Model	Notes
`grok-voice-latest` (default)	Alias — currently resolves to `grok-voice-think-fast-1.0`. Use it so your app tracks the recommended model.
`grok-voice-think-fast-1.0`	Flagship voice model. Reasoning is on by default (`reasoningEffort: "high"`).
`grok-voice-fast-1.0`	Legacy model — deprecated. Available for existing integrations; new apps should use `grok-voice-latest`.

Session options

All optional with safe defaults; unset knobs are omitted from the wire so xAI applies its own server defaults.

TypeScript	Python	Default	Notes
`apiKey`	`api_key`	—	Reads `XAI_API_KEY` when omitted.
`model`	`model`	`"grok-voice-latest"`	See Models.
`voice`	`voice`	`"eve"`	Built-in or custom voice ID.
`reasoningEffort`	`reasoning_effort`	—	`"high"` enables reasoning (xAI server default); `"none"` disables it for lower latency.
`vadThreshold`	`vad_threshold`	—	Server VAD activation threshold `[0.1, 0.9]` (xAI default `0.85`). Higher requires louder audio to trigger.
`silenceDurationMs`	`silence_duration_ms`	—	Trailing silence (ms) the VAD waits for before ending the user’s turn.
`prefixPaddingMs`	`prefix_padding_ms`	—	Audio (ms) captured before detected speech start (xAI default `333`).
`idleTimeoutMs`	`idle_timeout_ms`	—	When set, the server re-engages the user after this many ms of silence following an assistant turn.
`languageHint`	`language_hint`	—	BCP-47 code biasing input transcription toward one language (e.g. `"es-MX"`).
`keyterms`	`keyterms`	`[]` / `()`	Up to 100 bias terms (≤50 chars each) for transcription.
`speed`	`speed`	—	Assistant playback speed multiplier `[0.7, 1.5]`.
`replace`	`replace`	—	Pronunciation map applied before TTS, e.g. `{ "Acme Mobile": "Acme Mobull" }`. Only the spoken audio changes; the transcript keeps the original.
`resumption`	`resumption`	`false`	Opt in to session resumption — the server caches conversation turns and replays them on reconnect.
`serverTools`	`server_tools`	`[]` / `()`	Raw xAI server-side tool objects — see Server-side tools.

Tool calling

Function tools declared on phone.agent({ tools: [...] }) work exactly as they do on OpenAI Realtime — Patter’s tool bridge forwards the model’s function calls and returns your results. The built-in transfer_call and end_call tools are auto-injected into every agent, so a Grok voice agent can hand off or hang up out of the box.

Server-side tools

serverTools / server_tools passes raw xAI tool objects through to session.tools verbatim — web_search, x_search, mcp, and file_search. These execute server-side at xAI; the client never handles their results, and xAI bills them separately from Patter’s per-minute metering.

const engine = new XaiRealtime({
  serverTools: [
    { type: "web_search" },
    { type: "x_search" },
    { type: "mcp", server_url: "https://mcp.example.com/mcp", server_label: "my-tools" },
  ],
});

engine = XaiRealtime(
    server_tools=(
        {"type": "web_search"},
        {"type": "x_search"},
        {"type": "mcp", "server_url": "https://mcp.example.com/mcp", "server_label": "my-tools"},
    ),
)

Pronunciation replacements

Fix how the model pronounces brand names or domain terms with replace / replace — matched case-insensitively in the model’s output and swapped before TTS, so only the spoken audio changes:

const engine = new XaiRealtime({
  replace: { "Acme Mobile": "Acme Mobull" },
});

engine = XaiRealtime(
    replace={"Acme Mobile": "Acme Mobull"},
)

Telephony audio

Over Twilio / Telnyx the engine inherits Patter’s GA-compatible audio path: it negotiates PCM-16-LE @ 24 kHz with xAI and transcodes to / from the carrier’s μ-law 8 kHz internally — you don’t configure any of this.

xAI also supports native G.711 (audio/pcmu @ 8 kHz), which would let telephony calls skip the transcode entirely. That is a flagged future optimization, not yet wired into this engine.

When to use xAI Realtime vs alternatives

Use xAI Realtime when…	Use OpenAI Realtime when…	Use Pipeline mode when…
You want Grok voices, on-by-default reasoning, and built-in server-side `web_search` / `x_search`.	You need the broadest tool-calling ecosystem and OpenAI’s GA reasoning tiers.	You need provider-by-provider control (e.g. `DeepgramSTT` + `AnthropicLLM` + `XaiTTS`).

Rates

xAI bills the Grok Voice Agent per minute of session audio (provider key xai_realtime), not per token:

Item	Rate
Voice Agent session audio	$0.05 / min ($ 3.00 / hr)
Text input messages	$0.004 per `conversation.item.create` — documented, not metered by Patter metrics v1

Override the metered rate per-project via new Patter({ pricing: { xai_realtime: { price: ... } } }). See Metrics for the full rate table.

Notes

Beta — the engine is spec-validated, not yet live-call-validated. Pin a versioned model (grok-voice-think-fast-1.0) in production for stability.
Caller transcript display (known Beta limitation). The adapter sets xAI’s input-transcription model (grok-transcribe), and xAI emits conversation.item.input_audio_transcription.updated events rather than OpenAI’s .completed. This first Beta release does not yet consume the .updated variant, so the caller-side transcript does not populate the transcript display. Audio, model responses, barge-in, and tool calling are unaffected — and the assistant-side transcript still flows; full caller-transcript support is a follow-up.
Reasoning is enabled by default. Set reasoningEffort: "none" to disable it for lower latency on simple flows.
serverTools results are billed by xAI and never surface to your client code.

What’s Next

Engines

All engines side by side.

OpenAI Realtime

The default engine.

Agents

System prompts, tools, first messages.

Tools

Function calling inside a realtime session.

​xAI Realtime

​Install

​Constructor

​Usage

​Models

​Session options

​Tool calling

​Server-side tools

​Pronunciation replacements

​Telephony audio

​When to use xAI Realtime vs alternatives

​Rates

​Notes

​What’s Next

Engines

OpenAI Realtime

Agents

Tools

xAI Realtime

Install

Constructor

Usage

Models

Session options

Tool calling

Server-side tools

Pronunciation replacements

Telephony audio

When to use xAI Realtime vs alternatives

Rates

Notes

What’s Next