Skip to main content

ElevenLabs ConvAI

ElevenLabsConvAI is a Patter engine — an end-to-end speech-to-speech runtime that handles STT, LLM, and TTS in a single managed WebSocket. Instead of wiring stt=, llm=, and tts= separately (pipeline mode), you point Patter at an agent you already configured in the ElevenLabs Conversational AI dashboard and Patter bridges the carrier media stream straight to it. This is a different mental model from the pipeline-mode TTS providers — even though both use ElevenLabs:
ModeWhen to useConfig surface
Pipeline with ElevenLabsTTSYou want to keep STT, LLM, prompts, and tools in your code; ElevenLabs is just the voice.stt=, llm=, tts=, system_prompt=, tools=
Engine with ElevenLabsConvAIYou want a managed agent (prompt, tools, voice, knowledge base) authored in ElevenLabs’s UI.engine=, agent_id=
If you don’t yet know which mode fits, start with pipeline mode — it gives you Patter’s full feature set (custom tools, guardrails, dashboard, metrics). ConvAI is the right pick when your team treats the agent as a deployable artifact configured outside code.

Install

ElevenLabsConvAI ships in the core getpatter package — no extras needed:

Quickstart

The agent is created in the ElevenLabs ConvAI dashboard. Patter only needs the agent_id (and an API key) to connect to it.

Engine constructor

The engine config object is intentionally tiny — most settings live on the ElevenLabs side:
ParameterTypeDefaultDescription
api_keystr""ElevenLabs API key. Reads from ELEVENLABS_API_KEY when empty.
agent_idstr""ElevenLabs agent ID. Reads from ELEVENLABS_AGENT_ID when empty.
voicestr""Optional override for the agent’s default voice ID.

Lower-level adapter

When you need the full ConvAI feature set — first-message overrides, language pinning, signed-URL auth, audio-format negotiation — drop down to ElevenLabsConvAIAdapter:

Telephony optimisation

Both for_twilio / forTwilio and for_telnyx / forTelnyx negotiate ulaw_8000 for both output_audio_format and input_audio_format. This matches Twilio’s μ-law @ 8 kHz wire format and Telnyx’s PCMU bidirectional default — the SDK detects this and skips the 8 kHz ↔ 16 kHz resamples and the PCM ↔ μ-law transcodes. Saves ~30–80 ms first-byte plus per-frame CPU on every turn. If your Telnyx profile is pinned to L16/16000 instead, use the bare constructor (defaults to PCM16).

Signed-URL auth

When you cannot ship the API key to the WebSocket layer (browser bridges, untrusted runners), pass use_signed_url=True and Patter will fetch a short-lived signed URL from /v1/convai/conversation/get-signed-url instead:

Events

The adapter normalises ConvAI server messages into a unified event stream so the rest of Patter (metrics, dashboard, hooks) treats it like any other engine:
EventPayloadMeaning
audiobytesAgent audio chunk to forward to the caller.
transcript_inputstrWhat the user said.
transcript_outputstrWhat the agent said (full turn text).
response_start{"text": str}Agent began speaking (turn start).
response_done{}Agent turn finished — emitted on silence (~500 ms) or interruption.
interruptionNoneUser interrupted the agent.
errorstrServer-side error text.
response_done is emitted by a silence watcher because ConvAI does not send an explicit “turn complete” frame.

Limitations

  • No client-side system_prompt injection. The agent’s prompt is owned by the ElevenLabs dashboard. Patter passes system_prompt to the adapter as a first_message override only — to change behaviour, edit the agent in the ConvAI UI.
  • No client-side tools. ConvAI tools are configured in the ElevenLabs dashboard. Patter’s tools=[...] is ignored when an engine= is set.
  • model_id override is informational. The Python and TypeScript SDKs preserve the model_id field for introspection, but ConvAI does not currently honour a client-side model override — the model is whatever the agent was configured with.

Pricing

ElevenLabs ConvAI is billed by ElevenLabs per-minute (premium tiers add features like RAG and analytics). Patter does not currently meter ConvAI minutes — the engine cost shows up on your ElevenLabs invoice. Carrier minutes (Twilio / Telnyx) are still tracked as normal in CallMetrics.

See also