Skip to main content

ElevenLabs ConvAI

ElevenLabsConvAI is a Patter engine — an end-to-end speech-to-speech runtime that handles STT, LLM, and TTS in a single managed WebSocket. Instead of wiring stt, llm, and tts separately (pipeline mode), you point Patter at an agent you already configured in the ElevenLabs Conversational AI dashboard and Patter bridges the carrier media stream straight to it. This is a different mental model from the pipeline-mode TTS providers — even though both use ElevenLabs: If you don’t yet know which mode fits, start with pipeline mode — it gives you Patter’s full feature set (custom tools, guardrails, dashboard, metrics). ConvAI is the right pick when your team treats the agent as a deployable artifact configured outside code.

Install

ElevenLabsConvAI ships in the core getpatter package — no extras needed:

Quickstart

The agent is created in the ElevenLabs ConvAI dashboard. Patter only needs the agentId (and an API key) to connect to it.

Engine constructor

The engine config is intentionally tiny — most settings live on the ElevenLabs side:

Lower-level adapter

When you need the full ConvAI feature set — first-message overrides, language pinning, signed-URL auth, audio-format negotiation — drop down to ElevenLabsConvAIAdapter:

Telephony optimisation

Both forTwilio and forTelnyx negotiate ulaw_8000 for both outputAudioFormat and inputAudioFormat. This matches Twilio’s μ-law @ 8 kHz wire format and Telnyx’s PCMU bidirectional default — the SDK detects this and skips the 8 kHz ↔ 16 kHz resamples and the PCM ↔ μ-law transcodes. Saves ~30–80 ms first-byte plus per-frame CPU on every turn. If your Telnyx profile is pinned to L16/16000 instead, use the bare constructor (defaults to PCM16).

Signed-URL auth

When you cannot ship the API key to the WebSocket layer (browser bridges, untrusted runners), pass useSignedUrl: true and the adapter will fetch a short-lived signed URL from /v1/convai/conversation/get-signed-url instead:

Events

The adapter normalises ConvAI server messages into a unified event stream so the rest of Patter (metrics, dashboard, hooks) treats it like any other engine: response_done is emitted by a silence watcher because ConvAI does not send an explicit “turn complete” frame.

Limitations

  • No client-side systemPrompt injection. The agent’s prompt is owned by the ElevenLabs dashboard. Patter passes systemPrompt to the adapter as a firstMessage override only — to change behaviour, edit the agent in the ConvAI UI.
  • No client-side tools. ConvAI tools are configured in the ElevenLabs dashboard. Patter’s tools array is ignored when an engine is set.
  • modelId override is informational. The TypeScript and Python SDKs preserve the modelId field for introspection, but ConvAI does not currently honour a client-side model override.

Pricing

ElevenLabs ConvAI is billed by ElevenLabs per-minute (premium tiers add features like RAG and analytics). Patter does not currently meter ConvAI minutes — the engine cost shows up on your ElevenLabs invoice. Carrier minutes (Twilio / Telnyx) are still tracked as normal in CallMetrics.

See also