ElevenLabs ConvAI

ElevenLabsConvAI is a Patter engine — an end-to-end speech-to-speech runtime that handles STT, LLM, and TTS in a single managed WebSocket. Instead of wiring stt, llm, and tts separately (pipeline mode), you point Patter at an agent you already configured in the ElevenLabs Conversational AI dashboard and Patter bridges the carrier media stream straight to it. This is a different mental model from the pipeline-mode TTS providers — even though both use ElevenLabs:

Mode	When to use	Config surface
Pipeline with `ElevenLabsTTS`	You want to keep STT, LLM, prompts, and tools in your code; ElevenLabs is just the voice.	`stt`, `llm`, `tts`, `systemPrompt`, `tools`
Engine with `ElevenLabsConvAI`	You want a managed agent (prompt, tools, voice, knowledge base) authored in ElevenLabs’s UI.	`engine`, `agentId`

If you don’t yet know which mode fits, start with pipeline mode — it gives you Patter’s full feature set (custom tools, guardrails, dashboard, metrics). ConvAI is the right pick when your team treats the agent as a deployable artifact configured outside code.

Install

ElevenLabsConvAI ships in the core getpatter package — no extras needed:

npm install getpatter

Quickstart

// npx tsx example.ts
import { Patter, Twilio, ElevenLabsConvAI } from "getpatter";

const phone = new Patter({ carrier: new Twilio(), phoneNumber: "+15550001234" });

const agent = phone.agent({
  engine: new ElevenLabsConvAI({ agentId: "agent_abc123" }),  // ELEVENLABS_API_KEY from env
  systemPrompt: "You are a warm and friendly concierge.",     // used as override
});

await phone.serve({ agent });

import asyncio
from getpatter import Patter, Twilio, ElevenLabsConvAI

phone = Patter(carrier=Twilio(), phone_number="+15550001234")

agent = phone.agent(
    engine=ElevenLabsConvAI(agent_id="agent_abc123"),
    system_prompt="You are a warm and friendly concierge.",
)

asyncio.run(phone.serve(agent))

The agent is created in the ElevenLabs ConvAI dashboard. Patter only needs the agentId (and an API key) to connect to it.

Engine constructor

The engine config is intentionally tiny — most settings live on the ElevenLabs side:

Parameter	Type	Default	Description
`apiKey`	`string`	—	ElevenLabs API key. Reads from `ELEVENLABS_API_KEY` when omitted.
`agentId`	`string`	—	ElevenLabs agent ID. Reads from `ELEVENLABS_AGENT_ID` when omitted.
`voice`	`string`	—	Optional override for the agent’s default voice ID.

Lower-level adapter

When you need the full ConvAI feature set — first-message overrides, language pinning, signed-URL auth, audio-format negotiation — drop down to ElevenLabsConvAIAdapter:

import { ElevenLabsConvAIAdapter } from "getpatter";

// Twilio-native μ-law @ 8 kHz — skips every transcoding hop.
const adapter = ElevenLabsConvAIAdapter.forTwilio("API_KEY", "agent_abc123", {
  voiceId: "EXAVITQu4vr4xnSDxMaL",
  modelId: "eleven_flash_v2_5",
  language: "it",
  firstMessage: "Ciao!",
});

// Telnyx — same μ-law optimisation when streaming_bidirectional_codec=PCMU.
const adapterTelnyx = ElevenLabsConvAIAdapter.forTelnyx("API_KEY", "agent_abc123");

// Bare constructor — defaults to PCM16 16 kHz (best for non-telephony).
const adapterBare = new ElevenLabsConvAIAdapter({
  apiKey: "API_KEY",
  agentId: "agent_abc123",
});

from getpatter.providers.elevenlabs_convai import ElevenLabsConvAIAdapter

adapter = ElevenLabsConvAIAdapter.for_twilio(
    api_key="API_KEY",
    agent_id="agent_abc123",
    voice_id="EXAVITQu4vr4xnSDxMaL",
    model_id="eleven_flash_v2_5",
    language="it",
    first_message="Ciao!",
)

adapter_telnyx = ElevenLabsConvAIAdapter.for_telnyx(api_key="API_KEY", agent_id="agent_abc123")
adapter_bare = ElevenLabsConvAIAdapter(api_key="API_KEY", agent_id="agent_abc123")

Telephony optimisation

Both forTwilio and forTelnyx negotiate ulaw_8000 for both outputAudioFormat and inputAudioFormat. This matches Twilio’s μ-law @ 8 kHz wire format and Telnyx’s PCMU bidirectional default — the SDK detects this and skips the 8 kHz ↔ 16 kHz resamples and the PCM ↔ μ-law transcodes. Saves ~30–80 ms first-byte plus per-frame CPU on every turn. If your Telnyx profile is pinned to L16/16000 instead, use the bare constructor (defaults to PCM16).

Signed-URL auth

When you cannot ship the API key to the WebSocket layer (browser bridges, untrusted runners), pass useSignedUrl: true and the adapter will fetch a short-lived signed URL from /v1/convai/conversation/get-signed-url instead:

const adapter = new ElevenLabsConvAIAdapter({
  apiKey: "API_KEY",
  agentId: "agent_abc123",
  useSignedUrl: true,
});

Events

The adapter normalises ConvAI server messages into a unified event stream so the rest of Patter (metrics, dashboard, hooks) treats it like any other engine:

Event	Payload	Meaning
`audio`	`Buffer`	Agent audio chunk to forward to the caller.
`transcript_input`	`string`	What the user said.
`transcript_output`	`string`	What the agent said (full turn text).
`response_start`	`{ text: string }`	Agent began speaking (turn start).
`response_done`	`null`	Agent turn finished — emitted on silence (~500 ms) or interruption.
`interruption`	`null`	User interrupted the agent.
`error`	`string`	Server-side error text.
`close`	`{ code, reason }`	WebSocket closed.

response_done is emitted by a silence watcher because ConvAI does not send an explicit “turn complete” frame.

adapter.onEvent((type, data) => {
  if (type === "audio") forwardToCaller(data as Buffer);
  if (type === "transcript_output") console.log("Agent said:", data);
});

Limitations

No client-side systemPrompt injection. The agent’s prompt is owned by the ElevenLabs dashboard. Patter passes systemPrompt to the adapter as a firstMessage override only — to change behaviour, edit the agent in the ConvAI UI.
No client-side tools. ConvAI tools are configured in the ElevenLabs dashboard. Patter’s tools array is ignored when an engine is set.
modelId override is informational. The TypeScript and Python SDKs preserve the modelId field for introspection, but ConvAI does not currently honour a client-side model override.

Pricing

ElevenLabs ConvAI is billed by ElevenLabs per-minute (premium tiers add features like RAG and analytics). Patter does not currently meter ConvAI minutes — the engine cost shows up on your ElevenLabs invoice. Carrier minutes (Twilio / Telnyx) are still tracked as normal in CallMetrics.

​ElevenLabs ConvAI

​Install

​Quickstart

​Engine constructor

​Lower-level adapter

​Telephony optimisation

​Signed-URL auth

​Events

​Limitations

​Pricing

​See also