Skip to main content

What Patter does

Patter is an open-source SDK that lets an AI agent answer and make phone calls. You write the agent logic — Patter handles the telephony plumbing (phone carrier, audio streams, speech recognition, speech synthesis, barge-in, call transfer, recording). A “hello world” is four lines:
Call your Twilio number; the AI picks up, talks to the caller, hangs up. Everything else — audio transcoding, webhook configuration, session lifecycle — is automatic.

Anatomy of a call

When someone dials your number, audio flows through five layers. Each layer is pluggable:
The five building blocks you’ll touch:

1. Carrier — the phone line

The carrier puts the call on the internet. Patter ships two:
CarrierEnv varsNotes
Twilio()TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKENmulaw 8 kHz; webhook signature verification via HMAC-SHA1
Telnyx()TELNYX_API_KEY, TELNYX_CONNECTION_ID, TELNYX_PUBLIC_KEYPCM 16 kHz native; Ed25519 webhook verification
Plivo()PLIVO_AUTH_ID, PLIVO_AUTH_TOKENmulaw 8 kHz; V3 (HMAC-SHA256) webhook verification; native DTMF send
You pass one instance to Patter(carrier=...). See Carrier.

2. Engine vs pipeline — how the AI talks

Patter supports two ways to get from audio-in to audio-out: Engine mode (easiest) — a single provider does everything:
EngineWhat it doesBest for
OpenAIRealtime()Speech-to-speech via OpenAI’s Realtime APILowest latency, general-purpose
ElevenLabsConvAI()Managed conversational agent on ElevenLabsPremium voice quality
XaiRealtime()Speech-to-speech via xAI’s Grok Voice AgentReasoning + built-in web / X search
Pipeline mode (full control) — you pick each stage:
Want fully custom LLM logic (multi-model routing, local models, an internal gateway)? Drop llm= and pass an async on_message callback to serve() instead. llm= and on_message are mutually exclusive. Three independent stages — swap any of them for a different vendor or local model:
  • STT (Speech-to-Text) — transcribes caller audio in real time. Providers: DeepgramSTT, WhisperSTT, CartesiaSTT, SonioxSTT, SpeechmaticsSTT (Python-only), AssemblyAISTT, XaiSTT. See STT.
  • LLM (Large Language Model) — generates the reply. Pass a class instance via llm=: OpenAILLM, AnthropicLLM, GroqLLM, CerebrasLLM, GoogleLLM. Tool calling works across all five. For anything else, use on_message. See LLM.
  • TTS (Text-to-Speech) — synthesizes the reply audio. Providers: ElevenLabsTTS, OpenAITTS, CartesiaTTS, RimeTTS, LMNTTTS, XaiTTS. See TTS.
Pick engine mode when you want minimum code. Pick pipeline mode when you need a specific LLM, a custom voice, or fine-grained control over latency / costs.

3. Tools — letting the AI do things

Tools are functions the LLM can call mid-conversation — “look up order #123”, “transfer to billing”, “schedule a callback for 3pm”. When the LLM decides to use a tool, Patter runs your handler (or POSTs to a webhook) and feeds the result back into the conversation.
Two system tools are always available: transfer_call (move the call to a human) and end_call (hang up). See Tools.

4. Guardrails — what the AI can’t say

Guardrails run on every LLM output before it reaches TTS. They can block terms, run a custom check, or substitute a safe reply:
See Guardrails.

5. Tunnel — making your laptop reachable

A carrier needs a public HTTPS URL to deliver webhooks. In dev you don’t have one — Patter ships a built-in Cloudflare Quick Tunnel:
In production you point webhook_url="api.yourcompany.com" at your own server and skip the tunnel entirely. See Tunneling.

Barge-in

When the caller starts speaking while the AI is mid-reply, Patter cancels the in-flight audio and routes the new input to the LLM. Tracking is mark-based (per-frame) so cancellation happens on the exact syllable that got interrupted — not at a coarse chunk boundary. Default sensitivity is 300 ms of sustained voice. Tune with agent(barge_in_threshold_ms=500) if you’re on a noisy line (speakerphone, ngrok relay).

Audio pipeline — transcoding details

The carriers don’t all speak the same format. Patter transcodes so every STT / engine sees the same PCM 16 kHz stream:
CarrierWire formatPatter does
Twiliomulaw 8 kHz (ulaw/8000)Decode to PCM 16 kHz in, encode back to mulaw out
TelnyxPCM 16 kHz (L16/16000)Passthrough
Plivomulaw 8 kHz (audio/x-mulaw)Decode to PCM 16 kHz in, encode back to mulaw out
OpenAI TTS returns 24 kHz PCM — Patter resamples to 16 kHz before sending to the carrier. You never touch any of this.

What’s next

Python Quickstart

Answer a real phone call in 5 minutes.

TypeScript Quickstart

Same, but in TS.

Carrier setup

Twilio / Telnyx / Plivo credentials.

Engines

OpenAI Realtime, ElevenLabs ConvAI.