Skip to main content

Agent Configuration

An Agent defines how your voice AI behaves: what it says, how it sounds, what tools it can use, and what guardrails it follows.

Creating an Agent

Use the phone.agent() factory method. The simplest form leans on env-var fallback and a default engine (OpenAIRealtime):
To pick the engine explicitly (flat imports):
To use pipeline mode (pick STT, LLM, TTS independently):
Available LLM providers: OpenAILLM, AnthropicLLM, GroqLLM, CerebrasLLM, GoogleLLM. Tool calling works across all five. See LLM for the full reference. For fully custom logic (multi-model routing, local models), drop llm= and pass an on_message callback to serve() instead — llm= and on_message are mutually exclusive. The same pipeline using namespaced imports:

Agent Parameters

Agent Dataclass

Agent is a frozen (immutable) dataclass. You can construct it directly when you need a dataclass outside of phone.agent():
Prefer phone.agent() over constructing Agent directly — the factory method validates credentials, unpacks the engine/STT/TTS instances, and surfaces clear errors up front.

System Prompt

The system_prompt defines the agent’s personality, instructions, and constraints:

Dynamic Variables

Use {placeholder} syntax in the system prompt to inject dynamic values at call start. Values are limited to 500 characters each.

First Message

When first_message is set, the agent speaks it immediately when a call connects:

Pre-warming the first message

Pipeline-mode agents can pre-render the first_message audio during the ringing window and stream the cached buffer the instant the call connects — eliminating the 200–700 ms TTS first-byte latency on the greeting. Opt in explicitly:
The trade-off is paying for the greeting’s TTS even when the call rings out unanswered (typically 0.0010.001–0.005 per ring depending on TTS provider). Good for inbound calls and low-noise deployments; disable for very high-volume outbound where un-answered TTS spend matters. Realtime / ConvAI engines don’t consume the pre-rendered cache (their first message goes through the engine’s own audio path); the flag is silently ignored with a WARN log when set on a non-pipeline agent.

Voice Selection

Voice is usually inferred from the engine or TTS instance — e.g. OpenAIRealtime(voice="nova") or ElevenLabsTTS(voice_id="rachel"). Available voices depend on the provider.
"alloy", "ash", "ballad", "coral", "echo", "fable", "nova", "onyx", "sage", "shimmer", "verse"

Voice Activity Detection (VAD)

Pipeline-mode agents can plug a VAD provider into the vad= parameter to gate STT around real speech and drive barge-in detection. The SDK ships Silero VAD (an ONNX model, ~1 MB) with a telephony-tuned factory:
SileroVAD.for_phone_call(**overrides) is identical to SileroVAD.load(...) but pins sample_rate to 16 000 Hz — the only sample rate Patter’s pipeline-mode audio bus uses (8 kHz mulaw from Twilio is upsampled to 16 kHz PCM before reaching the VAD). Parameters are tuned for telephony-band audio (not the upstream Silero studio defaults): Override per call site rather than as a global default. A common tweak: deployments that experience truncation on natural pauses raise min_silence_duration to 0.5–1.0 s:
SileroVAD.load(...) and SileroVAD.for_phone_call(...) are synchronous (they load the ONNX model). Wrap them in asyncio.to_thread(...) so the event loop stays responsive during process startup.

Engine vs Pipeline Mode

See LLM for a deeper comparison.

Complete Example