Agent Configuration
AnAgent defines how your voice AI behaves: what it says, how it sounds, what tools it can use, and what guardrails it follows.
Creating an Agent
Use thephone.agent() factory method. The simplest form leans on env-var fallback and a default engine (OpenAIRealtime):
OpenAILLM, AnthropicLLM, GroqLLM, CerebrasLLM, GoogleLLM. Tool calling works across all five. See LLM for the full reference. For fully custom logic (multi-model routing, local models), drop llm= and pass an on_message callback to serve() instead — llm= and on_message are mutually exclusive.
The same pipeline using namespaced imports:
Agent Parameters
Agent Dataclass
Agent is a frozen (immutable) dataclass. You can construct it directly when you need a dataclass outside of phone.agent():
Prefer
phone.agent() over constructing Agent directly — the factory method validates credentials, unpacks the engine/STT/TTS instances, and surfaces clear errors up front.System Prompt
Thesystem_prompt defines the agent’s personality, instructions, and constraints:
Dynamic Variables
Use{placeholder} syntax in the system prompt to inject dynamic values at call start. Values are limited to 500 characters each.
First Message
Whenfirst_message is set, the agent speaks it immediately when a call connects:
Pre-warming the first message
Pipeline-mode agents can pre-render thefirst_message audio during the ringing window and stream the cached buffer the instant the call connects — eliminating the 200–700 ms TTS first-byte latency on the greeting. Opt in explicitly:
WARN log when set on a non-pipeline agent.
Voice Selection
Voice is usually inferred from the engine or TTS instance — e.g.OpenAIRealtime(voice="nova") or ElevenLabsTTS(voice_id="rachel"). Available voices depend on the provider.
- OpenAI Realtime
- ElevenLabs
- Pipeline
"alloy", "ash", "ballad", "coral", "echo", "fable", "nova", "onyx", "sage", "shimmer", "verse"Voice Activity Detection (VAD)
Pipeline-mode agents can plug a VAD provider into thevad= parameter to gate STT around real speech and drive barge-in detection. The SDK ships Silero VAD (an ONNX model, ~1 MB) with a telephony-tuned factory:
SileroVAD.for_phone_call(**overrides) is identical to SileroVAD.load(...) but pins sample_rate to 16 000 Hz — the only sample rate Patter’s pipeline-mode audio bus uses (8 kHz mulaw from Twilio is upsampled to 16 kHz PCM before reaching the VAD). Parameters are tuned for telephony-band audio (not the upstream Silero studio defaults):
Override per call site rather than as a global default. A common tweak: deployments that experience truncation on natural pauses raise
min_silence_duration to 0.5–1.0 s:
SileroVAD.load(...) and SileroVAD.for_phone_call(...) are synchronous (they load the ONNX model). Wrap them in asyncio.to_thread(...) so the event loop stays responsive during process startup.
