Skip to main content

Agent Configuration

An Agent defines how your voice AI behaves: what it says, how it sounds, what tools it can use, and what guardrails it follows.

Creating an Agent

Use the phone.agent() factory method. The simplest form leans on env-var fallback and a default engine (OpenAIRealtime):
To pick the engine explicitly (flat imports):
To use pipeline mode (pick STT, LLM, TTS independently):
Available LLM providers: OpenAILLM, AnthropicLLM, GroqLLM, CerebrasLLM, GoogleLLM. Tool calling works across all five. See LLM for the full reference. For fully custom logic (multi-model routing, local models), drop llm= and pass an on_message callback to serve() instead — llm= and on_message are mutually exclusive. The same pipeline using namespaced imports:

Agent Parameters

ParameterTypeDefaultDescription
system_promptstrrequiredInstructions that define the agent’s behavior.
engineOpenAIRealtime | OpenAIRealtime2 | ElevenLabsConvAI | NoneNone → OpenAI RealtimeEnd-to-end engine. See Engines. Omit for pipeline mode.
sttSTTProvider | NoneNoneSTT instance for pipeline mode (DeepgramSTT(), CartesiaSTT(), …). See STT.
llmLLMProvider | NoneNoneLLM instance for pipeline mode (AnthropicLLM(), GroqLLM(), …). Mutually exclusive with on_message on serve(). Ignored when engine is set. See LLM.
ttsTTSProvider | NoneNoneTTS instance for pipeline mode (ElevenLabsTTS(), RimeTTS(), …). See TTS.
voicestr"alloy"Voice name. Usually inferred from the engine or TTS instance.
modelstr"gpt-realtime-mini"Model ID for OpenAI Realtime. Usually inferred from the engine.
languagestr"en"BCP-47 language code.
first_messagestr""If set, the agent speaks this immediately when a call connects.
toolslist[Tool] | NoneNoneTool(...) instances for function calling. See Tools.
variablesdict | NoneNoneDynamic variable substitutions for {placeholder} patterns in the system prompt. Values limited to 500 chars.
guardrailslist[Guardrail] | NoneNoneGuardrail(...) instances applied to LLM output. See Guardrails.
hooksPipelineHooks | NoneNonePipeline hooks for intercepting STT/TTS processing. Pipeline mode only. See Events.
text_transformslist[Callable] | NoneNoneText transformation functions applied to LLM output before TTS. Pipeline mode only.
vadVADProvider | NoneNoneVoice activity detection provider (e.g. Silero). Pipeline mode only.
audio_filterAudioFilter | NoneNonePre-STT audio filter (e.g. Krisp noise suppression). Pipeline mode only.
background_audioBackgroundAudioPlayer | NoneNoneHold music / ambient-cue mixer. Pipeline mode only.
barge_in_threshold_msint300Sustained-voice window (ms) before treating caller audio as barge-in. Set to 0 to disable.
aggressive_first_flushboolFalseOpt-in low-latency mode: emits the first clause on a soft punctuation boundary (,, em-dash, en-dash) once the buffer reaches ~40 chars. Saves 200–500 ms TTFA on the first sentence at the cost of slightly clipped prosody. Hard-disabled when language starts with "it" (Italian decimal commas would split mid-number). Pipeline mode only.
disable_phone_preambleboolFalseWhen False (default), Patter prepends a phone-friendly preamble to system_prompt that instructs the LLM to avoid markdown, emojis, bullet lists, and code blocks; spell out numbers and dates; and keep replies short. Set to True to ship system_prompt verbatim.
prewarm_first_messageboolFalsePre-render first_message to TTS audio bytes during the ringing window and stream the cached buffer the instant the call connects, eliminating the 200–700 ms TTS first-byte latency on the greeting. Pipeline mode only — the flag is silently ignored (with a WARN log) on Realtime / ConvAI engines. Trade-off: pays for the greeting’s TTS even when the call rings out unanswered (~0.0010.001–0.005 per ring). Opt in explicitly for inbound calls and low-noise deployments: prewarm_first_message=True.

Agent Dataclass

Agent is a frozen (immutable) dataclass. You can construct it directly when you need a dataclass outside of phone.agent():
Prefer phone.agent() over constructing Agent directly — the factory method validates credentials, unpacks the engine/STT/TTS instances, and surfaces clear errors up front.

System Prompt

The system_prompt defines the agent’s personality, instructions, and constraints:

Dynamic Variables

Use {placeholder} syntax in the system prompt to inject dynamic values at call start. Values are limited to 500 characters each.

First Message

When first_message is set, the agent speaks it immediately when a call connects:

Pre-warming the first message

Pipeline-mode agents can pre-render the first_message audio during the ringing window and stream the cached buffer the instant the call connects — eliminating the 200–700 ms TTS first-byte latency on the greeting. Opt in explicitly:
The trade-off is paying for the greeting’s TTS even when the call rings out unanswered (typically 0.0010.001–0.005 per ring depending on TTS provider). Good for inbound calls and low-noise deployments; disable for very high-volume outbound where un-answered TTS spend matters. Realtime / ConvAI engines don’t consume the pre-rendered cache (their first message goes through the engine’s own audio path); the flag is silently ignored with a WARN log when set on a non-pipeline agent.

Voice Selection

Voice is usually inferred from the engine or TTS instance — e.g. OpenAIRealtime(voice="nova") or ElevenLabsTTS(voice_id="rachel"). Available voices depend on the provider.
"alloy", "ash", "ballad", "coral", "echo", "fable", "nova", "onyx", "sage", "shimmer", "verse"

Voice Activity Detection (VAD)

Pipeline-mode agents can plug a VAD provider into the vad= parameter to gate STT around real speech and drive barge-in detection. The SDK ships Silero VAD (an ONNX model, ~1 MB) with a telephony-tuned factory:
SileroVAD.for_phone_call(**overrides) is identical to SileroVAD.load(...) but pins sample_rate to 16 000 Hz — the only sample rate Patter’s pipeline-mode audio bus uses (8 kHz mulaw from Twilio is upsampled to 16 kHz PCM before reaching the VAD). Parameters are tuned for telephony-band audio (not the upstream Silero studio defaults):
FieldDefaultUpstream equivalent
activation_threshold0.8threshold (tuned for telephony, not studio)
deactivation_threshold0.65neg_threshold = threshold − 0.15 (tuned for telephony)
min_speech_duration0.25 smin_speech_duration_ms = 250
min_silence_duration0.1 smin_silence_duration_ms = 100
prefix_padding_duration0.03 sspeech_pad_ms = 30
Override per call site rather than as a global default. A common tweak: deployments that experience truncation on natural pauses raise min_silence_duration to 0.5–1.0 s:
SileroVAD.load(...) and SileroVAD.for_phone_call(...) are synchronous (they load the ONNX model). Wrap them in asyncio.to_thread(...) so the event loop stays responsive during process startup.

Engine vs Pipeline Mode

See LLM for a deeper comparison.

Complete Example