Skip to main content

Advanced Features

Patter includes production-ready features for handling real-world telephony scenarios.

Call Recording

Enable recording on a per-call basis by passing recording=True to serve(). Recordings are created via the Twilio Recordings API.
Call recording is available in local mode with Twilio. Recordings are managed through the Twilio Recordings API and stored in your Twilio account.

Accessing Recordings

Use the Twilio API to list and download recordings:

Local Recording (Carrier-Neutral)

local_recording records each call on your own machine, with no carrier API involved. The SDK taps the audio already flowing through the per-call stream handler and writes an interleaved stereo WAV — left channel = caller, right channel = agent — as PCM16 at 16 kHz. μ-law 8 kHz carrier audio is upsampled and 24 kHz Realtime audio is downsampled, so the output format is the same on every carrier and engine mode.
ParameterTypeDefaultDescription
local_recordingbool | strFalseSDK-side stereo WAV recording (left = caller, right = agent, PCM16 16 kHz). Pass a directory string to set the output directory. Works on every carrier (Twilio, Telnyx, Plivo) and engine mode.
Where the file lands:
  1. Directory string → <dir>/<call_id>.wav.
  2. Call logging enabled → <call_log_dir>/recording.wav, next to metadata.json and transcript.jsonl (so it is covered by the same PATTER_LOG_RETENTION_DAYS cleanup).
  3. Otherwise → ./recordings/<call_id>.wav under the current working directory.
The final path is surfaced as recording_path in the on_call_end payload and persisted in the call-log metadata.json:
The WAV header is finalized on every teardown path — including abnormal carrier WebSocket drops — so a truncated call still yields a playable file. local_recording is independent of the carrier-side recording flag: both can be enabled on the same call.

Answering Machine Detection (AMD)

Detect whether a human or machine answered an outbound call. When a machine is detected, optionally leave a voicemail message and hang up. AMD is on by default since 0.6.2. On Twilio, Patter uses MachineDetection=DetectMessageEnd + Async AMD so there is no answer-latency penalty on human pickups — the call connects immediately and the classification arrives via the /webhooks/twilio/amd callback. Pass machine_detection=False to skip per-call AMD billing when the destination is known to be a human.
ParameterTypeDefaultDescription
machine_detectionboolTrueEnable answering machine detection. Defaults on since 0.6.2. Pass False to skip AMD billing.
voicemail_messagestr""Message to speak when a machine is detected. If empty, the call hangs up silently. A non-empty value implicitly enables AMD even if machine_detection=False.
on_machine_detectionCallable[[MachineDetectionResult], Awaitable[None] | None] | NoneNoneFires once when the carrier reports the AMD outcome (human or machine). Useful for acceptance tests that need to mark a run INVALID when classification is not human.

How It Works

  1. Patter initiates the outbound call with AMD enabled.
  2. The telephony provider analyzes the audio to determine if a human or machine answered.
  3. Human detected: The call proceeds normally with the agent.
  4. Machine detected: If voicemail_message is set, it is spoken and the call ends. Otherwise, the call is disconnected.
You can also set voicemail_message on serve() for use with any outbound call made during the server’s lifetime:

DTMF Input

Keypad presses (DTMF tones) during a call are captured and forwarded to the AI agent as natural language text in the format [DTMF: N], where N is the key pressed (0-9, *, #).
No configuration is required. DTMF input is automatically handled and sent to the AI as part of the conversation transcript. The AI can interpret keypresses based on its system prompt:

Call Transfer

Patter automatically injects a transfer_call system tool into every agent. The AI decides when to transfer based on the conversation context and system prompt instructions.
The transfer is executed via the Twilio API as a redirect. The caller hears hold music briefly while the transfer completes.
You do not need to define transfer_call as a tool. It is injected automatically by Patter.

Barge-In (Interruption Handling)

Patter uses mark-based tracking for precise interruption handling. When a caller speaks while the agent is talking, the system:
  1. Detects the interruption via audio marks sent by the telephony provider.
  2. Stops the current TTS playback at the exact point of interruption.
  3. Processes the caller’s new input immediately.
This creates a natural conversational experience where the caller can interrupt the agent mid-sentence, just like a real phone call.

Configuration

Barge-in is enabled by default with a 300 ms hang-over window. Customize the sensitivity using barge_in_threshold_ms:
ParameterTypeDefaultDescription
barge_in_threshold_msint300Hang-over window in milliseconds. Set to 0 to disable barge-in. Higher values delay interruption detection.
A hang-over window of 300 ms prevents false positives from background noise while remaining responsive to genuine interruptions.

Echo Cancellation (NLMS AEC)

On speakerphone or dev-tunnel deployments the agent’s outbound TTS bleeds back into the inbound mic feed. The pipeline-mode VAD then sees continuous voice-like energy and never registers silence — barge-in only fires during natural pauses in the TTS, producing the intermittent “interrupt sometimes works, other times the agent keeps talking” symptom. Acoustic echo cancellation (AEC) subtracts the estimated echo from the mic stream before VAD/STT see it. Patter ships a built-in NLMS (normalised least-mean-squares) adaptive filter with Geigel double-talk detection. Enable it with one flag — pipeline mode only:
ParameterTypeDefaultDescription
echo_cancellationboolFalseWhen True (pipeline mode only), instantiates an NlmsEchoCanceller per call that subtracts the agent’s own TTS bleed from the inbound mic stream before VAD/STT see it.

When to enable

  • Enable for speakerphone callers, ngrok / Cloudflare tunnel demos, laptop-mic test harnesses, and any deployment where the agent can hear itself.
  • Leave off for handset / headset callers — there is no bleed to cancel, and the 0.5–2 s convergence period would briefly attenuate caller speech if they spoke before any TTS played.
  • See Barge-In above — AEC is the fix when barge-in only fires intermittently because of self-bleed.

Tuning

The default NlmsEchoCanceller is tuned for narrowband mono 16 kHz PCM (the format Patter’s pipeline pushes between transcoding and STT). For lower-level control — custom tap counts, step size, warmup behaviour — instantiate one directly and wire it into your pipeline:
Constructor argDefaultNotes
sample_rate160008000 or 16000 only.
filter_taps51232 ms @ 16 kHz — covers typical cellular / VoIP echo paths.
step_size0.1NLMS step in (0, 1] post-warmup.
warmup_step_size0.5Aggressive 5× ramp during the first ~0.5 s for fast convergence.
warmup_seconds0.5Duration of the warmup phase.
leakage0.9999Slow forgetting of stale tap estimates.
double_talk_rho0.6Geigel threshold — freezes adaptation when caller speaks over agent.
NLMS AEC adds CPU work proportional to filter_taps × frame_samples per inbound frame (~0.5–1 ms per 20 ms frame at the defaults). On commodity CPUs this is well under the per-frame budget, but profile if you stack AEC with heavy VAD + STT in the same event loop.
This is a lightweight time-domain AEC, not a drop-in replacement for production-grade DSP (WebRTC’s AEC3, Speex AEC). For tight integration with battle-tested DSP, wrap a binding to libwebrtc-audio-processing externally and feed it via audio_filter= instead.
Browser / native only — a no-op by design on PSTN. The NLMS filter only models an echo path that fits inside its 32 ms window. On a phone call the audio traverses a 250–1500 ms carrier jitter buffer + loop, so the round-trip echo lands far outside that window and the filter passes the frame through unchanged. PSTN line echo is already handled by the carrier (ITU-T G.168) plus the caller’s own device, so echo_cancellation should stay False on Twilio / Telnyx / Plivo — enable it only for browser/WebRTC or native-mobile deployments where Patter owns the mic-and-speaker path end-to-end. The SDK logs a one-shot warning if you enable it on a PSTN carrier.

Inbound Audio Front-End (High-Pass & AGC)

Two opt-in, provider-agnostic stages clean up the caller audio before it reaches VAD and STT — they run once per frame regardless of which STT you choose, and both fail open (a stage error degrades to passthrough, never drops the call). The full inbound order is high-pass → resample → AEC → noise suppression → AGC → VAD → STT.
ParameterTypeDefaultDescription
high_pass_hzint | NoneNoneHigh-pass / DC-block cutoff in Hz (typical 80–120). Runs as the first stage, before AEC, removing DC offset, mains hum (50/60 Hz) and handling rumble that otherwise bias the echo canceller and inflate the VAD energy estimate. None disables it.
agcbool | AgcConfigFalseSpeech-selective automatic gain control. Runs after noise suppression and before VAD/STT, normalising the caller’s level toward a target RMS to cut word-error rate on quiet / variable-distance talkers. True uses defaults; pass an AgcConfig to tune.
The AGC is speech-selective — gain is only driven up on speech frames, so the quiet gaps between words are never amplified into a hiss — and peak-limited so an aggressive boost never clips. Tune it with AgcConfig:
Both stages are pure DSP (no numpy required) and cost well under 1 % of one core. They are pipeline-mode only — Realtime / ConvAI providers own their own inbound audio path.

Aggressive First-Flush (Low-Latency)

In pipeline mode, the sentence chunker normally waits for a hard sentence terminator (., !, ?, etc.) before emitting a chunk to TTS. With aggressive_first_flush=True on phone.agent(...), the chunker emits the first clause of each response on a soft punctuation boundary (,, em-dash , en-dash ) once the buffer reaches ~40 characters.
Trade-off: Saves 200–500 ms of time-to-first-audio (TTFA) on the first sentence of each turn, at the cost of slightly clipped prosody on the very first chunk.
aggressive_first_flush is hard-disabled when language starts with "it" (Italian). Italian uses the comma as a decimal separator (12,5), so an aggressive flush would split mid-number. The flag silently has no effect for Italian agents.

Sentence chunker — abbreviations & terminators

The chunker does not split on common abbreviations (no spurious sentence breaks after Dr., vs., etc.). Coverage:
  • English: Mr, Mrs, Ms, Dr, St, Jr, Sr, Prof, Hon, Rev, vs, etc, Gen, Sen, plus the standard month/measurement set.
  • Italian: Sig, Sig.ra, Sgr, Dott, Dott.ssa, Prof, Avv, Ing, Geom, Rag, Arch, On, Egr, Spett, Gent, Ill, plus business/legal abbreviations like S.p.A., S.r.l., S.a.s., ecc.
  • Multilingual sentence terminators: Latin (. ! ?), Western ellipsis (), CJK (。 ! ? 。 . ;), Hindi/Devanagari (। ॥), Arabic (؟ ؛ ۔ ؏), Armenian (։ ՜ ՞), Ethiopic (։ ፧), Khmer (។ ៕), Burmese (), Tibetan (༎ ༏).
This means the chunker streams cleanly on multilingual responses without hand-tuning. The SentenceChunker constructor accepts an optional language= argument (BCP-47 code) — Patter forwards agent.language automatically, but you can construct one directly with the language you want when wiring the chunker manually:

Phone Preamble (System Prompt Wrapper)

By default, Patter prepends a phone-friendly preamble to every agent’s system_prompt before sending it to the LLM. The preamble instructs the model to:
  • Avoid markdown, emojis, bullet lists, and code blocks.
  • Spell out numbers and dates (e.g., “two thousand twenty-six”, not 2026).
  • Keep replies short — phone calls reward brevity over completeness.
Most callers benefit from this. If you ship a custom prompt that already encodes phone conventions — or you want to drive a non-voice LLM channel through the same agent — opt out:
ParameterTypeDefaultDescription
disable_phone_preambleboolFalseWhen True, ship system_prompt verbatim to the LLM. When False (default), prepend the phone-friendly preamble.

AI Disclosure

Many jurisdictions require disclosure that the caller is speaking with an AI. Patter does not automatically inject a disclosure message. Instead, use the first_message field on your agent configuration to include an appropriate disclosure at the start of every call:
You are responsible for ensuring your AI disclosure complies with the regulations in your jurisdiction. Always consult legal counsel for compliance requirements.

Conversation History

All callbacks receive the full conversation history as data.history. Each entry includes the speaker role, text content, and timestamp:

History Entry Format

History is available in on_transcript, on_message, and on_call_end callbacks.

Complete Example