Skip to main content

Events & Callbacks

Patter fires async callbacks at key moments in the call lifecycle. Use them to log calls, update CRMs, trigger workflows, or control conversation flow. All callbacks are async functions. They are passed as parameters to serve().

Available Callbacks

CallbackTrigger
on_call_startA call connects
on_call_endA call ends
on_transcriptEach utterance is transcribed
on_messageUser message received (pipeline mode)
on_metricsAfter each conversation turn (real-time cost/latency)
For fine-grained pipeline observability see the Speech-edge events and Tool events via on_transcript sections below — they complement the lifecycle callbacks rather than replacing them. For mutating prompts and responses (RAG augmentation, output validation, PII redaction) use PipelineHooks — they sit inside the LLM step rather than firing alongside it.

on_call_start

Fires when a call connects. Use it to log call starts, initialize state, or fetch customer data.

Event Fields

FieldTypeDescription
call_idstrUnique identifier for this call.
callerstrThe caller’s phone number (E.164).
calleestrThe callee’s phone number (E.164).
directionstr"inbound" or "outbound".
custom_paramsdictCustom parameters passed with the call (if any).

on_call_end

Fires when a call ends. Use it to save transcripts, calculate duration, or trigger post-call workflows.

Event Fields

FieldTypeDescription
call_idstrUnique identifier for this call.
callerstrThe caller’s phone number (E.164).
calleestrThe callee’s phone number (E.164).
ended_atfloatUnix timestamp when the call ended (e.g. 1710489601.234).
transcriptlist[dict]Full conversation transcript. Each entry has role ("user" or "assistant") and text.
metricsCallMetrics | NoneCall metrics with cost and latency breakdowns. None if metrics collection failed. See Metrics & Cost Tracking.
recording_pathstr | NonePath to the local stereo WAV. Only present when local_recording was enabled on serve(); None if finalizing the file failed. See Local Recording.

on_transcript

Fires each time an utterance is transcribed during the call. Use it for real-time logging, sentiment analysis, or live dashboards.

Event Fields

FieldTypeDescription
rolestr"user" or "assistant".
textstrThe transcribed text.
call_idstrUnique identifier for this call.
historylist[dict]Conversation history so far. Each entry has role, text, and timestamp.

on_message

Fires when a user message is received in pipeline mode. Your callback processes the message and returns the agent’s response as a string, which is then synthesized to speech.
on_message is only used in pipeline mode (when you pass stt= / tts= instead of engine=). In engine mode (OpenAI Realtime, ElevenLabs ConvAI) the engine handles responses directly.

Event Fields

FieldTypeDescription
textstrThe user’s transcribed message.
call_idstrUnique identifier for this call.
callerstrThe caller’s phone number.
calleestrThe callee’s phone number.
historylist[dict]Conversation history. Each entry has role, text, and timestamp.

Return Value

Return a str with the agent’s response. This text is sent to the TTS provider and played back to the caller.

Speech-edge events

For turn-taking, TTFT measurement, and barge-in / interrupt observability, set the speech-edge callbacks directly on the Patter instance. They proxy to a per-process SpeechEvents dispatcher and fire from any in-flight call.
AttributeFires
on_user_speech_startedRaw VAD positive edge (caller begins speaking).
on_user_speech_endedRaw VAD trailing edge (caller stops speaking).
on_user_speech_eosCommitted end-of-utterance — anchor TTFT here.
on_agent_speech_startedFirst wire-time agent audio chunk — turn-start marker for the caller.
on_agent_speech_endedLast agent audio chunk. Payload includes interrupted flag for barge-in.
on_llm_tokenFirst LLM token of the turn — TTFT marker.
on_audio_outFirst TTS audio bytes produced — TTS warmup signal.
Callbacks are async. Throwing inside a callback logs the error but does not interrupt the call.

Tool events via on_transcript

Tool invocations (including the built-in transfer_call and end_call) surface through the same on_transcript callback you pass to phone.serve(...). Filter on role == "tool" to handle them:
The event payload for tool calls carries:
KeyTypeNotes
role"tool"Always "tool" for tool events.
tool_namestrThe tool that was dispatched.
tool_argsdictArguments emitted by the LLM.
tool_resultstr | NoneResult returned by the tool handler (truncated for log readability).
call_idstrThe active call ID.
textstrPre-formatted “tool_name(args) → result” string.

Pipeline Hooks

PipelineHooks lets you intercept data at each stage of the pipeline mode STT → LLM → TTS pipeline. Pass an instance via phone.agent(hooks=...). Hooks may be sync or async; if a hook throws, the error is logged and the original value passes through unchanged (fail-open).

after_llm — 3-tier API

after_llm accepts either a dict with on_chunk / on_sentence / on_response keys, or any object exposing those attributes (dataclass, custom class, Protocol implementation).
TierSync/AsyncLatency budgetWhen it runsReturn semantics
on_chunk(chunk: str) -> strsync~0 msPer LLM token chunk, before sentence aggregationReturn new string. Use for cheap text rewrites.
on_sentence(sentence: str, ctx: HookContext) -> str | Noneasync50–300 msPer complete sentence, between chunker and TTSReturn new sentence, None to keep original, or "" to drop the sentence.
on_response(text: str, ctx: HookContext) -> str | Noneasync500 ms–2 sOnce at end of LLM stream, blocks streaming TTSReturn new text, or None to keep original.
Pick the lowest tier that does the job — on_chunk for fast string ops, on_sentence for per-sentence I/O (PII redaction, translation), on_response only when you need the whole response (JSON-schema validation, full-context moderation).

Migration: legacy after_llm callable

The legacy single-callable form is still supported for backward compatibility but is deprecated:
The legacy callable is mapped internally to the on_response slot and emits a one-shot PatterDeprecationWarning on first use. Migrate to the 3-tier dict to silence the warning and unlock the lower-latency on_chunk / on_sentence tiers.

HookContext

Hooks that take a ctx argument receive a frozen HookContext dataclass:
PipelineHooks also exposes before_stt / after_stt and before_tts / after_tts for audio-stage interception. See the API Reference for the full signature.

Conversation History

All callbacks that include history receive it as a list of dictionaries:
Timestamps are Unix floats (from Python’s time.time()), not ISO-8601 strings.

Complete Example


Speech-Edge Events (Turn-Taking)

The callbacks above describe the transcript-level lifecycle of a call. For turn-taking instrumentation — barge-in, end-of-utterance, time-to-first-token, TTS warmup vs. wire-time — Patter exposes seven additional async callbacks plus a read-only conversation_state snapshot directly on the Patter instance. These events expose the canonical voice-agent metric set (user/agent state transitions, turn boundaries, TTFT, audio first-byte) and align with OpenAI Realtime (input_audio_buffer.speech_started/_stopped/_committed) so downstream metrics work without translation.
Every callback defaults to None. Existing code that does not register any speech-edge callback sees exactly the previous behaviour and zero overhead. The state machine is updated regardless of whether callbacks are registered, so conversation_state is always usable.

The seven events

EventFires onSignal
on_user_speech_startedVAD positive edge of inbound audioRaw VAD start — not end-of-utterance. Use for cross-talk detection.
on_user_speech_endedVAD trailing edgeRaw VAD stop — not committed EOU. Use for talk-ratio.
on_user_speech_eosCommitted end-of-utteranceCanonical “user finished” signal. Anchor eos_to_first_token_ms here.
on_agent_speech_startedFirst wire-time chunk of the agent turnWhat the user actually hears (distinct from TTS warmup). Anchor barge-in latency here.
on_agent_speech_endedLast wire chunk of the agent turnPayload includes interrupted: bool. True = barge-in cancelled the turn.
on_llm_tokenFirst LLM token of the turnTTFT marker. Idempotent — fires once per turn.
on_audio_outFirst TTS audio chunk producedTTS warmup arrival (distinct from wire-time). Idempotent — fires once per turn.

Payload signature matrix

Compute end-to-end latency by anchoring eos_to_first_token_ms to on_user_speech_eos. It marks the moment the SDK has committed that the user is done speaking — VAD trailing edge plus trailing silence (and optionally a semantic turn-detector agreement). Anchoring to on_user_speech_ended instead would over-count by the silence window and double-fire on mid-utterance VAD blips. Hamming AI thresholds: <800 ms good, >1500 ms critical.

State machine

conversation_state returns a snapshot {"user": <user_state>, "agent": <agent_state>} you can read at any time:
SideStatesInitialSet by
userlistening · speaking · thinking · awaylisteningon_user_speech_startedspeaking, on_user_speech_ended / on_user_speech_eoslistening
agentinitializing · idle · listening · thinking · speakinginitializingcall accepted → idle, EOU committed → thinking, on_agent_speech_startedspeaking, on_agent_speech_endedidle
A monotonic turn_idx counter (also exposed on the dispatcher) increments on every committed EOU. The agent_speech_*, llm_token, and audio_out payloads all carry the current turn_idx so a per-turn metric can correlate them.

Sequence for a normal turn

Sequence for a barged-in turn

Full example — wire all seven callbacks

Barge-in detection

The cleanest way to detect a barge-in is to inspect on_agent_speech_ended.interrupted:
For barge-in latency (how fast the agent stopped after the user started talking), pair on_user_speech_started with the next on_agent_speech_ended({"interrupted": True}):

Wiring

The realtime stream handler fires user_speech_started/_ended/_eos and agent_speech_started/_ended automatically on the OpenAI Realtime + Twilio/Telnyx path — no extra setup required. on_llm_token and on_audio_out are exposed on the dispatcher (phone.speech_events) so custom adapters and pipeline-mode integrations can call them. If you are building a custom provider, call phone.speech_events.fire_llm_first_token(...) on your first streamed chunk and phone.speech_events.fire_audio_out(...) on your first synthesized audio buffer; both are idempotent within a turn.

Public exports

ExportTypeUse
SpeechEventsclassThe dispatcher. One instance per Patter (auto-created).
SpeechEventCallbacktype aliasCallable[[dict], Awaitable[None] | None].
ConversationStateSnapshotdict shape{"user": <user_state>, "agent": <agent_state>}.
UserStatestr literal"listening" | "speaking" | "thinking" | "away".
AgentStatestr literal"initializing" | "idle" | "listening" | "thinking" | "speaking".
EouTriggerstr literal"vad_silence" | "semantic_turn_detector" | "manual_commit".

OpenTelemetry attach contract

Every speech-edge event also records a span event on the active call span when PATTER_OTEL_ENABLED=1 and the optional opentelemetry peer dep is installed. When OTel is missing or disabled, the OTel branch is a zero-cost no-op — there is no overhead and no failure.
CallbackSpan event nameSelected attributes
on_user_speech_startedpatter.event.user_speech_startedpatter.audio.offset_ms, patter.vad.confidence
on_user_speech_endedpatter.event.user_speech_endedpatter.speech.duration_ms
on_user_speech_eospatter.event.user_speech_eospatter.eos.trigger, patter.eos.trailing_silence_ms
on_agent_speech_startedpatter.event.agent_speech_startedpatter.turn.idx, patter.tts.provider, patter.engine
on_agent_speech_endedpatter.event.agent_speech_endedpatter.turn.idx, patter.speech.duration_ms, patter.turn.interrupted
on_llm_tokenpatter.event.llm_first_tokengen_ai.request.model, gen_ai.provider.name (per OTel GenAI semconv), patter.turn.idx
on_audio_outpatter.event.tts_first_audiopatter.turn.idx, patter.tts.provider
See Tracing for the OTel installation and exporter setup.

Callback safety

Observer exceptions are caught and logged, never propagated to the live call. A misbehaving callback cannot crash the call or break audio. Errors are logged at WARNING level under the getpatter.events logger with the offending span event name for easy correlation.

Design notes

  • on_user_speech_ended vs. on_user_speech_eos: surfaced as separate events because they are two different signals. silence_gap_ms_max wants the EOU; cross_talk_pct wants the raw VAD edge.
  • on_agent_speech_started vs. on_audio_out: on_audio_out is when TTS bytes arrive in the buffer (warmup metric). on_agent_speech_started is when those bytes hit the carrier wire — what the user actually hears. Subtract the two to measure carrier-side jitter.
  • Idempotency: on_llm_token and on_audio_out fire at most once per turn. The guard is reset on on_user_speech_eos so the next turn re-arms cleanly.