> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getpatter.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Hermes Agent

> Give your Hermes Agent a phone with Patter — make Hermes the voice the caller talks to, connect the Patter MCP server so Hermes can place calls, or let an in-call Patter agent consult Hermes as its brain.

[Hermes Agent](https://github.com/NousResearch/hermes-agent) is a self-hosted AI
assistant that grows with you through modular skills and persistent memory. If you
already run Hermes, Patter adds phone calling in three directions:

* **Direction A — Patter is the voice shell, Hermes is the brain.** Patter answers the
  phone — carrier, speech-to-text, turn-taking, barge-in, text-to-speech — and routes
  every conversation turn to your Hermes agent as the LLM, using the new
  [`HermesLLM`](#what-if-hermes-could-pick-up-the-phone) pipeline provider. **This is
  the headline integration**; the rest of this page covers the other two directions.
* **Direction B — Hermes places calls.** Connect [Patter](https://github.com/PatterAI/Patter)
  as an MCP server and Hermes gains tools to dial out, read transcripts, end calls,
  and inspect metrics.
* **Direction C — Patter consults Hermes on demand.** During a live call, a Patter agent
  can reach back to Hermes for deeper reasoning, fresh information, or a decision the
  in-call prompt can't make on its own.

Pick one direction or wire up several — they are independent.

## What if Hermes could pick up the phone?

You already have a Hermes agent with memory, skills, and tools. Direction A puts it on
the end of a phone line: a caller dials your number, and they are *talking to Hermes* —
not to a separate voice bot that occasionally asks Hermes for help. Patter is the **voice
shell**: it owns everything that has to happen in real time (the carrier leg, transcribing
the caller, deciding when a turn ends, handling barge-in, and speaking the reply), and for
the one thing that requires intelligence — *what to say next* — it calls your Hermes agent.

This is the same shape as wiring a custom LLM into a hosted voice platform, except **you
own the voice layer** and run it next to Hermes. Because Hermes is reached over its
OpenAI-compatible HTTP API (`POST /v1/chat/completions`), the new `HermesLLM` provider
plugs straight into Patter's **pipeline mode** as the LLM stage.

### Architecture

```
Caller (any number)
  │
  ▼
Twilio / Telnyx DID  ──  Patter verifies the carrier signature at the public boundary
  │
  ▼
PATTER VOICE SHELL  (pipeline mode)
  · STT  — transcribes the caller
  · turn-taking / VAD / barge-in  — decides when the caller is done
  · TTS  — speaks Hermes's reply
  │
  │  each turn  →  HermesLLM  →  POST http://127.0.0.1:8642/v1/chat/completions
  ▼
HERMES AGENT RUNTIME  (tools · memory · skills run here, before it replies)
  │
  ▼
reply text  →  Patter speaks it  →  back to the caller
```

Hermes runtimes execute tools, recall memory, and run skills *before* they answer, so a
single turn can take **30–90 s**. That is why `HermesLLM` defaults to a **120 s** request
timeout (the generic provider's 60 s, raised for the preset) instead of the short ceiling
used for raw inference providers — a turn that runs a tool isn't cut off mid-thought.

Because a tool-running turn can leave the caller in **silence** for several seconds, the
agent supports an opt-in spoken **filler**: set `long_turn_message` / `longTurnMessage`
(with `long_turn_message_after_s` / `longTurnMessageAfterS`, default 4 s) and Patter speaks
a short line if no audio has reached the caller yet by then. It fires once per turn, only
on slowness, and never overlaps the real reply. (A separate `llm_error_message` /
`llmErrorMessage` covers the gateway-down / timeout **error** case.)

```python theme={null}
agent = phone.agent(
    stt=DeepgramSTT(),
    llm=HermesLLM(),
    tts=ElevenLabsTTS(),
    long_turn_message="One moment, let me check that.",
    long_turn_message_after_s=4,
)
```

<Note>
  **Where the session lives.** Hermes is **stateless** and keys continuity off
  **HTTP headers**, not the OpenAI `user` field. Each phone call maps to **one** Hermes
  session: Patter sends `X-Hermes-Session-Id: patter-call-<call_id>` on every turn, so
  multi-turn session and transcript continuity inside a call works without any extra
  wiring. This is on by default for `HermesLLM`.

  For **long-term memory scoping** across calls, set `session_key` (Python) /
  `sessionKey` (TypeScript) on `HermesLLM` — Patter then also sends
  `X-Hermes-Session-Key: <your value>`, which tells Hermes which memory scope this
  caller belongs to. It is **off by default** (no header sent) and is opt-in:

  ```python theme={null}
  llm = HermesLLM(session_key="customer-42")   # scopes long-term memory to this caller
  ```

  ```ts theme={null}
  const llm = new HermesLLM({ sessionKey: 'customer-42' });
  ```

  For **per-caller memory without storing the raw phone number**, derive the key from a
  caller hash instead of a static value — `HermesLLM(session_key_from="caller_hash")` /
  `new HermesLLM({ sessionKeyFrom: 'caller_hash' })` emits
  `X-Hermes-Session-Key: patter-caller-<hash>` (SHA-256, 16 hex chars), so Hermes
  remembers a caller across calls while the raw number never reaches the wire or the
  logs. For a custom scheme, pass `session_key_factory` / `sessionKeyFactory`, a callback
  that receives a `SessionContext` (`call_id` / `caller` / `callee` / `caller_hash`) and
  returns the scope value (a falsy return omits the header for that call).

  (Patter also still sends `user=patter-call-<call_id>` for upstream-log correlation,
  but that field is **not** what drives the Hermes session — the headers are.)
</Note>

### Prerequisites for the voice shell

```bash theme={null}
# Python SDK
pip install getpatter

# TypeScript SDK
npm install getpatter
```

You also need:

* A running **Hermes agent runtime** with its OpenAI-compatible API server enabled
  (next section).
* A **phone number** from Twilio or Telnyx, and the matching carrier credentials.
* An **STT** and a **TTS** provider for the pipeline (e.g. Deepgram STT + ElevenLabs TTS).
  Patter handles the audio; Hermes only ever sees text.

### Setting up the Hermes gateway

Hermes exposes an OpenAI-compatible HTTP API. Enable it and bind it to loopback so only
processes on the same machine — including Patter — can reach it:

```bash theme={null}
export API_SERVER_ENABLED=true
export API_SERVER_HOST=127.0.0.1          # loopback only — never expose to the internet
export API_SERVER_PORT=8642               # the HermesLLM default base_url targets :8642
export API_SERVER_KEY="choose-a-strong-key"
export API_SERVER_MODEL_NAME="hermes-agent"   # the model id HermesLLM sends; optional
```

Start the gateway (per your Hermes install), then confirm it answers on loopback before
wiring Patter to it:

```bash theme={null}
curl -s http://127.0.0.1:8642/v1/models \
  -H "Authorization: Bearer $API_SERVER_KEY"
```

A JSON list of models (including your `hermes-agent`) means the gateway is up. If `curl`
hangs or refuses the connection, fix that before going further — Patter can't reach a
gateway that isn't listening.

<Note>
  **`HermesLLM` reads these env vars for you.** With `API_SERVER_KEY` and
  (optionally) `API_SERVER_MODEL_NAME` set, you can construct `HermesLLM()` with no
  arguments — it defaults `base_url` to `http://127.0.0.1:8642/v1`, the api key from
  `API_SERVER_KEY`, and the model from `API_SERVER_MODEL_NAME` (falling back to
  `hermes-agent`).
</Note>

### Running Patter locally

Build a pipeline-mode agent whose LLM is `HermesLLM`. Patter wraps the carrier, STT, and
TTS around it; Hermes is the brain for every turn.

#### Python example

```python theme={null}
from getpatter import Patter, HermesLLM, DeepgramSTT, ElevenLabsTTS, Twilio

phone = Patter(
    carrier=Twilio(),                    # reads TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN
    phone_number="+15550001234",
)

# Passing stt + tts (and no engine=) selects pipeline mode: STT -> LLM -> TTS.
agent = phone.agent(
    system_prompt=(
        "You are the front desk. Keep replies short and speakable; "
        "you have phone callers, not chat users."
    ),
    stt=DeepgramSTT(),                   # reads DEEPGRAM_API_KEY
    llm=HermesLLM(),                     # base_url http://127.0.0.1:8642/v1, 120 s timeout
    tts=ElevenLabsTTS(),                 # reads ELEVENLABS_API_KEY
)

phone.serve(agent)                       # listens for inbound calls
```

`HermesLLM()` with no arguments is the common case — every default comes from the env. To
target a remote Hermes or pin a model explicitly:

```python theme={null}
llm = HermesLLM(
    base_url="http://hermes-host.internal:8642/v1",
    model="hermes-agent",
    api_key="...",          # or leave unset to read API_SERVER_KEY
    timeout=120.0,          # raise further if your turns run long
)
```

#### TypeScript example

```ts theme={null}
import { Patter, HermesLLM, DeepgramSTT, ElevenLabsTTS, Twilio } from 'getpatter';

const phone = new Patter({
  carrier: new Twilio(),                 // reads TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN
  phoneNumber: '+15550001234',
});

// Passing stt + tts (and no engine) selects pipeline mode: STT -> LLM -> TTS.
const agent = phone.agent({
  systemPrompt:
    'You are the front desk. Keep replies short and speakable; ' +
    'you have phone callers, not chat users.',
  stt: new DeepgramSTT(),                // reads DEEPGRAM_API_KEY
  llm: new HermesLLM(),                  // baseUrl http://127.0.0.1:8642/v1, 120 s timeout
  tts: new ElevenLabsTTS(),              // reads ELEVENLABS_API_KEY
});

await phone.serve(agent);
```

To target a remote Hermes or pin a model explicitly:

```ts theme={null}
const llm = new HermesLLM({
  baseUrl: 'http://hermes-host.internal:8642/v1',
  model: 'hermes-agent',
  apiKey: process.env.API_SERVER_KEY,    // or omit to read it automatically
  timeout: 120,                          // seconds; raise if turns run long
});
```

### Connecting Twilio or Telnyx

The voice shell answers a real phone number. `phone.serve(agent)` exposes a webhook that
the carrier calls when a number rings; point your number's voice webhook at Patter's
public URL.

* **Local dev:** run Patter behind a tunnel (a stable `cloudflared` hostname is best — an
  ephemeral quick-tunnel rotates its URL on every restart) and set that URL as the number's
  voice webhook.
* **Twilio:** in the [Twilio console](https://console.twilio.com/us1/develop/phone-numbers/manage/incoming),
  set the number's "A call comes in" webhook to your Patter URL. Patter verifies
  `X-Twilio-Signature` on every inbound request.
* **Telnyx:** in the [Telnyx portal](https://portal.telnyx.com/#/app/numbers/my-numbers),
  point the number's voice connection at your Patter URL. Patter verifies the Ed25519
  signature.

Only **Patter** is exposed to the carrier. Hermes stays on loopback (see Security notes).

### Production deployment

For an always-on line, run Patter and Hermes **on the same machine** and keep the gateway
private:

* Bind the Hermes gateway to `127.0.0.1:8642` (`API_SERVER_HOST=127.0.0.1`) — Patter
  reaches it over loopback, so it never needs to be tunnelled or ngrok-exposed.
* Expose **only** Patter's carrier webhook to the internet — via a stable production
  webhook URL or a persistent `cloudflared` tunnel with a fixed hostname.
* Set a strong `API_SERVER_KEY` even on loopback; defence in depth.
* Pin the SDK version and your provider/carrier keys via environment variables, never in
  source.

### Security notes

* **Hermes does not need to be on the public internet.** When Patter and Hermes run on the
  same box, Patter reaches Hermes on `127.0.0.1:8642` — only Patter is exposed to the
  carrier. This is **safer than the hosted custom-LLM path**, where your brain endpoint has
  to be publicly reachable for the platform's cloud to call it. Here the brain stays
  loopback-only and the only public surface is the carrier webhook, which verifies the
  carrier signature at the boundary.
* **Use a strong `API_SERVER_KEY`.** Even on loopback, set a real key; `HermesLLM` sends it
  as a bearer token and Patter never logs it.
* **Keep the prompt voice-safe.** Phone replies are spoken, not read — ask the agent for
  short, plain sentences and no markdown, lists, or code blocks. Long replies bloat the
  voice context and slow the next turn.
* **PII and recording consent.** Calls may carry personal information; if you record or
  store transcripts, follow your jurisdiction's consent rules (two-party-consent states,
  GDPR, etc.). Treat transcripts as PII and don't log full caller numbers (Patter logs only
  the last four digits by default).

<Note>
  **Generic OpenAI-compatible runtimes.** `HermesLLM` is a thin preset over the generic
  `OpenAICompatibleLLM` provider, which drives *any* OpenAI-compatible chat endpoint —
  Hermes, OpenClaw, Ollama, vLLM, LM Studio, or a custom gateway. To point the voice shell
  at one of those instead, use `OpenAICompatibleLLM(base_url=..., model=...)` directly. See
  the [OpenClaw page](/integrations/openclaw#openclaw-as-the-primary-llm-voice-shell) for
  the OpenClaw preset and the generic-runtime note.
</Note>

## Prerequisites

Patter's MCP server runs **locally** — it is not published to npm or PyPI. Clone and
build it from [`PatterAI/patter-mcp`](https://github.com/PatterAI/patter-mcp):

```bash theme={null}
git clone https://github.com/PatterAI/patter-mcp
cd patter-mcp
npm install
npm run build
npm start          # Streamable HTTP on http://localhost:3000/mcp
```

It can also run over stdio (see Direction A below). Provider and carrier keys are
read from the server's environment: `OPENAI_API_KEY`, `ELEVENLABS_API_KEY`,
`TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`, `TELNYX_API_KEY`, and friends.
See the [patter-mcp README](https://github.com/PatterAI/patter-mcp#readme) for full setup.

***

## Direction B — Hermes places calls via Patter

Hermes is a native [Model Context Protocol](https://modelcontextprotocol.io) client.
Register the Patter MCP server under the top-level `mcp_servers` key in
`~/.hermes/config.yaml`. The seven Patter tools then surface to Hermes:
`make_call`, `call_third_party`, `get_calls`, `get_transcript`, `end_call`,
`get_metrics`, and `configure_inbound` (plus a set of MCP resources).

### HTTP recipe

With the server running locally (`npm start`) on
`http://localhost:3000/mcp`, point Hermes at it:

```yaml theme={null}
# ~/.hermes/config.yaml
mcp_servers:
  patter:
    url: "http://localhost:3000/mcp"
    # Raise the per-server timeout well above the 120s default so a long
    # wait:true call (which blocks until the call ends) is not killed.
    timeout: 600
    tools:
      include:
        - make_call
        - call_third_party
        - get_calls
        - get_transcript
        - end_call
        - get_metrics
        - configure_inbound
```

### stdio recipe

Prefer to let Hermes spawn the server itself? Point it at your local build
(`patter-mcp/dist/index.js` from the clone above) with the `--stdio` flag and pass
the provider keys through `env`:

```yaml theme={null}
# ~/.hermes/config.yaml
mcp_servers:
  patter:
    command: "node"
    args:
      - "/absolute/path/to/patter-mcp/dist/index.js"
      - "--stdio"
    env:
      OPENAI_API_KEY: "${OPENAI_API_KEY}"
      ELEVENLABS_API_KEY: "${ELEVENLABS_API_KEY}"
      TWILIO_ACCOUNT_SID: "${TWILIO_ACCOUNT_SID}"
      TWILIO_AUTH_TOKEN: "${TWILIO_AUTH_TOKEN}"
    tools:
      include:
        - make_call
        - call_third_party
        - get_calls
        - get_transcript
        - end_call
        - get_metrics
        - configure_inbound
```

<Warning>
  **Raise the timeout for `wait:true` calls.** Hermes defaults its per-server MCP
  timeout to 120 seconds. A phone call that runs `make_call` with `wait: true`
  blocks until the call ends, which can easily exceed two minutes. Set
  `timeout: 600` (ten minutes) on the `patter` server, or your long calls get cut
  off mid-conversation and Hermes never receives the transcript.
</Warning>

### Example task

Ask Hermes something like:

> *"Call +15550001234 and ask whether they have an opening tomorrow morning.
> Wait for the call to finish, then summarise what they said."*

Hermes calls `make_call` with `wait: true`. The tool blocks until the call
completes and returns the outcome plus the full transcript in one response, so
Hermes can summarise without polling `get_transcript` separately:

```json theme={null}
{
  "name": "make_call",
  "arguments": {
    "to": "+15550001234",
    "wait": true,
    "agent": {
      "system_prompt": "Ask whether they have an opening tomorrow morning."
    }
  }
}
```

***

## Direction C — Patter consults Hermes on demand

Here the roles are split differently: Patter runs the call with a *local* in-call agent and
only reaches Hermes when that agent decides it needs help — it **consults Hermes** over
HTTP. Use this when most turns are simple and only a few need Hermes's full reasoning;
Direction A (above) routes *every* turn to Hermes. This is the *dispatch + consult* pattern
described
on the [Python consult page](/python-sdk/consult) and the
[TypeScript consult page](/typescript-sdk/consult) — Hermes stays off the
per-turn path and is only consulted when the in-call agent decides it needs to.

### 1. Enable the Hermes API server

Hermes exposes an OpenAI-compatible HTTP API. Enable it with environment
variables; it listens on `127.0.0.1:8642` by default:

```bash theme={null}
export API_SERVER_ENABLED=true
export API_SERVER_KEY="choose-a-strong-key"
```

Hermes then serves `POST /v1/chat/completions` (and `POST /v1/runs` for async +
SSE), returning the standard OpenAI shape with the answer in
`choices[0].message.content`.

### 2. Run a tiny adapter

Patter's consult feature POSTs a body shaped like
`{ "request": "...", "call_id": "...", "caller": "...", "callee": "..." }` and
reads the reply back from a `reply` field (it also accepts `response` / `text` /
`result` / `answer` / `message`). Hermes speaks OpenAI's chat-completions shape.
A roughly 30-line adapter bridges the two: map `request` into an OpenAI
`messages` array, forward to Hermes, and map `choices[0].message.content` back to
`{ "reply": ... }`.

```python theme={null}
# consult_adapter.py — bridges Patter consult -> Hermes chat completions
import os
import httpx
from fastapi import FastAPI, Request

app = FastAPI()

HERMES_URL = os.environ["HERMES_API_URL"]          # e.g. http://hermes-host:8642
HERMES_KEY = os.environ["API_SERVER_KEY"]

@app.post("/consult")
async def consult(req: Request) -> dict:
    body = await req.json()
    user_request = body.get("request", "")

    payload = {
        "model": "hermes",
        "messages": [{"role": "user", "content": user_request}],
    }
    headers = {"Authorization": f"Bearer {HERMES_KEY}"}

    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(
            f"{HERMES_URL}/v1/chat/completions",
            json=payload,
            headers=headers,
        )
        resp.raise_for_status()
        data = resp.json()

    reply = data["choices"][0]["message"]["content"]
    return {"reply": reply}
```

### 3. Point Patter's consult URL at the adapter

```python theme={null}
from getpatter import Patter, ConsultConfig

phone = Patter(carrier=...)

agent = phone.agent(
    system_prompt="You are front-desk support. If you can't answer directly, "
                  "consult your back-office agent.",
    stt=...,
    tts=...,
    consult=ConsultConfig(
        url="https://consult-adapter.example.com/consult",
        headers={"Authorization": "Bearer ${ADAPTER_TOKEN}"},
        timeout_s=30.0,
    ),
)

phone.serve(agent)
```

When `consult` is set, Patter auto-injects a `consult_agent` tool into the agent
(Realtime and Pipeline modes). The model calls it with a single `request` string,
the adapter forwards that to Hermes, and the answer is spoken back to the caller.

<Warning>
  **Loopback / SSRF caveat.** By default Patter's consult URL validator rejects
  loopback and private hosts (`127.0.0.1`, `localhost`, `10.x`, `172.16–31.x`,
  `192.168.x`, link-local) as an SSRF guard, and the Hermes API server binds to
  `127.0.0.1:8642`. When everything runs on one box, opt in with `allow_loopback`
  (Python) / `allowLoopback` (TypeScript) so the consult URL can point at the local
  adapter directly. The flag relaxes the loopback / private / link-local host checks
  **for the consult URL only** — every other webhook path stays strict, and
  non-HTTP(S) schemes are still rejected even with the flag on. Only enable it for a
  URL you control; the consult URL is your own configuration, not caller-derived
  input.

  ```python theme={null}
  consult=ConsultConfig(
      url="http://localhost:8000/consult",      # local adapter on loopback
      headers={"Authorization": "Bearer ${ADAPTER_TOKEN}"},
      timeout_s=30.0,
      allow_loopback=True,
  )
  ```

  ```ts theme={null}
  consult: {
    url: 'http://localhost:8000/consult',       // local adapter on loopback
    headers: { Authorization: `Bearer ${process.env.ADAPTER_TOKEN}` },
    timeoutMs: 30_000,
    allowLoopback: true,
  }
  ```

  Prefer to keep the strict default? Expose the adapter on a routable interface or
  hostname rather than `localhost`, and keep the adapter — not Hermes — as the public
  hop. The adapter still reaches Hermes over loopback.
</Warning>

## What's next

* **Read the [Concepts](/concepts) page** to understand Patter's voice modes
  (Realtime end-to-end vs. Pipeline composed STT + LLM + TTS) — the voice shell
  (Direction A) is a pipeline-mode agent with `HermesLLM` as the LLM stage.
* **OpenClaw** runs the same way — see
  [OpenClaw as the primary LLM](/integrations/openclaw#openclaw-as-the-primary-llm-voice-shell)
  for the `OpenClawLLM` preset and the generic `OpenAICompatibleLLM` provider
  (Ollama / vLLM / LM Studio).
* **Consult feature reference**: [Python](/python-sdk/consult) ·
  [TypeScript](/typescript-sdk/consult).
* **Get a phone number**: [Twilio console](https://console.twilio.com/us1/develop/phone-numbers/manage/incoming)
  or [Telnyx portal](https://portal.telnyx.com/#/app/numbers/my-numbers).

## Need help?

* Patter SDK source: [https://github.com/PatterAI/Patter](https://github.com/PatterAI/Patter)
* Patter MCP server: [https://github.com/PatterAI/patter-mcp](https://github.com/PatterAI/patter-mcp)
* Patter Discord: [https://discord.gg/patter](https://discord.gg/patter)
* Hermes Agent docs: [https://github.com/NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent)