Skip to main content

Groq LLM

GroqLLM plugs Groq’s OpenAI-compatible Chat Completions API at https://api.groq.com/openai/v1 into Patter’s pipeline mode. Groq’s LPU inference engine serves Llama models at very high throughput with low time-to-first-token, making it a strong pick when latency matters more than long-context reasoning. The provider is a thin wrapper around OpenAILLMProvider with a Groq-specific base URL — every OpenAI sampling kwarg (response_format, parallel_tool_calls, tool_choice, seed, top_p, frequency_penalty, presence_penalty, stop, temperature, max_tokens) is forwarded to chat.completions.create automatically.

Install

Usage

The namespaced import (from getpatter.llm import groq / import * as groq from "getpatter/llm/groq") auto-resolves the API key from GROQ_API_KEY and exposes a uniform LLM class — the same pattern Patter uses for STT and TTS namespaces.
Plug it into an agent:

Supported models

Pricing in USD per 1M tokens. Availability depends on account tier — Groq’s free tier rate-limits more aggressively than the paid plans.
ModelInputOutputNotes
llama-3.3-70b-versatile (default)$0.59$0.79General-purpose Llama 3.3, long context.
llama-3.1-8b-instant$0.05$0.08Cheapest fast option.
llama-3.3-70b-specdecn/an/aSpeculative decoding variant.
llama3-70b-8192n/an/aLlama 3, 8K context.
llama3-8b-8192n/an/aLlama 3, 8K context.
mixtral-8x7b-32768n/an/aMixtral MoE, 32K context.
gemma2-9b-itn/an/aGoogle Gemma 2 instruct.
Models without listed rates are available on the API but aren’t yet pinned to a LLM_PRICING entry — pass pricing={...} overrides if your dashboard needs cost figures for them.

Environment variables

VariableRequiredNotes
GROQ_API_KEYyesAuto-loaded when api_key / apiKey is omitted.

Options

OptionDefaultNotes
api_key / apiKeyNoneReads from GROQ_API_KEY when omitted.
model"llama-3.3-70b-versatile"Any Groq chat model id.
base_url / baseUrlhttps://api.groq.com/openai/v1Override the Groq endpoint (rarely needed).
temperature, max_tokens, top_p, seed, frequency_penalty, presence_penalty, stop, response_format, parallel_tool_calls, tool_choiceunsetAll forwarded to chat.completions.create. See the Groq API docs for accepted values.

Notes

  • Groq returns the standard OpenAI Chat Completions stream shape, so tool calls, JSON mode, and seeded sampling all work without provider-specific code.
  • Time-to-first-token on Groq’s LPU is typically < 200 ms for the 70B model and < 100 ms for the 8B model — well below most TTS startup latency.
  • Long-context calls (32K+) use Mixtral; everything else fits comfortably in the Llama 3.3 context.