Groq

Groq is a cloud-based AI inference platform available at console.groq.com, renowned for delivering some of the fastest token generation speeds in the industry. Unlike traditional GPU-based inference, Groq uses custom-designed Language Processing Units (LPUs) -- purpose-built hardware engineered specifically for large language model inference. This architectural difference enables dramatically lower latency and higher throughput compared to conventional GPU solutions.

Groq serves a wide range of open-weight models including Meta's Llama 4 and Llama 3 series, Mistral's Mixtral, Google's Gemma, Alibaba's Qwen, and OpenAI's Whisper for speech recognition. It offers a generous free tier with thousands of requests per day, making it accessible for developers, hobbyists, and production workloads alike. The API is fully OpenAI-compatible, so existing applications can switch to Groq with minimal code changes.

• Ultra-Fast Inference - LPU-powered architecture delivers token generation speeds of 200-1,000+ tokens per second, far exceeding typical GPU-based solutions. Some models reach up to 1,000 tps.
• Custom LPU Architecture - Language Processing Units are purpose-built for LLM inference, eliminating the bottlenecks of traditional GPU architectures. Designed for deterministic, low-latency compute.
• Generous Free Tier - Thousands of requests per day at no cost with the Developer plan. No credit card required to start. Rate limits scale with usage plan.
• OpenAI-Compatible API - Drop-in replacement for OpenAI's API format. Use the same SDKs, libraries, and code patterns. No vendor lock-in for your integrations.
• Multi-Modal Support - Text generation, speech-to-text (Whisper), text-to-speech, OCR/image recognition, and reasoning under a single API.
• Structured Outputs - Guarantee model responses conform to JSON schemas with strict or best-effort modes for reliable data extraction.
• Prompt Caching - Automatically cache and reuse common prompt prefixes to reduce latency and cost on repeated queries.
• LoRA Inference - Run fine-tuned LoRA adapters on supported models for custom behavior without deploying separate endpoints.
• Compound AI Systems - Build agentic workflows with built-in tools, MCP connectors, and composable AI system architectures.
• Google Workspace Connectors - MCP-based connectors for Gmail, Google Calendar, and Google Drive enable AI agents to interact with productivity tools.

Groq hosts a curated selection of open-weight models optimized for LPU inference. Models are categorized into Production Models (ready for production workloads) and Preview/Research models. All models run on dedicated LPU hardware.

Production Models

Model	Speed (t/s)	Context	Price/1M Input
Llama 3.1 8B	560 t/s	131K	$0.05
Llama 3.3 70B	280 t/s	131K	$0.59
Llama 4 Maverick 17B	300+ t/s	1M	$0.25
Llama 4 Scout 109B	180 t/s	131K	$0.50
Mixtral 8x7B	480 t/s	32K	$0.27
Gemma 2 9B	520 t/s	8K	$0.08
Gemma 2 27B	340 t/s	8K	$0.27
Qwen 2.5 72B	220 t/s	131K	$0.90
Qwen 2.5 Coder 32B	250 t/s	131K	$0.21
DeepSeek R1 Distill 70B	200 t/s	131K	$0.60

Specialized Models

• Whisper Large V3 - OpenAI's speech-to-text model for transcription and translation. Supports 100+ languages. Priced at $0.111/hour.
• Whisper Large V3 Turbo - Faster variant of Whisper with optimized latency. Priced at $0.04/hour.
• OpenAI GPT-OSS 120B - OpenAI's flagship open-weight model with built-in browser search, code execution, and reasoning. ~500 t/s.
• OpenAI GPT-OSS 20B - Smaller, faster variant of GPT-OSS reaching up to 1,000 t/s ideal for high-throughput applications.
• Llama Guard 3 8B - Content moderation and safety classification model for filtering model inputs and outputs.

Groq offers a comprehensive API that goes beyond standard text generation. All features are accessible through the OpenAI-compatible API format, making integration straightforward for existing applications.

Core Capabilities

Feature	Description
Text Generation	Chat completions with all supported models. Streaming, function calling, system prompts, and multi-turn conversations.
Speech-to-Text	Transcribe audio to text using Whisper Large V3 and Turbo variants. Supports 100+ languages with optional translation.
Text-to-Speech	Convert text to natural-sounding speech audio. Multiple voices and formats supported.
OCR & Image Recognition	Extract text and analyze content from images. Supports document OCR, scene understanding, and visual question answering.
Reasoning	Chain-of-thought reasoning with DeepSeek R1 and other reasoning models. Configurable reasoning effort levels.
Structured Outputs	Guarantee JSON Schema conformance with strict or best-effort modes. Ideal for data extraction and API integration.
Prompt Caching	Automatic caching of common prompt prefixes reduces latency and cost. Seamless cache hit detection.
Tool Use	Define custom tools/functions with JSON Schema parameters. Models call tools in parallel or sequence. Supports local tool calling, remote tools, and MCP integration.
Content Moderation	Built-in safety classification using Llama Guard 3 to filter harmful content in inputs and outputs.
Code Generation	Optimized for code generation with models like Qwen 2.5 Coder. Code execution via built-in tools in compound AI systems.
LoRA Inference	Deploy fine-tuned LoRA adapters on supported base models. Adapters are loaded at inference time with no cold start.

SDK Libraries

Groq provides official SDK libraries for Python and TypeScript, available through pip and npm. Third-party integrations are also available through Vercel AI SDK, LiteLLM, LangChain, and the OpenAI Python/JS libraries (via endpoint change).

Getting started with Groq takes just a few minutes. Sign up for a free account, generate an API key, and start making requests immediately -- no credit card required.

Step 1: Create an Account

Visit console.groq.com and sign up with your Google or GitHub account. The Developer plan includes thousands of free requests per day.

Step 2: Generate an API Key

Navigate to the API Keys page in the console and create a new key. Copy it and set it as an environment variable:

export GROQ_API_KEY=gsk_your_api_key_here

Step 3: Make Your First Request (cURL)

curl https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "What makes LPUs faster than GPUs?"}],
    "temperature": 0.7
  }'

Step 4: Using the Python SDK

pip install groq

from groq import Groq

client = Groq()

completion = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of LPU architecture."}
    ],
    temperature=0.7,
    max_tokens=1024,
)

print(completion.choices[0].message.content)

Streaming Response

stream = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Write a haiku about AI."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

For production deployments, consult the Production Checklist for best practices around error handling, retry logic, rate limiting, and monitoring with Prometheus metrics.

Groq offers three service tiers to match different workload requirements. The tier is selected via the service_tier parameter in the API request body.

Tier	Latency	Availability	Best For
Performance	Lowest latency, guaranteed	Enterprise only	Critical production applications requiring consistent low-latency responses
Flex	Standard (default)	All users	Development, testing, and general-purpose workloads with higher throughput limits
Batch	Best effort, async	All users	Large-scale offline processing, data pipelines, and non-real-time workloads

The Flex tier is the default when no service_tier parameter is specified. It provides high throughput but may occasionally return over-capacity errors during peak demand. The Performance tier guarantees capacity and is available to enterprise customers. The Batch tier is ideal for asynchronous, high-volume workloads where latency is not critical, offering a separate Batch API endpoint for submitting and polling job results.

Example selecting the Flex tier:

curl https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "Summarize the latest release highlights."}],
    "service_tier": "flex"
  }'

Groq supports building compound AI systems -- architectures that combine multiple models, tools, and data sources into a single cohesive agent. These systems are designed for agentic AI workflows where models need to reason, use tools, and coordinate across steps.

Built-In Tools

Groq provides built-in tools that models can invoke automatically during chat completion. These include web search and code execution, enabling models to access real-time information and run code to answer queries. The Groq Compound model system demonstrates this capability, intelligently selecting tools to use based on the user's question.

Tool Use & MCP

The Groq API supports the Model Context Protocol (MCP) for integrating with external services and data sources. This enables AI agents to connect to databases, file systems, cloud APIs, and version control systems. MCP connectors are also available for Google Workspace, allowing agents to interact with Gmail, Google Calendar, and Google Drive directly.

• Web Search - Models can search the web for up-to-date information.
• Code Execution - Run Python code in a sandboxed environment to compute results.
• Remote Tools & MCP - Connect to external APIs and services via MCP protocol.
• Local Tool Calling - Call custom functions running on your own infrastructure.
• Google Workspace Connectors - MCP-based connectors for Gmail, Calendar, and Drive.

Groq Compound Model

Groq Compound is a featured AI system on the platform that intelligently combines multiple open-weight models with built-in tools. It reaches approximately 450 tokens per second and can autonomously decide when to use web search or code execution based on the user's request. This represents a shift from single-model inference to compound, agentic architectures.

Agentic AI Use Cases

• Research Assistants - Models search the web, read documents, and synthesize findings.
• Data Analysis - Connect to databases, query data, run code, and generate visualizations.
• Automated Workflows - Multi-step pipelines with tool use, decision-making, and result aggregation.
• Customer Support Bots - Access knowledge bases, query order systems, and take actions on behalf of users.
• Code Review & Testing - Review pull requests, run tests, and generate reports.

Feature	Groq	OpenRouter	Together AI	Cerebras
Hardware	Custom LPUs	GPU routing (3rd party)	GPU clusters	Wafer-scale engines (WSE)
Token Speed	200-1,000+ t/s	Varies by provider	100-400 t/s	500-1,800+ t/s
Free Tier	Generous (thousands/day)	Limited free models	Limited free tier	Free with daily limits
Model Selection	Curated open models	400+ models (all providers)	200+ open models	Curated selection
API Compatibility	OpenAI-compatible	OpenAI-compatible	OpenAI-compatible	OpenAI-compatible
Speech-to-Text	Yes (Whisper)	No	No	No
Text-to-Speech	Yes	No	No	No
OCR / Image Recognition	Yes	Via model support	Via model support	No
Structured Outputs	Strict + best-effort	Via model	JSON mode on some models	Via model
MCP / Tool Use	Full support (built-in tools, remote MCP)	Function calling only	Function calling only	Function calling
Service Tiers	Performance / Flex / Batch	Single tier + credits	On-demand / Batch	On-demand
LoRA Adapters	Supported	No	Via custom models	No