LocalAI

LocalAI is an open-source, self-hosted AI engine created by Ettore Di Giacinto (mudler). It provides a drop-in replacement for OpenAI's API that runs entirely on your own infrastructure, supporting LLMs, image generation, audio transcription, text-to-speech, video generation, and embeddings — all behind a single, unified API endpoint.

LocalAI uses a composable backend architecture: each model type (LLM, vision, voice, image) wraps a best-in-class open-source engine in its own container image, pulled only when needed. Core backends include llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX, kokoro, and parakeet.cpp. The project is MIT-licensed and hosted on GitHub at github.com/mudler/LocalAI.

LocalAI can be installed via Docker, Homebrew, or built from source. Docker is the recommended approach for most users.

Docker (Recommended)

docker run -d -p 8080:8080 \
  -v $PWD/models:/models \
  -v $PWD/config:/config \
  -e DEBUG=true \
  localai/localai:latest

macOS (Homebrew)

brew install localai

A native macOS .dmg installer is also available from the GitHub releases page.

Kubernetes

helm repo add localai https://localai.io/helm-charts/
helm install localai localai/localai

Modality	Engine/Backend	API
LLMs	llama.cpp, vLLM, MLX	OpenAI Chat Completions
Image Generation	Stable Diffusion, FLUX	OpenAI Images, Anthropic
Audio (STT)	whisper.cpp, Parakeet	OpenAI Audio, ElevenLabs
Text-to-Speech	Kokoro, Piper, Bark	OpenAI TTS, ElevenLabs
Video	Stable Video Diffusion	OpenAI-compatible
Embeddings	llama.cpp, BERT models	OpenAI Embeddings

LocalAI provides drop-in API compatibility with multiple major providers, so you can use existing tools and libraries without modification. The OpenAI API is the primary compatibility layer, covering chat completions, embeddings, image generation, audio transcription, text-to-speech, and model listing. Anthropic and ElevenLabs API compatibility are also supported for relevant modalities.

This means you can point any OpenAI SDK or tool at your LocalAI instance by simply changing the base_url. Applications like chatbots, IDEs, automation tools, and custom scripts work without any code changes. LocalAI also supports tool calling / function calling, structured output (JSON mode), streaming, and vision inputs.

LocalAI's composable architecture keeps the core small while each backend wraps a specialized engine in its own container image. Backends are pulled on demand when a model requests them, so you never install anything you don't use. Supported hardware includes NVIDIA GPUs (CUDA), AMD GPUs (HIP), Intel GPUs (SYCL), Apple Silicon (Metal), and CPU-only. The vLLM backend supports high-throughput serving with PagedAttention.

Backends are built in Go or Python and communicate with the core via gRPC, allowing each backend to be developed and maintained independently. The open backend interface means you can write custom backends in any language.

LocalAI includes built-in autonomous AI agents with tool use capabilities, Retrieval-Augmented Generation (RAG), MCP server integration, and a skills system. Agents can use tools like web search, file operations, code execution, and database queries. RAG supports multiple vector databases including ChromaDB, Qdrant, and PostgreSQL with pgvector.

• Autonomous agents — Goal-driven agents with tool use and self-directed task execution
• RAG pipeline — Document ingestion, chunking, embedding, and retrieval
• MCP support — Connect to MCP servers for extended capabilities
• Skills — Composable, reusable agent behavior modules
• Fine-tuning — Built-in fine-tuning and quantization workflows

LocalAI is configured through a YAML-based model configuration system. Each model has its own definition file specifying the backend, model file path, prompt template, and inference parameters. The web UI provides a management dashboard for monitoring usage, configuring API keys, setting user quotas, and managing RBAC.

Feature	Description
Multi-user Auth	API key authentication with per-user quotas and RBAC
Usage Metrics	Per-user token and request tracking with dashboards
Web UI	Browser-based management and chat interface
Model Gallery	Curated list of tested models at models.localai.io
P2P Discovery	Automatic peer discovery for distributed deployments