NVIDIA NIM

NVIDIA NIM (NVIDIA Inference Microservices) is a production-grade inference platform hosted at build.nvidia.com that provides an OpenAI-compatible API endpoint for hundreds of open and proprietary AI models. It solves the core challenge of deploying and scaling large language models (LLMs) by offering pre-optimized inference microservices that run on NVIDIA's DGX Cloud infrastructure or your own GPU hardware.

Unlike managing your own inference stack from scratch, NIM eliminates the operational burden of model serving, GPU optimization, security patching, and infrastructure management. It provides validated containers with curated model weights, optimal quantization profiles, and direct alignment with upstream inference engines like vLLM. Whether you need a quick API call for prototyping or a self-hosted deployment behind your own firewall, NIM delivers a unified experience across both workflows.

NIM serves as a bridge between the fast-moving open-source model ecosystem and enterprise production requirements. It offers two main offerings: NIM Day 0 for rapid access to newly released models with functional validation, and NIM Certified for long-lived production deployments with CVE patching, security updates, FedRAMP readiness, and NVIDIA AI Enterprise support.

• OpenAI-Compatible API - Drop-in replacement for any OpenAI SDK or tooling. Use the same chat completions, embeddings, and image generation endpoints you already know.
• 140+ Models - Access hundreds of open models from publishers including NVIDIA, Meta, Mistral AI, Google, Qwen, DeepSeek, Z.ai, Moonshot AI, and more. Covers LLMs, vision models, speech models, embedding models, and domain-specific models.
• 77 Free Endpoints - Over 75 models available with free serverless API endpoints for development and prototyping. No credit card required to get started.
• Self-Hosted Containers - Download and deploy NIM containers on your own GPU infrastructure (H100, B200, A100, L40S, RTX, DGX Spark, DGX Station). Run behind your firewall with full control.
• Enterprise Security - Continuous CVE patching, security updates, OSRB compliance, and FedRAMP-ready branches for government and regulated use cases.
• Multi-Provider Inference - Partner inference providers including Deepinfra, OpenRouter, Together AI, GMI Cloud, and Bitdeer extend model availability beyond NVIDIA's own infrastructure.
• Agentic Skills - Pre-built capabilities (143 AI/ML skills, 25 accelerated computing skills, 37 physical AI skills, 10 developer tool skills) that your agent can call for specialized tasks.
• vLLM-Powered - Built on vLLM with direct upstream alignment, giving you access to the latest optimizations, CUDA versions, and hardware support without abstraction layer overhead.
• Model-Free NIMs - Deploy custom-trained or fine-tuned models stored in your own infrastructure (S3, Hugging Face, local storage) using flexible model-free containers.
• Blueprints and Playbooks - Ready-to-use application blueprints for common AI workflows including RAG, agentic AI, code generation, and visual design.

NIM follows a microservice architecture where each model (or family of models) is packaged into a containerized inference microservice. These containers are built on vLLM and include model weights, optimal quantization profiles, and validated runtime configurations. The architecture is organized into three primary components:

• nim-llm (Orchestration Layer) - The entry point that orchestrates startup, manages configuration priorities (CLI flags, environment variables, runtime configs), and injects enterprise features like custom middleware and LoRA adapters.
• nimlib (Profile and Model Management) - Handles model licensing, hardware-aware profile selection, model downloading, and NIM-specific management API endpoints (health checks, readiness probes, model metadata).
• Inference Engine (vLLM) - The core engine that executes model inference and provides native OpenAI-compatible API endpoints including chat completions, embeddings, and tokenization.

You interact with NIM through two primary modes: serverless API (using NVIDIA's hosted infrastructure at build.nvidia.com) or self-hosted containers (deploying NIM on your own GPU infrastructure via Docker, Kubernetes, or directly on supported workstations). Both modes expose the same OpenAI-compatible API, making it trivial to switch between them.

NIM supports two container modalities: Model-Specific NIMs include a model-specific manifest with curated weights and validated configurations for a seamless out-of-the-box experience. Model-Free NIMs generate their model manifest at runtime, supporting remote repositories (NGC, Hugging Face, Amazon S3, Google Cloud Storage) and local directories for custom or fine-tuned models.

NVIDIA NIM hosts 141 models across a wide range of categories, with 77 free endpoints and 108 downloadable containers. Models are published by NVIDIA (76 models), Meta (11), Google (6), Mistral AI (6), Qwen (5), and many others. The catalog spans the following use case categories:

Category	Model Count	Examples
Large Language Models	50+	Nemotron, Llama, Mistral, Qwen, DeepSeek, GLM
Reasoning Models	15+	DeepSeek-V4-Pro, Nemotron-3-Ultra, Cosmos3-Nano-Reasoner
Vision Models	15+	Image-to-Text, Visual Design, Image Editing
Speech Models	9+	Speech-to-Text, Text-to-Speech
Embedding Models	10+	Retrieval, RAG-optimized embeddings
Drug Discovery	13	BioNeMo, MolMIM, ESM-2
Code Generation	8+	Code Llama, DeepSeek Coder, StarCoder
Simulation & Physical AI	10+	Cosmos, Physical AI models
Climate & Weather	5+	FourCastNet, GraphCast
Safety & Moderation	5+	Content moderation, guardrail models

Notable models available on NIM include NVIDIA Nemotron-3-Ultra-550B-A55B (hybrid Mamba-Transformer MoE with 1M context), DeepSeek-V4-Pro (1M-token MoE for coding), Kimi K2.6 (1T multimodal MoE), GLM-5.2 (agentic workflows), and Llama 3.x family models from Meta.

Getting started with NVIDIA NIM is free and requires no credit card. Follow these steps to make your first API call:

Step 1: Sign Up and Get an API Key

Visit build.nvidia.com, sign up for a free account, and navigate to your API key management page to generate a key. Free tier includes access to over 75 free model endpoints with rate limits suitable for development and prototyping.

Step 2: Make a curl Request

Here is an example using curl to call a chat completion model:

curl -X POST "https://api.build.nvidia.com/v1/chat/completions" \
  -H "Authorization: Bearer $NVIDIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3-ultra-550b-a55b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain what NVIDIA NIM is in one sentence."}
    ],
    "temperature": 0.7,
    "max_tokens": 256
  }'

Step 3: Python Example

Using the OpenAI Python SDK (fully compatible):

from openai import OpenAI

client = OpenAI(
    base_url="https://api.build.nvidia.com/v1",
    api_key="YOUR_NVIDIA_API_KEY"
)

completion = client.chat.completions.create(
    model="nvidia/nemotron-3-ultra-550b-a55b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is NVIDIA NIM?"}
    ],
    temperature=0.7,
    max_tokens=256
)

print(completion.choices[0].message.content)

Step 4: Self-Host with Docker

To deploy a NIM container on your own GPU infrastructure:

# Pull and run a NIM container (requires NVIDIA GPU + NVIDIA Container Toolkit)
docker run -d --gpus all \
  -e NVIDIA_API_KEY=$NVIDIA_API_KEY \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

Once the container is running, it exposes the same OpenAI-compatible API at http://localhost:8000/v1.

NVIDIA NIM offers a flexible pricing model that scales from free prototyping to enterprise production deployments:

Tier	Cost	Includes
Free Tier	$0	77+ free serverless API endpoints with rate limits. No credit card required. Ideal for development, prototyping, and learning.
Paid API	Usage-based	Pay-per-token pricing for premium models, higher rate limits, and priority access. Pricing varies by model and is published per endpoint.
Partner Endpoints	Provider-set	44 models served through partners (Deepinfra, OpenRouter, Together AI, GMI Cloud, Bitdeer) at their respective pricing.
Self-Hosted Containers	NVIDIA AI Enterprise license	Download NIM containers for deployment on your own GPU infrastructure. Requires NVIDIA AI Enterprise subscription for production use. Runs on H100, B200, A100, L40S, RTX, DGX Spark, and DGX Station.

The free tier is particularly generous, offering access to cutting-edge models like Nemotron-3-Ultra, DeepSeek-V4-Pro, GLM-5.2, and Kimi K2.6 at no cost. This makes NIM an excellent choice for developers who want to experiment with different models without committing to a paid plan.

NVIDIA NIM supports a wide range of AI use cases across industries, powered by its diverse model catalog and flexible deployment options:

• Retrieval Augmented Generation (RAG) - Combine embedding models with LLMs to build knowledge-grounded chatbots. Use NIM embedding endpoints for document retrieval and chat completion models for answer synthesis, all through a single API.
• Code Generation - Generate, review, and refactor code using models like DeepSeek Coder, Code Llama, and StarCoder. The OpenAI-compatible API works directly with tools like Cursor, Continue.dev, and custom IDEs.
• Agentic AI - Build autonomous agents that use tool calling and reasoning to solve complex tasks. Models like Nemotron-3-Ultra and GLM-5.2 excel at agentic workflows with native function calling support.
• Synthetic Data Generation - Generate high-quality synthetic training data for fine-tuning smaller models. NIM's image editing and visual design models support synthetic data pipelines for computer vision tasks.
• Chatbots and Virtual Assistants - Deploy conversational AI with human-like language understanding using Llama, Mistral, or Qwen models. Suitable for customer support, internal knowledge bases, and personal assistants.
• Drug Discovery - Leverage specialized BioNeMo and MolMIM models for molecular generation, protein folding, and drug target identification. NVIDIA's biology models are purpose-built for pharmaceutical research.
• Speech-to-Text and Translation - Transcribe audio and translate between languages using NIM's speech models. Useful for meeting transcription, content localization, and accessibility tools.
• Content Moderation - Deploy safety and moderation models to filter harmful content, detect bias, and enforce usage policies in user-facing applications.
• Climate and Weather Simulation - Run physics-based AI models like FourCastNet for weather forecasting and climate simulation directly through NIM endpoints.
• Image Understanding and Generation - Use vision models for image captioning, visual question answering, and image editing tasks through OpenAI-compatible vision API endpoints.

Feature	NVIDIA NIM	Together AI	Groq	OpenRouter
Model Count	141 (77 free)	200+	~30	300+ (routed)
Free Tier	77 free endpoints	$25 free credits	Rate-limited free tier	Some free models
Self-Hosted	Yes (108 downloadable containers)	No	No	No (routing only)
GPU Hardware	NVIDIA (H100, B200, A100, RTX, DGX)	NVIDIA H100	LPU (custom hardware)	Multi-provider
API Format	OpenAI-compatible	OpenAI-compatible	OpenAI-compatible	OpenAI-compatible
Enterprise Security	CVE patching, FedRAMP, OSRB	SOC 2	SOC 2	Varies by provider
Inference Speed	NVIDIA GPU-optimized	Fast (H100 clusters)	Fastest (LPU hardware)	Varies by provider
Specialized Models	Drug discovery, climate, physical AI	General LLMs	General LLMs	Wide variety
Best For	Enterprise, self-hosting, domain-specific AI	High-throughput API	Low-latency inference	Model routing and comparison

NVIDIA NIM distinguishes itself through its unique combination of a generous free tier, deep NVIDIA GPU optimization, and the ability to self-host containers on your own infrastructure. While Together AI offers a broader model catalog with competitive pricing and Groq delivers the lowest latency through custom LPU hardware, NIM is the only platform that lets you start with free API endpoints and seamlessly transition to self-hosted containers behind your own firewall. OpenRouter excels as a routing layer across providers but does not offer dedicated infrastructure or self-hosting capabilities.