AirLLM

AirLLM is an open-source inference optimization library created by Lyogavin that dramatically reduces the GPU memory requirements for running large language models. Its headline capability is running a 70 billion parameter model (like Llama 2 70B) on a single consumer GPU with just 4GB of VRAM - something that would typically require 140GB+ of GPU memory.

AirLLM achieves this without quantization, pruning, or distillation, meaning the model retains its full precision and accuracy. It is one of the most popular projects in the local LLM ecosystem with over 22,500 GitHub stars. The project is hosted at github.com/lyogavin/airllm.

• Single GPU Inference - Run 70B models on a single 4GB GPU without model sharding across multiple GPUs.
• No Quantization Required - Full FP16/BF16 precision inference with no accuracy loss from quantization.
• Layer-by-Layer Streaming - Processes one transformer layer at a time, swapping layers between CPU RAM and GPU VRAM as needed.
• Automatic Memory Management - Calculates optimal memory allocation between CPU, GPU, and disk for any model size.
• Multi-Model Support - Works with Llama, Falcon, Mistral, Qwen, DeepSeek, and other popular architectures.
• Inference Optimization - Includes KV cache offloading, attention optimizations, and batch processing support.
• Easy API - Simple Python API compatible with HuggingFace Transformers patterns.

AirLLM employs a technique called layer-by-layer streaming inference. Instead of loading all model layers into GPU memory simultaneously (the conventional approach), AirLLM loads and processes one transformer layer at a time. After a layer computes its output, the result is passed to the next layer while the previous layer is unloaded from GPU memory.

This approach uses CPU RAM as a staging area for model weights, with only the actively computing layer needing to reside in GPU VRAM. For very large models, AirLLM can also use disk as a backing store. The KV cache is similarly managed through intelligent offloading, keeping only the cache for the current layer's computation on the GPU.

While this approach increases inference latency (each token requires more data movement between CPU and GPU), it makes it possible to run models that would otherwise be completely impossible on consumer hardware. AirLLM's benchmarks show that even on a 4GB GPU, a 70B model can generate tokens at a usable rate for many applications.

Install via pip:

pip install airllm

Basic usage example:

from airllm import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf"
)
input_text = "The capital of France is"
output = model.generate(input_text)
print(output)

AirLLM supports models in HuggingFace format, as well as custom model paths for locally downloaded checkpoints. The library automatically handles memory configuration based on your hardware.

AirLLM's performance varies significantly based on hardware configuration. The trade-off for running large models on limited hardware is inference speed. Typical measured performance:

Model Size	Minimum GPU	Typical Speed
7B	2 GB VRAM	5-10 tokens/sec
13B	3 GB VRAM	2-5 tokens/sec
34B	4 GB VRAM	1-2 tokens/sec
70B	4 GB VRAM	0.5-1 tokens/sec

Performance can be improved by having more GPU memory (reduces swapping frequency), faster CPU RAM and PCIe bandwidth, and using AirLLM's batching support for throughput-oriented workloads. The library is continuously optimized with each release.

AirLLM solves a specific problem: running large models on very limited GPU hardware. Here is how it compares to other approaches:

Criterion	AirLLM	llama.cpp (GGUF)	Full Precision
GPU Requirement	4 GB (70B)	8 GB (70B Q4)	140 GB (70B)
Accuracy	Full precision	Quantized (lossy)	Full precision
Speed	Slow (CPU/GPU swap)	Fast (CPU only)	Very fast
Model Formats	HF Transformers	GGUF	Any
Best For	Limited GPU, need accuracy	CPU inference, efficiency	Production servers

AirLLM vs llama.cpp: AirLLM is the better choice when you need full model accuracy and have a GPU available (even a modest one), while llama.cpp excels for pure CPU inference with quantization. AirLLM preserves the original model weights and delivers higher quality outputs, while llama.cpp can run on machines with no GPU at all. For developers who need both accuracy and have limited VRAM, AirLLM is the clear winner.

While AirLLM is a groundbreaking tool for democratizing LLM access, it has important limitations to consider:

• Inference Speed - Layer-by-layer streaming introduces significant latency compared to full GPU inference. Real-time applications like chatbots may feel sluggish.
• CPU RAM Requirement - The technique requires sufficient CPU RAM to hold all model weights. A 70B model needs roughly 140GB of system RAM.
• Limited Batch Processing - While supported, batch processing throughput is constrained by the same memory bottleneck.
• GPU Utility - The GPU is underutilized during most of the inference cycle since only one layer is active at a time.
• Model Support - Not all model architectures are supported; newer models may require library updates.