Whisper

Whisper is a general-purpose speech recognition model developed by OpenAI. Trained via large-scale weak supervision on 680,000 hours of multilingual and multitask data collected from the web, it achieves state-of-the-art performance on speech recognition, translation, and language identification across 99+ languages. With over 105,000 GitHub stars, it is one of the most successful open-source AI projects ever.

The project is hosted at github.com/openai/whisper. Whisper processes audio in 30-second segments, converting them into log-Mel spectrograms that are passed through an encoder-decoder transformer architecture. The model outputs text transcripts with timestamps in multiple formats.

Whisper offers five model sizes to balance accuracy, speed, and resource requirements:

Model	Parameters	VRAM	Speed	Use Case
tiny	39M	~1 GB	~10x	Real-time, edge devices
base	74M	~1 GB	~7x	General transcription
small	244M	~2 GB	~4x	Balanced quality
medium	769M	~5 GB	~2x	High quality transcription
large	1.55B	~10 GB	1x	Maximum accuracy

Speed is relative to the large model. Actual performance depends on hardware and audio length.

• Multilingual Transcription - Transcribe audio in 99+ languages with high accuracy.
• Speech-to-Text Translation - Translate non-English speech into English text in a single pass.
• Language Identification - Automatically detect the language being spoken in audio.
• Timestamped Output - Generate word-level or segment-level timestamps for alignment.
• Multiple Output Formats - Supports plain text, VTT, SRT, TSV, JSON, and more.
• Voice Activity Detection - Automatically detects speech segments and filters silence.

pip install openai-whisper

Whisper requires Python 3.8-3.11, FFmpeg, and PyTorch. GPU acceleration is recommended but not required. On macOS, PyTorch with MPS acceleration is supported for Apple Silicon Macs.

# Install ffmpeg (macOS)
brew install ffmpeg
# or Ubuntu/Debian
sudo apt update && sudo apt install ffmpeg

Command-line:

whisper audio.mp3 --model small --language English
whisper audio.mp3 --task translate  # Translate to English
whisper audio.mp3 --output_format srt  # Generate subtitles

Python API:

import whisper
model = whisper.load_model("small")
result = model.transcribe("audio.mp3")
print(result["text"])

Whisper achieves word error rates (WER) below 5% on clean English speech and below 10% on most accented variants. For multilingual transcription, it maintains competitive accuracy across a diverse set of languages, with particularly strong performance on high-resource languages like Mandarin, Spanish, French, German, and Japanese. The model's robustness to background noise, reverberation, and varying recording conditions makes it suitable for real-world applications including meeting transcription, content captioning, and voice interfaces.