UI-TARS-desktop

UI-TARS-desktop is an open-source multimodal AI agent stack developed by ByteDance. It bridges cutting-edge AI models with agent infrastructure, enabling desktop automation through visual understanding and natural language interaction. With over 37,900 GitHub stars and 3,800 forks, it is one of the most popular open-source agent frameworks.

The project is hosted on GitHub at github.com/bytedance/UI-TARS-desktop. UI-TARS (User Interface - Thinking, Acting, Reasoning, and Seeing) represents a family of models and tools designed to understand and interact with graphical user interfaces through vision-based reasoning.

• Multimodal Understanding - Processes screenshots and visual UI elements to understand desktop context.
• Agent Infrastructure - Provides a complete runtime for deploying and managing AI agents on desktop.
• MCP Protocol Support - Full integration with Model Context Protocol for tool extensibility.
• Cross-Platform - Works across macOS, Windows, and Linux desktop environments.
• Vision-Based Reasoning - Uses vision-language models to interpret UI layouts and decide on actions.
• Automation Pipelines - Supports complex multi-step automation workflows with conditional logic.

UI-TARS-desktop is built as a monorepo with a modular architecture. The apps/ui-tars directory contains the main desktop application, while supporting packages provide MCP server integration, vision model inference, and automation pipeline management. The architecture follows a microservices pattern where different components handle vision processing, action execution, and session management independently.

The system uses a perception-action loop: it captures the current screen state, processes it through a vision-language model to understand UI elements, plans actions based on user goals, executes them via system commands, and observes the results to inform the next step. This loop enables autonomous desktop navigation and task completion.

UI-TARS-desktop includes native support for the Model Context Protocol (MCP), allowing agents to extend their capabilities through external MCP servers. The framework provides an MCP HTTP server implementation that defaults to 127.0.0.1 for secure local communication. This enables integration with a wide ecosystem of tools including web search, database access, file system operations, and API services.

The MCP integration is configurable per agent session, allowing different automation workflows to have access to different tool sets based on security requirements and task needs.

From source:

git clone https://github.com/bytedance/UI-TARS-desktop.git
cd UI-TARS-desktop
pnpm install
pnpm build

Pre-built binaries for macOS, Windows, and Linux are available from the GitHub releases page.

• Desktop Automation - Automate repetitive desktop tasks like data entry, form filling, and file management.
• UI Testing - Visual-based testing of desktop applications without requiring accessibility hooks.
• Workflow Orchestration - Chain together multiple desktop applications to accomplish complex business processes.
• Assistive Technology - Help users with accessibility needs by providing voice-controlled desktop navigation.
• Research and Data Collection - Automate web research tasks that span multiple websites and desktop tools.