Alibaba-backed Qwen has open-sourced Qwen-Drive-1.0-4B, a vision-language foundation model built specifically for autonomous driving. It is the first model of its kind to unify three normally separate capabilities within a single pretrained vision-language model (VLM): explicit 3D perception, driving scene reasoning through visual question answering (VQA), and end-to-end motion planning. The 4-billion-parameter weights are live on Hugging Face, ModelScope, and GitHub, alongside a full technical report deposited on arXiv.
One Backbone, Two New Organs: How the Architecture Works
Rather than retraining a model from scratch, the authors bolt two external modules onto a stock natively multimodal Qwen3.5-4B and leave the VLM weights untouched. The first is a bird's-eye-view (BEV) perception head. It reads two complementary feature streams per camera: the vision encoder features that capture low-level appearance before tokens enter the VLM, and the VLM's output features that encode broader scene context. A depth-based view transform lifts single-scale features into a 3D volume, and a query-based BEV transformer aggregates them onto the ego-frame plane.
The head jointly predicts three tasks, which is a natural fit for a bulleted breakdown:
- 3D object detection with a DETR-style deformable-attention decoder.
- Semantic occupancy prediction through a shallow 3D UNet.
- BEV map segmentation of driveable surface, lanes, edges, and crosswalks.
Because the VLM architecture is frozen, the head acts as an inspectable 3D probe: you can literally read out the detections, occupancy, and maps rather than trusting implicit spatial reasoning. It also generalizes across camera rigs without rig-specific embeddings, so one trained model handles both the six-camera nuScenes configuration and the eight-camera OpenScene configuration.
The second module is a Planning Expert, a 32-layer diffusion transformer of roughly 1.1 billion parameters. It conditions on cached keys and values from the VLM's grouped-query softmax attention layers and generates future ego trajectories via flow matching with an output-parameterized objective. Each trajectory spans 50 waypoints over five seconds at 10 Hz, decoded with a 10-step Euler solver. An optional textual planning reason can condition the rollout, letting the model explain why it is choosing a path.
A Staged Recipe Built to Fight Catastrophic Forgetting
The authors argue that pure VQA adaptation has two failure modes: it never constrains explicit 3D layout, and heavy domain fine-tuning destroys pretrained general knowledge. Their four-stage recipe attacks both problems at once. It is worth enumerating because it is the core of the method:
- Stage 1: train only the BEV head while the VLM stays frozen.
- Stage 2: train the head, vision encoder, and VLM jointly on perception plus vision-language data.
- Stage 3: train the Planning Expert by supervised imitation on unified trajectory data.
- Stage 4: optimize the Planning Expert with reinforcement learning against task-level rewards.
The reinforcement stage is notable for using a group-relative advantage with eight sampled reasoning traces per scene, and for confining stochastic exploration to the final three Euler steps along a low-frequency trajectory subspace. This keeps rollout diversity meaningful without injecting high-frequency jitter into the plans.
The data recipe is equally laborious. Twenty-four public driving VQA datasets are rewritten into a common conversational schema using Qwen3.5-Plus, then a consistency filter using Qwen3.5-Flash prunes the pool from 5.53 million to 3.09 million samples, a 55.9 percent retention. Perception training draws on nuScenes and OpenScene, while planning uses roughly 2.83 million samples aggregated from NAVSIM, OpenScene, WOD-E2E, and PAI-AV. Self-constructed chain-of-causation reasoning traces, view-shuffling tasks, and 30,000 in-house Chinese perception QA examples fill the gaps the public data do not cover.
The Numbers That Matter
On 3D perception, the model reaches 43.95 mAP and 60.99 map mIoU on nuScenes, and 43.45 mAP and 71.27 map mIoU on OpenScene. That bests the unified BEVFormerV2 benchmark by 2.01 mAP on detection and PETRv2 by 3.37 on map mIoU, all from a single 4B weight set.
On driving scene understanding it clearly separates from its own general-purpose base. The driving VQA average climbs to a best-in-class 69.43, while causal reasoning jumps to 58.30 against a 22.05 ceiling for the nearest competitor. SURDS improves 13.18 points and Ego3D distance estimation error drops 40.9 percent.
Most striking is that this driving competence does not come at the cost of generality. The model preserves a 66.41 average across ten general knowledge and recognition benchmarks, within one point of the untouched Qwen3.5-4B, and its spatial-grounding average of 53.96 actually exceeds the base model's 52.99. An engineer could plausibly run one instance for both cockpit dialogue and driving perception.
On planning, the reinforced variant reaches a Predictive Driver Model Score of 90.7 on NAVSIM and a 7.91 Rater Feedback Score on the WOD-E2E test split, beating the 8.13 human-driver reference on validation. Closed-loop AlpaSim simulation halves the off-road rate and matches a larger 10B rival on at-fault close encounters, trends that indicate a shift toward safer, more conservative behavior at a modest cost to progress.
Qwen-Drive-1.0 is explicitly an initial step rather than a finished autonomous stack. The authors flag that planning reasoning can still misidentify the governing cause when distant and immediate hazards coexist, and that the generated trajectory does not always follow its stated rationale. But as an open foundation model that sees in 3D, reasons about traffic, and plans a path all from one 4B backbone, it sets a new benchmark for what an accessible driving model can do.
Comments