NVIDIAVFlash

NVIDIA Research · System-1 Physical AI

VFlash

Fast, accurate
visual planning.

Predict complete visual primitives in parallel.
Turn spatial predictions into useful plans for action.

Yu-Cheng Chou1,2*†Shihao Wang1,3*†Jin Huang1*Chi-Pin Huang1Chiao-An Yang1,4†Xianzhe Fan1,5†Jing Wang1Zhiqi Li1Yuanguo Kuang1,3†Ruijie Zheng1Johan Bjorck1Linxi “Jim” Fan1Sifei Liu1Hang Su1Abhishek Badki1Lei Zhang3Alan Yuille2Jan Kautz1Zhiding Yu1‡

1NVIDIA · 2Johns Hopkins University · 3Hong Kong Polytechnic University · 4Purdue University · 5The University of Hong Kong

* Equal contribution. † Work done during an internship at NVIDIA. ‡ Corresponding author.

ROBOT & DRIVING MODEL INFERENCE5×

faster on Jetson Thor
and DGX Spark

See the details ↓
42.6%

lower 3D trajectory ADE
on average vs. AR

70.2%

MolmoSpaces success
vs. 62.6% baseline

01 · Flash Primitive Decoding

System-1 visual planning
for Physical AI.

Embodied models do not necessarily think in text; they can also think in visual plans. Predicting where to grasp, how to move, and how a scene will evolve bridges perception and action for robots and autonomous vehicles. However, agents must continually update these plans, and autoregressive generation of coordinate tokens makes richer plans increasingly costly.

We introduce VFlash, a generalist decoding framework for Physical AI adapting VLM speculative decoding around a key insight: the known structure of visual primitives can be exploited to achieve spatial accuracy without reproducing the autoregressive target’s exact tokens. A lightweight drafter trained on annotated geometry can predict complete visual primitives in parallel and outperform its autoregressive target on robot trajectories.

Choose a primitive
Layout
Observed robot scene
INSTRUCTION

Plan the gripper’s next motion.

Target VLMfrozen
Visual + language context
REQUESTED LAYOUT10 waypoints · 2D + 3D
54 positions
VFlashOne parallel drafter pass
A complete primitive, accepted together

Encode the observation and instruction once to build the target context.

Finding 1

Boundary alignment matters.

Training and decoding blocks should match complete primitive boundaries, allowing the drafter to learn and predict the full geometry together rather than fragments split by fixed token windows. VFlash introduces a structured tool call to declare the length of a variable-size primitive; its type determines the coordinate layout. Flash Primitive Decoding allocates a block for the complete geometry and fills it in one parallel draft using the frozen VLM’s context.

Finding 2

Position alignment matters.

Each output position needs a known role before parallel prediction begins. Different coordinate values can occupy different numbers of text tokens and shift later positions. We use dedicated coordinate tokens or fixed-width numeric fields to keep waypoint, axis, and delimiter positions consistent. This supports parallel prediction with a consistent layout and per-position constraints at inference.

Training

Training on annotated geometry.

The target VLM remains frozen while we train a lightweight drafter on annotated geometry. The drafter has five non-causal transformer layers and is trained with token cross-entropy on annotated primitives. It predicts complete visual primitives in parallel without requiring token-level agreement, while the same drafter retains verified decoding for sub-task text.

Inside the decoder

DFlash uses a lightweight block-diffusion drafter conditioned on the target model’s hidden features as KV cache context, and produces drafts in parallel. VFlash extends the idea of box-aligned decoding from LocateAnything to richer types of Physical AI outputs. A generalist decoder needs variable-length support on demand: NAVSIM requires 8 future waypoints over 4 seconds, Waymo requires 20 over 5 seconds, and Alpamayo predicts 64 over 6.4 seconds. A generic token window does not capture where one such primitive begins or ends.

For text, the drafter uses 16-position blocks: one accepted anchor and 15 proposals. The target verifies the block in one pass and keeps the longest matching prefix plus its own next token. The tool call is generated on this path. When an accepted prefix reaches a primitive-start token, the declared schema and block length take over. After primitive prediction, one target pass updates its cache and produces the features that downstream action models consume.

View the pipeline figure ↗

02 · Beyond the AR target

A drafter that surpasses
its autoregressive target.

Trajectory errors decrease across all evaluated 2D/3D robot settings. VFlash reduces all six trajectory error metrics across ShareRobot-Bench and VABench-V relative to autoregressive (AR) decoding. Across three robot datasets and three target variants, VFlash reduces 3D average displacement error (ADE) by an average of 42.6% relative to its autoregressive target.

Trajectory evaluation
Training variant
Metric
AR targetVFlash

03 · Generalist decoding

Principled parallelism
across visual tasks.

Flash Primitive Decoding aligns block boundaries with complete primitives and token positions with coordinate roles. These boundary and position alignments support boxes, variable-length 2D/3D trajectories, and dense 3D motion, while sub-task text retains verified speculative decoding.

VFlash predicts complete 2D/3D trajectories in parallel. Training and decoding blocks match complete primitive boundaries, allowing the drafter to learn and predict the full geometry together. Trajectory errors decrease across all evaluated 2D/3D robot settings, with lower average 3D waypoint error.

04 · Robot manipulation

Visual plans improve
downstream control.

The downstream world action models and diffusion action heads differ in architecture, but share a common design: they cross-attend to VLM features encoding the generated visual plans. We obtain these features by passing VFlash’s predicted primitives through the target VLM. The action models are trained for robot manipulation or driving, while the VLM remains frozen.

Images + instruction→VLM + VFlash→Visual-plan features→Action policy

MolmoSpaces · world action model

The plan’s geometry matters.

Average success rate ↑
62.6%

Baseline

61.0%

AR plans

63.7%

VFlash · w/o visual plan

69.3%

VFlash · 3D-only

70.2%

VFlash

The robot policy is a world action model (WAM) obtained by post-training Cosmos-Predict2.5 for control. It keeps its camera and robot-state inputs and receives the frozen VLM’s question, sub-task, and trajectory features through its existing cross-attention pathway. The WAM predicts 20-step action chunks and replans every three chunks. VFlash reaches 70.2% average success versus 62.6% for the baseline.

With the same WAM architecture, autoregressive 2D+3D plans yield 61.0% success versus VFlash’s 70.2%. Replacing trajectory features with image features, while retaining the generated sub-task, lowers success to 63.7%—evidence that visual plans contribute beyond language alone.

Explore MolmoSpaces latency ↓
Four-suite results and comparison setup
MolmoSpaces ablations · context and success rates (%) ↑
ConfigurationVLM contextDecoderPrimitiveAvg. (%) ↑Success rate by suite (%) ↑
ImageQuestionSub-taskTrajectoryPickPick & PlaceOpenClose
Original baselineOriginal text encoder——62.681.155.749.364.2
AR plans—✓✓✓AR2D + 3D61.081.250.429.882.4
VFlash · w/o visual plan✓✓✓—VFlash—63.780.252.544.377.8
3D-only plans—✓✓✓VFlash3D69.381.773.048.374.2
VFlash—✓✓✓VFlash2D + 3D70.283.667.352.877.0

Checkmarks identify which VLM states condition the WAM. All variants use the same WAM architecture and training budget. Success is averaged equally across the four suites.

LIBERO family · diffusion action head

Gains remain under perturbations.

A PIv3 action head cross-attends to a frozen VLM. The baseline encodes images and instructions with Qwen3-VL-4B. Our VLM supplies states through the generated sub-task or full plan. From primary and wrist images, it predicts a sub-task and ten-waypoint 2D+3D trajectory before each eight-action chunk.

05 · Autonomous driving

Visual planning
for autonomous driving.

A ReCogDrive-derived action head uses VFlash plans on NAVSIM and Waymo. The head cross-attends to the VLM’s final-layer states over the prompt and generated plan, alongside ego history and vehicle state, and outputs a continuous trajectory. The planning VLM remains frozen, with separate action heads trained for NAVSIM and Waymo.

On NAVSIM, training the head with reinforcement learning raises PDMS from 87.1 to 90.7 while keeping the planning VLM and its plans unchanged. On Waymo, RFS increases from 7.79 to 7.86. These results extend the utility of visual planning from robot manipulation to driving.

Explore driving latency ↓
FROZENVisual planner
→
SFT → RLAction head
Supervised head87.1
→
RL-trained head90.7

PDMS ↑ · NAVSIM test

Driving results and evaluation
ConfigurationRLNAVSIMWaymo test
NC ↑DAC ↑EP ↑TTC ↑Comfort ↑PDMS ↑ADE₃ₛ (m) ↓ADE₅ₛ (m) ↓RFS ↑
VFlash-SFT—98.195.281.794.3100.087.11.232.717.79
VFlash-RL✓98.397.885.995.4100.090.71.222.707.86

NC: no at-fault collision; DAC: drivable-area compliance; EP: ego progress; TTC: time-to-collision within bound.

06 · NVIDIA deployment

Removing
the bottleneck.

Autoregressive decoding accounts for 75–87% of measured model-stage latency on Thor. VFlash reduces this stage by 12–42×, yielding 5.1–5.2× faster robot inference on Thor and Spark. NAVSIM driving inference is 3.4–3.6× faster than autoregressive decoding on the same platforms. On Thor, MolmoSpaces model latency falls from about 3.55 seconds to 0.70 seconds.

Platform
Task
Batch size 1
Preprocess + visionPrefillDecodingAction model
MODEL-PIPELINE SPEEDUP5.1×

vs. autoregressive decoding

2.6× vs. DFlash

Verification for text and trajectories.

On MolmoSpaces, DFlash commits 5.5 tokens per iteration on sub-task text but only 2.2 within trajectory blocks. A coordinate one bin off the target’s choice fails token-level verification even with small geometric error. VFlash uses one drafter pass and one target pass per primitive; the target pass updates the cache and extracts hidden states for the action head, amortized over the block.

NVIDIA’s inference ecosystem.

VFlash uses a customized TensorRT-Edge-LLM C++ runtime, with TensorRT engines for the vision encoder, target VLM, and drafter. CuTe DSL attention, XQA decoding plugins, and CUDA-optimized action models support deployment across Jetson Thor, DGX Spark, and H100. On Thor, decoding takes 71.9 ms for NAVSIM’s 36-position primitive and 78.3 ms for Waymo’s 84-position primitive, including target feature preparation. All measurements use batch size one.