Fine-grained grounding
Select the intended target among visually similar candidates using geometry, distance, viewpoint, relations, and canonical indices.
EMNLP 2026
RoboSPA: Can VLA models go beyondsimple scenes and short-horizon tasks?
RoboSPA is a large-scale robotic manipulation benchmark for diagnosing fine-grained spatial reasoning and long-horizon procedural planning.
Two capability dimensions10 categories
527Ktrajectories
997hinteraction video
108Mtimesteps
56base tasks
280task variants
5embodiments
01 / Abstract
Abstract
Vision-Language-Action (VLA) models have shown promising progress in language-conditioned robotic manipulation. However, existing datasets and benchmarks mainly evaluate task completion under predefined settings, offering limited insight into model reasoning under increasing spatial and procedural complexity. We introduce RoboSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation dataset and benchmark for diagnosing embodied reasoning in VLA models. RoboSPA focuses on two core dimensions, Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning, covering 10 task categories and 56 base tasks. Each task is instantiated across five difficulty levels, yielding 280 variants with increasing spatial ambiguity and procedural complexity. We collect 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPA introduces diagnostic metrics for more detailed evaluation. Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. These results establish RoboSPA as a challenging diagnostic benchmark for developing more capable, reliable, and generalizable embodied agents. Our data and code are available at github.com/fanzhenxuan/RoboSPA.
02 / Overview
A controlled stress test for embodied reasoning.
Existing evaluations often stop at final success in clean, short tasks. RoboSPA scales ambiguity, action horizon, and scene diversity while preserving task semantics—making model limitations measurable.
Extended execution
Complete multi-step procedures with repetition, ordering constraints, heterogeneous actions, and task-relevant memory.
Scalable complexity
Measure how behavior degrades as candidate counts, action horizons, distractors, and environment diversity increase.
03 / Capability taxonomy
Two dimensions.
Ten precise capabilities.
Each category isolates a distinct reasoning bottleneck instead of treating manipulation tasks as undifferentiated successes or failures.
01GAC
Geometric Attribute Cognition
Identify objects by subtle geometric or shape-related attributes beyond category recognition.
02SDE
Spatial Distance Estimation
Compare distances to a reference object and reason about relative proximity.
03CPI
Canonical Position Indexing
Ground row-column indices under different counting and scanning directions.
04RRR
Referential Relational Reasoning
Locate targets through directional relations to reference objects.
05CVR
Cross-View Reasoning
Transform spatial references across non-egocentric viewpoints.
04 / Hierarchical difficulty
Complexity grows in controlled steps.
Spatial tasks add plausible candidates. Procedural tasks extend the sequence. Five aligned levels reveal when capability begins to break.
L1
Foundation
L2
Increasing load
L3
Increasing load
L4
Increasing load
L5
Stress test
05 / Scale & diversity
Built across robots, scenes, and task difficulty.
RoboSPA combines clean and domain-randomized scenes across Aloha-AgileX, ARX-X5, Piper, Franka, and UR5.
- 463Krandomized-scene trajectories
- 5×robotic embodiments
- L1→L5more objects or longer horizons
06 / Results
Stronger models still fail as complexity rises.
<25%overall success for every evaluated model at L5
Overall benchmark success rateL1 versus L5
L1 L5
RDT−9.9 pts
GO-1−16.3 pts
π0.5−32.9 pts
X-VLA−30.5 pts
0204060%
Target selection stays close to chance.
Low ONTA exposes unreliable grounding when identification depends on subtle relations instead of object category.
Partial progress rarely becomes success.
Progress Score exceeds final Success Rate, showing that models start procedures but fail to complete later stages reliably.
The hardest memory tasks collapse.
All baselines fail on the most difficult memory-intensive planning settings.
07 / Diagnostic analysis
Not just whether models fail—how they fail.
Step-level evaluation separates grounding, execution, ordering, repetition, and memory failures.
01Target execution02Target grounding03Manipulation04Memory05Temporal ordering06Redundant repetition
08 / Resources
Use RoboSPA in your research.
Access the paper, implementation, and dataset resources for RoboSPA.
BibTeX
@inproceedings{robospa2026,
title = {RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?},
author = {Zhenxuan Fan and Bo Zhang and Yutong Lin and Yuqian Yuan and Juekai Lin and Liang Liang and Zhuoyi Huang and Wenqiao Zhang and Juncheng Li and Siliang Tang and Jun Xiao and Yueting Zhuang},
booktitle = {Proceedings of EMNLP},
year = {2026}
}