NeurIPS 2025 · Datasets & Benchmarks

Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes Surprise3D: 面向复杂3D场景空间理解与推理的数据集

Jiaxin Huang1,2  Ziwen Li3  Hanlue Zhang3  Runnan Chen1  Zhengqing Gao4  Xiao He2  Yandong Guo2  Wenping Wang4  Tongliang Liu3†  Mingming Gong5†

1MBZUAI 2AI2Robotics 3University of Sydney 4Texas A&M University 5University of Melbourne
†Corresponding authors †通讯作者

arXiv Paper Code 🤗 Dataset MLLM-For3D BibTeX
Surprise3D task overview
Surprise3D covers both spatial reasoning (relative position, narrative/parametric perspective, absolute distance) and knowledge reasoning (common sense, human intention) — with queries that deliberately exclude object names. Surprise3D 涵盖空间推理（相对位置、叙事/参数视角、绝对距离）和知识推理（常识、人类意图）——查询中刻意排除了目标对象名称。

Dataset at a Glance 数据集概览

900+
Indoor Scenes
(ScanNet++ v2)
室内场景
(ScanNet++ v2)
200k+
Vision-Language
Pairs
视觉-语言
对
89k+
Human-Annotated
Spatial Queries
人工标注
空间查询
2,800+
Unique Object
Classes
唯一对象
类别

Why Surprise3D? 为什么需要Surprise3D？

Current 3D vision-language datasets let models take semantic shortcuts — they match object names in queries rather than truly reasoning about spatial relationships. We found that even strong multimodal models drop to near-zero accuracy on pure spatial reasoning tasks when object names are removed.

当前的3D视觉-语言数据集允许模型走语义捷径——它们匹配查询中的对象名称，而非真正推理空间关系。我们发现，当移除对象名称后，即使是强大的多模态模型在纯空间推理任务上的准确率也接近于零。

Surprise3D fixes this by crafting queries that describe spatial relationships without naming the target object, covering four distinct spatial reasoning skills with human-written queries, and providing the first standardized 3D Spatial Reasoning Segmentation (3D-SRS) benchmark.

Surprise3D 通过以下方式解决这一问题：构建不包含目标对象名称的空间关系描述查询，覆盖四种不同的空间推理技能的人工标注查询，并提供首个标准化的3D空间推理分割（3D-SRS）基准。

Spatial Reasoning Types 空间推理类型

Relative Position

相对位置

"Find the object behind the chair"
"找到椅子后面的物体"

Narrative Perspective

叙事视角

"Entering the door, with your back to the door, the object on your left side"
"进门后，背对门，你左手边的物体"

Parametric Perspective

参数视角

"At the perspective (3.00, -0.76, -1.24, 0.19, 0.025, 0.034, 0.00, -1.00), what's the closest object to sit?"
"在视角(3.00, -0.76, -1.24, ...)处，最近的可坐物体是什么？"

Absolute Distance

绝对距离

"A chair that is 2.63 meters away from the balcony door"
"距阳台门2.63米的椅子"

Data Analysis 数据分析

Data analysis and statistics
Left: question mention enhancement for low-frequency objects. Center: object frequency distribution. Right: distribution of question types across spatial and knowledge reasoning categories. 左：低频对象的查询增强。中：对象频率分布。右：空间推理和知识推理类别的查询类型分布。

Annotation Pipeline 标注流水线

Surprise3D annotations are built through two complementary pipelines: a human spatial annotation process for high-quality spatial queries, and an automatic generation pipeline for scalable knowledge-based queries.

Surprise3D 的标注通过两条互补的流水线构建：用于高质量空间查询的人工空间标注流程，以及用于可扩展知识类查询的自动生成流水线。

1. Spatial Annotation (Human) 1. 空间标注（人工）

Spatial annotation pipeline
Step 1: Volunteers select a 3D scene. Step 2: They write spatial descriptions under two situations — Situation 1: describe their orientation and an object's position relative to it (narrative/parametric perspective); Situation 2: describe the spatial relationship between objects (relative position, absolute distance). Step 3: They verify and choose the correct target objects in the 3D scene. 步骤1：志愿者选择一个3D场景。步骤2：在两种情境下编写空间描述——情境1：描述自己的朝向以及对象相对于该朝向的位置（叙事/参数视角）；情境2：描述对象之间的空间关系（相对位置、绝对距离）。步骤3：在3D场景中验证并选择正确的目标对象。

2. Automatic Pipeline (LLM + Human Verification) 2. 自动流水线（LLM + 人工验证）

Automatic annotation and cleaning pipeline
For knowledge-based queries (common sense, human intention): LLM generates initial Q&A pairs from scene metadata and object frequencies, then an automated quality-control stage with human-in-the-loop verification identifies and corrects problematic queries (e.g., where the answer refers to the spatial reference itself rather than the target). 对于知识类查询（常识、人类意图）：LLM从场景元数据和对象频率生成初始问答对，随后通过自动化质量控制阶段配合人工验证，识别并修正有问题的查询（例如答案指向空间参考本身而非目标对象的情况）。

3D-SRS Benchmark Results 3D-SRS基准结果

We evaluate state-of-the-art 3D visual grounding methods and 3D-LLMs under both zero-shot and fine-tuned settings. Results reveal significant challenges — even strong models struggle with pure spatial reasoning.

我们在零样本和微调设置下评估了最先进的3D视觉定位方法和3D-LLM。结果揭示了巨大挑战——即使是强模型也难以应对纯空间推理。

MethodSettingmIoUAcc@0.25Acc@0.50
3D-VistaZero-shot———
ChatSceneZero-shot———
MLLM-For3DZero-shot———
Intent3DFine-tuned———
Reason3DFine-tuned———
MLLM-For3DFine-tuned———

See the paper for complete results broken down by reasoning type. 按推理类型细分的完整结果请参阅论文。

Get the Data 获取数据

Surprise3D annotations are publicly available on HuggingFace. The underlying 3D scenes come from ScanNet++ v2 (requires separate access request).

Surprise3D标注数据已在HuggingFace上公开发布。底层3D场景数据来自ScanNet++ v2（需单独申请访问权限）。

🤗 Download from HuggingFace ScanNet++ v2 (Scene Data)
🔗

Companion Work: MLLM-For3D (NeurIPS 2025) 关联工作：MLLM-For3D（NeurIPS 2025）

Label-free 3D reasoning segmentation via 2D-to-3D MLLM knowledge transfer. Achieves +55% mIoU improvement without any 3D annotations. 通过2D到3D的MLLM知识迁移实现无标注3D推理分割。在无需任何3D标注的情况下，mIoU提升55%。

Paper ·  Code ·  Project Page

BibTeX

@inproceedings{huang2025surprise3d, title = {SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes}, author = {Huang, Jiaxin and Li, Ziwen and Zhang, Hanlue and Chen, Runnan and Gao, Zhengqing and He, Xiao and Guo, Yandong and Wang, Wenping and Liu, Tongliang and Gong, Mingming}, booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks}, year = {2025} }

Acknowledgements 致谢

We thank the authors of Reason3D, Intent3D, and the ScanNet++ team for their outstanding work and data. 感谢 Reason3D、Intent3D 的作者和 ScanNet++ 团队的杰出工作和数据。