Component identification
Select the required component instances from the candidate library, including repeated reuse of identical library geometry.
OmniCAD
A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies
25Kassemblies
289Kmate relations
Qualitative results
Qualitative assembly examples
Interactive OmniCAD assembly cases. Drag to rotate, scroll to zoom, and pan within each 3D result.
The benchmark
Assembly intelligence needs more than shape recognition.
OmniCAD evaluates whether a multimodal model can recover a physically plausible industrial assembly from a deduplicated component library and visual/geometric evidence. Repeated component instances share library geometry while retaining independent IDs, poses, and mate incidences.
Each sample preserves component-to-library mappings, 6-DoF poses, component-level mate relations, directed assembly dependencies, assembly/component meshes, and eight fixed cube-vertex reference views under same-color and color-by-component settings.
Component LibraryDeduplicated OBJ parts
Mate GraphComponent connectivity + mate type
Ground TruthVerified 3D assembly + poses
Visual Evidence8 cube-vertex renders
Three evaluation tracks
From part selection to constraint-aware refinement.
The benchmark separates component identity, spatial placement, relational structure, and iterative tool use so that failure modes remain measurable rather than collapsing into a single end metric.
3D pose reasoning
Estimate absolute position and orientation for every component and recover accurate relative placement in the assembly frame.
Mate-graph reasoning
Recover component-level connectivity and mate types such as coincident, concentric, parallel, perpendicular, distance, and angle constraints.
Agentic assembly reasoning
Request visualization and geometric-conflict feedback, then revise poses, mates, and physical plausibility over multiple rounds.
Dataset statistics
Real assemblies expose repeated parts, dense constraints, and long-tail complexity.
The dataset spans robotics, manufacturing, electronics, packaging, welding, inspection, automation, aerospace, machine tools, and other industrial domains. Assemblies average 12 component instances and 11.56 mate relations, with long tails extending to hundreds of parts.
Quantitative results
Direct One-shot Results
SameColor, RefOBJ, ColorView, MG+Color, and MG+OBJ+Color isolate the effects of appearance cues, explicit assembly geometry, and connectivity priors. Higher is better except CD, SCD, PosErr, and runtime.
| Model | Size | Input | ID-NR ↑ | ID-R ↑ | Pos@10 ↑ | Rot@10 ↑ | CDᵢ ↓ | PA ↑ | PairF1 ↑ | TypeAcc ↑ | GSim ↑ | C-Free ↑ | SCD ↓ | PosErr ↓ | RT ↓ | PR ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.5 | - | SameColor | 100.00 | 60.00 | 15.30 | 48.40 | 245.6 | 32.00 | 36.40 | 44.10 | 12.10 | 3.10 | 144.0 | 368.9 | 111.0 | 37.00 |
| - | RefOBJ | 100.00 | 71.40 | 0.00 | 84.70 | 240.8 | 14.70 | 46.10 | 59.70 | 23.30 | 11.80 | 152.4 | 316.3 | 99.5 | 21.00 | |
| - | ColorView | 92.90 | 57.10 | 15.70 | 43.30 | 349.3 | 25.00 | 35.70 | 44.00 | 8.40 | 10.00 | 232.2 | 481.9 | 171.3 | 35.00 | |
| - | MG+Color | -- | -- | 27.90 | 63.20 | 279.7 | 46.70 | -- | -- | -- | 0.00 | 186.5 | 328.6 | 83.1 | 20.00 | |
| - | MG+OBJ+Color | -- | -- | 0.00 | 94.10 | 246.9 | 11.10 | -- | -- | -- | 12.50 | 159.6 | 319.6 | 87.4 | 18.00 | |
| GPT-5-mini | - | SameColor | 93.80 | 0.00 | 12.80 | 37.70 | 268.5 | 18.20 | 21.30 | 5.30 | 6.80 | 12.10 | 176.8 | 306.6 | 12.8 | 35.00 |
| - | RefOBJ | 90.00 | 0.00 | 11.20 | 41.20 | 337.7 | 23.80 | 21.10 | 3.40 | 8.20 | 26.30 | 186.2 | 378.3 | 8.6 | 21.00 | |
| - | ColorView | 93.80 | 15.00 | 11.40 | 36.70 | 305.7 | 16.30 | 22.10 | 6.30 | 5.50 | 11.80 | 204.1 | 351.1 | 12.1 | 36.00 | |
| - | MG+Color | -- | -- | 15.30 | 37.50 | 325.8 | 18.50 | -- | -- | -- | 12.10 | 186.0 | 359.9 | 12.0 | 35.00 | |
| - | MG+OBJ+Color | -- | -- | 3.90 | 42.90 | 392.1 | 12.90 | -- | -- | -- | 31.20 | 221.7 | 458.8 | 15.0 | 18.00 | |
| Claude Opus 4.8 | - | SameColor | 95.00 | 16.10 | 18.60 | 39.30 | 399.4 | 21.90 | 34.80 | 30.00 | 6.30 | 15.80 | 262.7 | 431.4 | 11.8 | 70.31 |
| - | RefOBJ | 95.30 | 16.10 | 6.20 | 36.10 | 393.1 | 10.70 | 37.80 | 30.00 | 8.10 | 22.80 | 263.6 | 458.7 | 12.6 | 50.60 | |
| - | ColorView | 95.10 | 24.10 | 19.50 | 39.90 | 389.5 | 22.70 | 35.80 | 31.80 | 5.60 | 14.40 | 254.4 | 426.4 | 9.2 | 72.34 | |
| - | MG+Color | -- | -- | 23.30 | 44.00 | 375.2 | 26.50 | -- | -- | -- | 11.60 | 240.8 | 407.3 | 14.2 | 70.11 | |
| - | MG+OBJ+Color | -- | -- | 9.60 | 39.50 | 372.2 | 13.70 | -- | -- | -- | 16.60 | 239.7 | 418.5 | 16.1 | 53.40 | |
| Gemini 3.1 Pro | - | SameColor | 100.00 | 11.10 | 5.30 | 34.70 | 171.6 | 12.50 | 25.00 | 21.50 | 3.60 | 0.00 | 74.6 | 203.9 | 84.9 | 4.40 |
| - | RefOBJ | 75.00 | 0.00 | 0.00 | 28.00 | 272.5 | 11.10 | 2.60 | 2.20 | 11.10 | 37.50 | 158.2 | 287.6 | 168.7 | 5.62 | |
| - | ColorView | 96.10 | 6.50 | 8.50 | 36.50 | 365.0 | 20.00 | 17.90 | 13.00 | 12.60 | 6.00 | 228.8 | 426.5 | 49.7 | 32.29 | |
| - | MG+Color | -- | -- | 16.70 | 67.00 | 162.7 | 55.00 | -- | -- | -- | 0.00 | 128.8 | 224.3 | 59.0 | 5.00 | |
| - | MG+OBJ+Color | -- | -- | 0.00 | 50.00 | 88.2 | 0.00 | -- | -- | -- | 0.00 | 84.2 | 111.9 | 42.6 | 1.50 | |
| Gemini 2.5 Flash-lite | - | SameColor | 95.70 | 8.30 | 8.40 | 35.00 | 253.8 | 16.40 | 22.60 | 13.00 | 8.30 | 2.60 | 160.0 | 277.6 | 1.6 | 39.00 |
| - | RefOBJ | 96.00 | 20.00 | 9.50 | 34.70 | 272.2 | 16.70 | 13.00 | 5.00 | 14.50 | 2.10 | 177.9 | 298.1 | 30.7 | 18.04 | |
| - | ColorView | 95.70 | 25.20 | 8.10 | 36.70 | 220.2 | 15.50 | 23.70 | 12.50 | 11.00 | 4.30 | 129.0 | 250.0 | 25.4 | 35.05 | |
| - | MG+Color | -- | -- | 8.40 | 38.20 | 271.2 | 17.00 | -- | -- | -- | 1.90 | 163.4 | 299.0 | 21.2 | 39.18 | |
| - | MG+OBJ+Color | -- | -- | 9.20 | 41.20 | 260.0 | 18.60 | -- | -- | -- | 5.60 | 163.5 | 291.8 | 22.8 | 20.27 | |
| LLaMA4 Maverick | 400B | SameColor | 95.40 | 16.90 | 21.80 | 47.10 | 602.0 | 27.80 | 41.30 | 20.90 | 13.90 | 18.10 | 405.9 | 647.9 | 18.2 | 22.54 |
| 400B | RefOBJ | 97.50 | 5.20 | 19.40 | 45.70 | 576.5 | 26.50 | 38.90 | 19.50 | 13.70 | 23.00 | 421.9 | 545.1 | 203.3 | 36.31 | |
| 400B | ColorView | 94.60 | 24.40 | 22.20 | 45.90 | 688.7 | 27.60 | 40.70 | 21.00 | 13.50 | 16.20 | 519.9 | 653.8 | 223.9 | 19.91 | |
| 400B | MG+Color | -- | -- | 22.60 | 50.50 | 664.3 | 27.80 | -- | -- | -- | 21.30 | 472.6 | 660.6 | 234.2 | 19.83 | |
| 400B | MG+OBJ+Color | -- | -- | 24.50 | 52.50 | 659.5 | 31.20 | -- | -- | -- | 25.80 | 447.8 | 596.7 | 190.9 | 10.43 | |
| Ministral | 14B | SameColor | 98.00 | 48.00 | 17.60 | 45.00 | 697.2 | 25.00 | 45.30 | 26.70 | 16.90 | 29.10 | 488.0 | 655.3 | 88.7 | 13.83 |
| 14B | RefOBJ | 97.40 | 90.00 | 12.30 | 52.50 | 164.2 | 20.70 | 37.30 | 14.90 | 15.30 | 4.10 | 140.7 | 195.7 | 571.7 | 1.40 | |
| 14B | ColorView | 93.00 | 53.20 | 16.10 | 42.80 | 758.7 | 23.40 | 43.10 | 28.50 | 17.30 | 33.50 | 588.2 | 753.3 | 559.1 | 5.71 | |
| 14B | MG+Color | -- | -- | 16.10 | 42.50 | 1860.2 | 20.10 | -- | -- | -- | 32.30 | 1689.3 | 1779.5 | 358.5 | 2.69 | |
| 14B | MG+OBJ+Color | -- | -- | 0.30 | 57.00 | 171.4 | 3.70 | -- | -- | -- | 27.30 | 110.5 | 223.9 | 388.1 | 0.31 | |
| Qwen3.5-9B | 9B | SameColor | 100.00 | 33.30 | 7.10 | 25.80 | 180.0 | 7.10 | 35.80 | 18.50 | 20.00 | 28.60 | 101.8 | 210.1 | 263.0 | 1.40 |
| 9B | RefOBJ | 100.00 | 0.00 | 2.60 | 39.60 | 390.4 | 2.60 | 30.20 | 20.40 | 9.10 | 9.10 | 223.9 | 493.1 | 302.9 | 2.20 | |
| 9B | ColorView | 100.00 | 28.60 | 20.20 | 48.00 | 241.7 | 28.10 | 29.00 | 19.40 | 9.30 | 22.20 | 165.9 | 250.7 | 153.8 | 3.60 | |
| 9B | MG+Color | -- | -- | 6.10 | 25.10 | 559.6 | 3.30 | -- | -- | -- | 33.30 | 277.7 | 654.8 | 166.7 | 1.20 | |
| 9B | MG+OBJ+Color | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |
| Qwen3.7-Plus | - | SameColor | 94.30 | 1.30 | 14.70 | 34.70 | 380.6 | 17.00 | 20.60 | 13.00 | 8.70 | 4.30 | 238.0 | 416.6 | 288.6 | 51.00 |
| - | RefOBJ | 98.30 | 3.40 | 7.50 | 37.10 | 290.9 | 9.50 | 26.00 | 19.30 | 11.40 | 12.90 | 188.5 | 334.4 | 112.1 | 23.20 | |
| - | ColorView | 94.40 | 3.30 | 15.00 | 35.10 | 384.7 | 17.10 | 20.80 | 14.50 | 8.30 | 4.70 | 240.6 | 421.2 | 400.8 | 51.60 | |
| - | MG+Color | -- | -- | 15.10 | 38.80 | 354.6 | 17.80 | -- | -- | -- | 2.50 | 213.7 | 390.0 | 331.9 | 48.40 | |
| - | MG+OBJ+Color | -- | -- | 2.50 | 40.20 | 376.7 | 5.10 | -- | -- | -- | 8.70 | 257.3 | 430.9 | 648.8 | 25.20 |
95.1%Average non-repeated component identification under ColorView, versus only 26.4% for repeated-instance identification.
27.9%Best one-shot Pos@10, showing that metric translation remains substantially harder than coarse orientation estimation.
23.3%Best one-shot graph similarity, highlighting incomplete or incorrect mate-edge recovery even when local pair metrics improve.
Quantitative results
Agentic Results
Agentic runs iteratively revise the complete structured prediction using render feedback, optional mate-graph conditioning, and geometric conflict reports. Runs stop when the model declines further tools or after ten rounds.
| Model | Size | Feedback | ID-NR ↑ | ID-R ↑ | Pos@10 ↑ | Rot@10 ↑ | CDᵢ ↓ | PA ↑ | PairF1 ↑ | TypeAcc ↑ | GSim ↑ | C-Free ↑ | SCD ↓ | PosErr ↓ | Step ↓ | RT ↓ | PR ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.5 | - | 2DVis | 98.40 | 54.50 | 18.70 | 46.70 | 182.5 | 30.30 | 32.70 | 43.40 | 15.50 | 16.00 | 120.6 | 224.7 | 1.41 | 148.0 | 62.00 |
| - | MG+2DVis | -- | -- | 21.20 | 50.10 | 189.5 | 33.30 | -- | -- | -- | 20.00 | 135.6 | 246.0 | 1.14 | 130.9 | 53.74 | |
| - | Conflict+2DVis | 100.00 | 46.30 | 17.50 | 42.70 | 227.9 | 28.20 | 30.50 | 33.40 | 14.60 | 20.70 | 169.1 | 291.6 | 2.18 | 218.9 | 70.96 | |
| GPT-5-mini | - | 2DVis | 98.10 | 13.90 | 18.20 | 44.90 | 209.5 | 27.20 | 28.60 | 11.90 | 13.00 | 27.30 | 124.6 | 230.3 | 1.28 | 22.0 | 50.22 |
| - | MG+2DVis | -- | -- | 18.30 | 48.30 | 163.4 | 29.70 | -- | -- | -- | 20.90 | 102.4 | 187.0 | 1.27 | 20.5 | 44.34 | |
| - | Conflict+2DVis | 100.00 | 11.10 | 16.20 | 47.30 | 232.4 | 27.20 | 31.80 | 13.60 | 15.10 | 27.70 | 151.3 | 270.5 | 1.64 | 32.7 | 49.21 | |
| Claude Opus 4.8 | - | 2DVis | 98.60 | 23.20 | 15.70 | 37.70 | 874.6 | 18.70 | 35.00 | 36.70 | 6.10 | 13.70 | 702.2 | 875.0 | 2.69 | 55.1 | 87.07 |
| - | MG+2DVis | -- | -- | 16.70 | 38.90 | 868.0 | 20.80 | -- | -- | -- | 13.40 | 716.7 | 893.4 | 3.17 | 60.0 | 87.82 | |
| - | Conflict+2DVis | 98.60 | 21.70 | 15.00 | 36.80 | 347.6 | 19.60 | 35.10 | 32.80 | 5.40 | 17.30 | 216.2 | 397.0 | 2.13 | 44.0 | 84.25 | |
| Gemini 3.1 Pro | - | 2DVis | 96.80 | 33.30 | 21.60 | 54.30 | 163.3 | 37.00 | 42.30 | 33.80 | 21.90 | 14.30 | 92.5 | 195.3 | 1.59 | 241.2 | 25.48 |
| - | MG+2DVis | -- | -- | 30.70 | 59.70 | 174.3 | 48.90 | -- | -- | -- | 15.00 | 100.1 | 227.5 | 1.67 | 255.9 | 32.21 | |
| - | Conflict+2DVis | 100.00 | 100.00 | 34.60 | 66.00 | 85.4 | 57.10 | 42.60 | 44.70 | 16.00 | 23.10 | 39.9 | 103.8 | 1.42 | 283.4 | 11.59 | |
| Gemini 2.5 Flash-lite | - | 2DVis | 100.00 | 42.90 | 22.70 | 48.80 | 187.0 | 37.80 | 19.60 | 15.80 | 23.10 | 16.00 | 101.9 | 221.2 | 3.85 | 46.4 | 36.50 |
| - | MG+2DVis | -- | -- | 21.20 | 56.50 | 141.6 | 29.30 | -- | -- | -- | 9.10 | 96.5 | 184.7 | 1.05 | 38.1 | 17.19 | |
| - | Conflict+2DVis | 100.00 | 25.00 | 17.70 | 65.20 | 104.7 | 30.40 | 34.20 | 25.10 | 14.10 | 43.80 | 79.7 | 118.4 | 1.88 | 66.5 | 10.92 | |
| LLaMA4 Maverick | 400B | 2DVis | 100.00 | 25.00 | 26.50 | 55.30 | 280.6 | 28.20 | 36.20 | 20.40 | 15.80 | 11.40 | 183.3 | 179.3 | 1.00 | 42.8 | 17.50 |
| 400B | MG+2DVis | -- | -- | 21.30 | 53.00 | 182.9 | 33.50 | -- | -- | -- | 18.60 | 143.9 | 200.6 | 1.00 | 51.5 | 22.50 | |
| 400B | Conflict+2DVis | 96.00 | 16.70 | 32.80 | 61.80 | 278.3 | 36.10 | 40.20 | 24.90 | 22.30 | 19.40 | 166.6 | 160.2 | 2.29 | 67.9 | 30.74 | |
| Ministral | 14B | 2DVis | 100.00 | 37.50 | 20.20 | 51.20 | 213.0 | 32.50 | 41.30 | 18.30 | 15.30 | 37.00 | 166.6 | 238.8 | 1.36 | 100.1 | 18.10 |
| 14B | MG+2DVis | -- | -- | 19.60 | 47.40 | 231.7 | 28.50 | -- | -- | -- | 40.70 | 164.1 | 266.7 | 2.71 | 96.4 | 30.65 | |
| 14B | Conflict+2DVis | 100.00 | 62.50 | 24.50 | 52.50 | 234.8 | 27.30 | 40.60 | 18.30 | 13.20 | 29.60 | 163.0 | 270.9 | 2.54 | 94.6 | 29.22 | |
| Qwen3.5-9B | 9B | 2DVis | 92.90 | 28.60 | 19.20 | 52.50 | 207.6 | 24.30 | 28.80 | 27.00 | 17.30 | 19.00 | 153.7 | 190.2 | 1.00 | 150.3 | 10.50 |
| 9B | MG+2DVis | -- | -- | 16.70 | 55.80 | 71.7 | 32.50 | -- | -- | -- | 40.00 | 44.2 | 100.5 | 1.00 | 867.1 | 2.50 | |
| 9B | Conflict+2DVis | 88.90 | 40.00 | 26.20 | 49.30 | 188.0 | 30.90 | 36.40 | 29.40 | 19.50 | 35.70 | 119.1 | 202.8 | 1.00 | 189.0 | 7.00 | |
| Qwen3.7-Plus | - | 2DVis | 97.90 | 17.60 | 21.90 | 54.80 | 613.1 | 29.50 | 33.70 | 36.20 | 21.40 | 24.60 | 378.9 | 633.9 | 2.03 | 272.9 | 49.44 |
| - | MG+2DVis | -- | -- | 29.30 | 66.50 | 260.1 | 37.40 | -- | -- | -- | 27.90 | 234.5 | 276.0 | 3.88 | 436.6 | 51.54 | |
| - | Conflict+2DVis | 97.60 | 14.30 | 24.70 | 55.50 | 899.7 | 31.20 | 36.20 | 33.60 | 24.50 | 23.20 | 535.7 | 990.4 | 2.12 | 300.4 | 45.25 |
34.6%Best agentic Pos@10, obtained by Gemini 3.1 Pro with conflict-guided visual feedback.
57.1%Best agentic placement accuracy, improving coarse assembly placement but still leaving substantial fine-alignment error.
43.8%Best conflict-free rate. More than half of predictions still contain geometric conflicts even with explicit diagnostics.
Authors & institutions
OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies
Shanghai Jiao Tong UniversityPennsylvania State UniversityIndependent ResearcherMicrosoft ResearchBinghamton University
Citation
BibTeX
arXiv:2608.22637 · cs.CV
@misc{wang2026omnicadlargescalebenchmark3d,
title={OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies},
author={Mingjia Wang and Taiting Lu and Ziwei Dong and Sisong Bei and Jingying Zeng and Runze Liu and Kaiyuan Lin and Hongxing Pan and Kai Zhang and Yizheng Hou and Yangshoudu Zheng and Chenchen Guo and Weiyuan Meng and Shubin Lyu and Zhijun Zheng and Dexu Wang and Xinyu Bai and Shurui Qian and Zhangzixin and Mengyu Pan and Guoliang Shi and Ling Ma and Yifan Yang and Qi He and Yi-Chao Chen and Yincheng Jin and Sung-Liang Chen and Mahanth Gowda},
year={2026},
eprint={2608.22637},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.22637},
}