MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs

1. Introduction

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable progress in complex perceptual and reasoning tasks. However, effectively solving many real-world tasks fundamentally depends on the ability to perceive and reason about scenes from multiple viewpoints.

Humans naturally perform multi-angle observations to construct coherent mental models of objects, resolving perceptual ambiguities that arise from single viewpoints. This ability is crucial when assembling complex objects, where each component must be rotated and inspected from multiple viewpoints to determine how it connects with others.

Current multi-view spatial reasoning benchmarks primarily assess egocentric spatial reasoning such as perspective-taking, view-dependent navigation, or dynamic viewpoint transformation. These benchmarks evaluate a model's ability to map pixels from one 2D plane to another or to predict the visual consequences of camera motion without targeting view-invariant (world-centric) spatial understanding and holistic 3D grounding. Such world-centric reasoning ability is a prerequisite for mechanical engineering and operating 3D modeling software (e.g. Blender) where the agent needs to manipulate in global space, independent of where the user is looking. We address this gap by introducing MultiView-Bench.

2. Motivation: Furniture Assembly as a Case Study

As LLMs and VLMs are increasingly deployed as agents to control software tools, measuring their proficiency within these interfaces is critical. To illustrate the necessity and practical importance of our benchmark, we motivate our study through a furniture part assembly task in Blender GUI. In this setting, a collection of labeled components (e.g., legs, table tops, backrests) must be connected and arranged to form a functional piece of furniture. Solving this task naturally demands multi-view perception, 3D spatial reasoning, and common-sense knowledge about how objects are typically used and combined.

While recent LLM-based approaches have shown success in environments like Minecraft or Blender where assemblies involve standardized blocks, real-world furniture assembly poses significantly greater complexity due to irregular, non-convex shapes that defy concise linguistic descriptions.

Figure 2: Top Left: Many real-world objects do not lend themselves to simple natural language description. The table on the right can be described using fundamental convex shapes and their bounding boxes but the chair on the left has non-convex parts without an analytical expression. Top Right: Often times single-view observation leads to visual misconception and does not reveal some alignment issues. The chair looks well-assembled in the view shown on the left but when it turns to the view shown on the right we see the backrest is slightly misplaced in the X-axis. Bottom: Only using the bounding box dimensions, we are unable to assemble furniture that have non-convex parts.

3. MultiView-Bench

MultiView-Bench is a foundational evaluation designed to test VLMs' multi-view spatial reasoning capabilities, preparing them for complex real-world tasks like mechanical engineering or 3D scene reconstruction. The core task of MultiView-Bench assesses a VLM's ability to reason about the relative positions of objects within a 3D scene. VLMs must observe scenes from multiple viewpoints to infer spatial relationships accurately。

3.1 Dataset Creation

We design a modular pipeline that procedurally generates diverse 3D scenes with controlled variations:

Figure 3: MultiView-Bench's extensible data generation pipeline supports plug-and-play 3D assets and flexible camera configurations

3D Assets

For synthetic tasks, we use fundamental geometric objects randomly sampled from cubes, spheres, cylinders, and cones. For real-world objects, we use the 3DCoMPaT++ dataset consisting of thousands of real-world objects from different categories like tables, chairs, and airplanes. All objects are rescaled to share a common bounding box to minimize visual ambiguity.

Object Placement

We fix a central object at the origin and randomize other objects' positions while enforcing minimum and maximum separation distances. To analyze model limitations, we construct controlled task variants:

• DoF=1: Objects placed along the same axis
• DoF=2: Objects lie on the same 2D plane
• DoF=3: No constraints of placement in 3D space

Camera Viewpoints

In the main benchmark, we render six viewpoints with uniformly distributed azimuth angles and slight elevations, guaranteeing visibility of all three axes. The pipeline supports arbitrary viewpoint configurations, enabling analysis of inductive biases and creation of specialized tasks.

3.2 Task Examples

Task 1

Show Answer

Answer:

The table is positioned at (-X, -Y, 0Z) relative to the chair.

Task 2

Show Answer

Answer:

The cube is positioned at (+X, +Y, 0Z) relative to the cylinder.

Task 3

Show Answer

Answer:

The cube is positioned at (+X, 0Y, 0Z) relative to the sphere.

Frontier Models Cannot Solve It Yet

We systematically evaluate seven leading VLMs on MultiView-Bench: Claude 3.7 Sonnet, Claude 4 Sonnet, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, GPT-5, and GPT-o3.

Figure: Model performance on MultiView-Bench across task variants. The red dashed line indicates random chance (~3.7%).

Performance Overview

The results reveal a striking finding: most frontier models perform near random chance on the hardest tasks. On 3D DoF=3 and 3D Real World, models like Claude series and GPT-4o barely exceed the 3.7% random baseline. Even GPT-5, the best-performing model, achieves only 50% accuracy on 3D DoF=3—far from reliable for real-world deployment. We have the following key observations:

• Task difficulty scales with DoF: As we decrease DoF from 3 to 1, accuracy improves dramatically. Claude 3.7 Sonnet reaches near GPT-5 level when DoF=1, suggesting models handle simpler spatial configurations but fail at full 3D reasoning.
• Single view is insufficient: Providing only a single isometric view (clearly displaying all three axes) drops all models to random chance, confirming that multi-view integration is essential—not merely helpful.
• Real-world objects are harder: Performance on 3D Real World (using 3DCoMPaT++ assets) is consistently lower than synthetic primitives, indicating additional challenges from complex geometry and textures.

Failure Pattern Analysis

To understand why models fail, we decompose the spatial reasoning task into intermediate sub-steps analogous to human reasoning:

• Step 1 - Object Identification: Recognizing and distinguishing individual objects.
• Step 2 - 2D Spatial Relation: Articulating relative object positions within a given 2D viewpoint.
• Step 3 - Axis Direction Identification: Interpreting the directionality of each axis within the 3D scene.
• Step 4 - 3D Position Translation: Translating observed 2D relationships into explicit 3D positional descriptions.

Our analysis reveals that VLMs succeed at Steps 1 and 2 but fail critically at Step 3—identifying axis directions in 3D context. Models consistently struggle to express axis directions using unambiguous 3D spatial language, even when the coordinate system is clearly visible.

Motivated by this observation, we explored whether decomposing the task into simpler 2D views can improve performance. We test two configurations: (1) Single-agent (all three views provided simultaneously) and (2) Multi-agent (each agent independently assesses one view).

Results reveal two key findings:

• (i) simplifying 3D to 2D dramatically improves performance
• (ii) decomposing multi-view into single-view sub-tasks yields further gains

This confirms that frontier VLMs struggle with both 3D perception and multi-view integration.

Figure: Simplifying 3D to 2D dramatically improves performance, and decomposing multi-view into single-view sub-tasks yields further gains. This confirms that frontier VLMs struggle with both 3D perception and multi-view integration.

Bias Analysis

Visual Enhancements: We explore how visual aids (color schemes, grids) influence performance. Prior work suggests structured visual aids improve VLM reasoning. Our experiments reveal model-specific effects: grids and distinct colors help Claude series but surprisingly hurt Gemini 2.5 Pro. We also observe color biases—models show preferential responses to particular color combinations that diverge from human perception.

Coordinate Rotation: We identify a striking pattern: models frequently ignore explicitly depicted coordinate directions, defaulting to the conventional right-handed coordinate system. To test this, we rotate axes to non-standard angles (23°, 45°, 68°, etc.). Results confirm severe performance degradation under unconventional orientations, revealing that VLMs heavily rely on learned coordinate-system priors rather than reasoning from visual evidence.

(a) Color bias: VLM performance fluctuates across color combinations in ways that diverge from human perception (e.g., performing worse on distinct Olive/Coral pairs).

(b) Rotation bias: Performance degrades under non-standard axis orientations, revealing reliance on memorized conventions.

6. ViewNavigator: Can We Augment VLMs with Agentic Reasoning?

We introduce ViewNavigator, a multi-agent system designed to actively reason about spatial relationships between objects within a 3D environment. Our agent architecture integrates a VLM and an LLM in a closed-loop manner without requiring post-training or external geometry-based image analysis.

Figure 8: ViewNavigator actively selects informative viewpoints, perceives, and fuses multi-view evidence through belief-updating

How ViewNavigator Works

• LLM Planner: Strategically plans the next viewpoint to maximize information gain
• VLM Perception: Processes visual inputs from selected viewpoints and jittered variations
• Belief State: Maintains probabilistic belief over spatial directions using Dirichlet distributions
• Active Selection: Iteratively selects viewpoints until confidence threshold is reached

The agent captures multiple images using micro-jitters (small perturbations around a base viewpoint) to assess stability in the VLM's answers, which serves as a confidence score. ViewNavigator significantly enhances the performance of diverse base models on MultiView-Bench by 3-5x, demonstrating its potential as a plug-and-play reasoning scaffold.

Figure 9: ViewNavigator framework significantly enhances various base models’ performances on the 3D DoF=3 tasks.

7. Conclusion

We present MultiView-Bench, a diagnostic benchmark for evaluating VLMs' ability to integrate multi-view observations into world-centric 3D understanding. Our evaluation reveals that frontier VLMs perform near random chance on challenging 3D tasks, failing primarily at axis direction identification and multi-view integration. We also introduce ViewNavigator, a multi-agent framework that improves performance by 3-5x through probabilistic belief aggregation and active view selection. Together, MultiView-Bench and its extensible pipeline serve as both a diagnostic tool for understanding VLM limitations and a selection criterion for building VLM-based agents in 3D applications such as CAD software and robotic assembly.