Real2Sim: Scan-to-Simulation Compiler

The description states that Real2Sim is a tool that converts synchronized RGB-D room scans into semantic, collision-ready USD scenes for robot simulation. It uses ConceptGraphs to detect and fuse objects across multiple camera views, generates static collision proxies, and exports artifacts in USD format with JSON manifests.

• Input: RGB images, metric depth, camera intrinsics, poses, reference mesh from ARKitScenes.
• Processing pipeline includes:
  • Conversion to ConceptGraphs-compatible dataset.
  • Multi-view object detection using YOLO-World, MobileSAM, and depth projection.
  • Filtering transient detections.
  • Export of retained objects in JSON format.
  • USD scene generation with semantic point-cloud visuals and static collision boxes.
• Output: USD scenes, JSON manifests, validation tests, rover route plans.

This is a technical tool for robotics developers working on simulation environments. It is not described as a SaaS product or platform.

Evidence Self-reported by the author. No third-party verification.

The description states that Real2Sim was inspired by the question: “Can we turn an ordinary mobile room scan into a useful robot-simulation scene automatically?” It aims to avoid perfect reproduction and instead focus on practicality — preserving recognizable objects, producing usable collision geometry, and exporting artifacts for downstream tools.

Key claims:

• Not about reproducing every surface perfectly.
• Focuses on usability, not visual fidelity.
• Produces artifacts that downstream robotics tools can consume.
• Emphasizes honesty in uncertainty, reproducibility, and validation.

The project evolved from an idea to a working MVP with automated tests, manifests, and a demonstration rover route. It does not claim to be a full simulation platform or autonomous navigation system — rather, it compiles real-world scans into simulation-ready assets.

Evidence Self-reported. No external positioning data or marketing materials provided.

The description implies that Real2Sim targets robotics developers working in simulation environments, particularly those using tools like Pixar USD and Isaac Sim. It is designed for users who want to convert real-world scans into simulation-ready scenes without manual rebuilding.

It does not state specific customer segments or personas beyond the general category of robotics teams.

Evidence Self-reported. No explicit ICP or customer segmentation described.

The description does not mention any pricing, monetization strategy, or business model. It is presented as a hackathon project with no indication of commercialization or revenue streams.

Evidence Not evidenced.

• Uses ARKitScenes RGB-D capture as input.
• Leverages ConceptGraphs for perception and object fusion.
• Integrates tools like YOLO-World, MobileSAM, OpenCV, Open3D, PyTorch, GPT-5.6 (via Codex), and Pixar USD.
• Processes 596-frame ARKitScenes sequence with stride five.
• Produces a 49-object semantic reconstruction aligned with the source mesh (median distance ~1.44 cm).
• Generates static collision boxes for each object.
• Includes JSON schemas, manifests, and automated tests.
• Does not require an OpenAI API key for final pipeline execution.

Evidence Self-reported. No independent verification of technical claims or performance metrics.

The description states that this is a hackathon submission (OpenAI 2026) and that the MVP includes:

• A complete RGB-D-to-USD pipeline.
• A 49-object semantic reconstruction.
• Static collision proxies.
• Strict JSON schemas and simulation manifests.
• Automated tests for import, conversion, export, route validation, and video generation.
• A one-command judge quickstart.
• Honest documentation of what is implemented vs. future work.

There is no mention of:

• Customers or users.
• Revenue or funding.
• Product adoption or usage metrics.
• Production deployment or integration with external tools beyond the MVP.

Evidence Self-reported. No traction or maturity data beyond internal development and demonstration.

The description does not reference any competitors or market positioning. It does not describe how Real2Sim compares to existing tools for converting real-world scans into simulation environments, nor does it mention similar projects or platforms in the robotics or simulation space.

Evidence Not evidenced.

• No external validation or adoption: The project is presented as a hackathon submission with no evidence of traction or customer use.
• Limited scope: Focuses on static collision boxes and does not yet support articulated objects, dynamic physics, or full simulation environments.
• Self-reported performance metrics: While it includes alignment data (median ~1.44 cm), there is no independent benchmarking or comparison to other tools.
• No commercialization strategy: No pricing, monetization, or business model described.
• Dependency on niche tools: Relies heavily on ConceptGraphs and specific frameworks like GPT-5.6, which may limit scalability or portability.

Evidence Self-reported. Risks inferred from lack of external data and limited scope.

1. What is the intended use case for Real2Sim beyond the MVP?
2. Are there any plans to commercialize this tool, and if so, what is the business model?
3. How does Real2Sim compare to existing tools in the robotics simulation space (e.g., Isaac Sim, Gazebo)?
4. What are the limitations of the current pipeline that would prevent it from scaling to larger or more complex environments?
5. Has the tool been tested with other RGB-D capture systems beyond ARKitScenes?
6. How is uncertainty in object detection and reconstruction handled in practice?
7. Are there any plans to support dynamic or articulated objects, or to integrate with physics engines?
8. What are the long-term goals for the project, and how do they align with current market needs?