SGAligner: 3D Scene Alignment with Scene Graphs

Marc Pollefeys, Iro Armeni, Sayan Sarkar, Daniel Barath, Ondrej Miksik

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 10 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• sayands/sgaligner — marked official in Papers with Code

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Building 3D scene graphs has recently emerged as a topic in scene representation for several embodied AI applications to represent the world in a structured and rich manner. With their increased use in solving downstream tasks (e.g., navigation and room rearrangement), can we leverage and recycle them for creating 3D maps of environments, a pivotal step in agent operation? We focus on the fundamental problem of aligning pairs of 3D scene graphs whose overlap can range from zero to partial and can contain arbitrary changes. We propose SGAligner, the first method for aligning pairs of 3D scene graphs that is robust to in-the-wild scenarios (i.e., unknown overlap -if anyand changes in the environment). We get inspired by multimodality knowledge graphs and use contrastive learning to learn a joint, multi-modal embedding space. We evaluate on the 3RScan dataset and further showcase that our method can be used for estimating the transformation between pairs of 3D scenes. Since benchmarks for these tasks are missing, we create them on this dataset. The code, benchmark, and trained models are available on the project website.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2304.14880")
get_code_for_paper("2304.14880")
have("2304.14880")

Connect an agent — have() is free.