Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Wonmin Byeon, Xiaolong Wang, Arash Vahdat, Shalini De Mello, Sifei Liu, Jiarui Xu

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.

Repositories linked to this paper

nvlabs/odise — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Figure 1. We learn open-vocabulary panoptic segmentation with the internal representation of text-to-image diffusion models. K-Means clustering of the diffusion model's internal representation shows semantically differentiated and localized information wherein objects are well grouped together (middle figure). We leverage these dense and rich diffusion features to perform open-vocabulary panoptic segmentation (right figure).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2303.04803")
get_code_for_paper("2303.04803")
have("2303.04803")

Connect an agent — have() is free.