SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2410.00201 · ECCV · 2024

DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation

Xin Li, Jeffrey Bigham, Jason Wu, Faria Huq, Yue Jiang, Yi-Hao Peng, Amy Pavel

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such understanding computationally has required manual data collection and annotation, which is time-consuming and labor-intensive. To overcome this challenge, we present a method to generate synthetic, structured visuals with target labels using code generation. Our method allows people to create datasets with built-in labels and train models with a small number of human-annotated examples. We demonstrate performance improvements in three tasks for understanding slides and UIs: recognizing visual elements, describing visual content, and classifying visual content types.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2410.00201")
get_code_for_paper("2410.00201")
have("2410.00201")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2410.00201.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2410.00201.svg)](https://syntology.ai/paper/2410.00201)

Connect an agent — have() is free.