Sejong Kim, Hyunjun Kim
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce MacroBench, a code-first benchmark that evaluates whether LLMs can synthesize reusable browser-automation programs (macros) from natural-language goals by reading HTML/DOM and emitting Python+Selenium. MacroBench instantiates seven self-hosted sites-Airbnb-like, TikTok-like, Redditlike, Instagram-like, Facebook-like, Discord-like, and Threads-like -covering 681 tasks across interaction complexity and targeting difficulty. Our end-to-end protocol validates generated code via static checks, sandboxed execution, and outcome verification (DOM assertions, database snapshots), and includes a safety suite for scraping, spam/abuse, and credential/privacy prompts. Across 2,636 model-task runs, we observe stratified success: GPT-4o-Mini (96.8%), GPT-4.1 (95.3%), Gemini-2.5-Pro (89.0%), DeepSeek-V3.1 (83.4%). Models handle simple tasks reliably (91.7%) but fail on complex workflows (0.0%), and none meet production-quality coding practices despite functional completion. We release our complete benchmark pipeline, evaluation framework, and experimental results at https://github.com/hyunjun1121/MacroBench to enable reproducible assessment of macro synthesis for web automation.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2510.04363")
get_code_for_paper("2510.04363")
have("2510.04363")
Connect an agent — have() is free.