Ludwig Schmidt, Yuhui Zhang, Xiaohan Wang, Elaine Sui, Serena Yeung-Levy, Chenyu Wang, Yuchang Su, Yiming Liu, James Burgess, Alejandro Lozano, Anjiang Wei, Josiah Aklilu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The rapid development of vision language models (VLMs) demands rigorous and reliable evaluation. However, current visual question answering (VQA) benchmarks often depend on open-ended questions, making accurate evaluation difficult due to the variability in natural language responses. To address this, we introduce Auto-Converter, an agentic framework that automatically converts these open-ended questions into multiple-choice format, enabling objective evaluation while reducing the costly multiple-choice question creation process. Our experiments demonstrate that AutoConverter can generate correct and challenging multiple-choice questions, with VLMs demonstrating consistently similar or lower accuracy on these questions compared to human-created ones. Using Au-toConverter, we construct VMCBench, a benchmark created by transforming 20 existing VQA datasets into a unified multiple-choice format, totaling 9,018 questions. We comprehensively evaluate 33 state-of-the-art VLMs on VM-CBench, setting a new standard for scalable, consistent, and reproducible VLM evaluation.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2501.03225")
get_code_for_paper("2501.03225")
have("2501.03225")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2501.03225.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2501.03225)
Connect an agent — have() is free.