Hongsheng Li, Kai-Wei Chang, Yichi Zhang, Yu Qiao, Renrui Zhang, Peng Gao, Pan Lu, Ziyu Guo, Aojun Zhou, Dongzhi Jiang, Haokun Lin, Pengshuo Qiu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
The remarkable progress of Multi-modal Large Language Models (MLLMs) has gained unparalleled attention. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows Math-Verse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging true or false, we employ GPT-4(V) to adaptively assess each step with error analysis to derive a total score, which can reveal the inner CoT reasoning quality by MLLMs. With MathVerse, we unveil that, most existing MLLMs struggle to understand math diagrams, relying heavily on textual questions. Surprisingly, some of them even achieve 5%+ higher accuracy without the visual input. Besides, GPT-4V and MAVIS-7B achieve the best overall performance within closedsource and open-source models, respectively. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.14624")
get_code_for_paper("2403.14624")
have("2403.14624")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2403.14624.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2403.14624)
Connect an agent — have() is free.