Hua Huang, Guang Liu, Xi Yang, Pengfei Zhou, Zheqi He, Richeng Xuan, Qiannan Zhu, Xinya Wu
We lifted 4 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| FlagOpen/CMMU | canonical | 4 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_hash | Ran | FlagOpen/CMMU/eval/chat_llm.py pointer only (licence: NONE) · get_code("60f0c4e169f36eba") |
| process_multiple_choice | Ran | FlagOpen/CMMU/eval/pre_process.py pointer only (licence: NONE) · get_code("e1f51b8f5fb011a3") |
| remove_special_characters | Ran | FlagOpen/CMMU/eval/pre_process.py pointer only (licence: NONE) · get_code("25043297679d4f48") |
| strip_answer | Ran | FlagOpen/CMMU/eval/pre_process.py pointer only (licence: NONE) · get_code("95acf3c47b2072f6") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domainspecific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fillin-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiplechoice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.14011")
get_code_for_paper("2401.14011")
have("2401.14011")
Connect an agent — have() is free.