Yu Wang, Yuhao Wang, Yanfeng Wang, Yusheng Liao, Hongcheng Liu, Heyang Liu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Language Models (MLLMs) have demonstrated exceptional capabilities in visual perception and understanding. However, these models also suffer from hallucinations, which limit their reliability as AI systems. We believe that these hallucinations are partially due to the models' struggle with understanding what they can and cannot perceive from images, a capability we refer to as self-awareness in perception. Despite its importance, this aspect of MLLMs has been overlooked in prior studies. In this paper, we aim to define and evaluate the selfawareness of MLLMs in perception. To do this, we first introduce the knowledge quadrant in perception, which helps define what MLLMs know and do not know about images. Using this framework, we propose a novel benchmark, the Self-Awareness in Perception for MLLMs (MM-SAP), specifically designed to assess this capability. We apply MM-SAP to a variety of popular MLLMs, offering a comprehensive analysis of their self-awareness and providing detailed insights. The experiment results reveal that current MLLMs possess limited selfawareness capabilities, pointing to a crucial area for future advancement in the development of trustworthy MLLMs. Code and data are available at https://github.com/YHWmz/ MM-SAP.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.07529")
get_code_for_paper("2401.07529")
have("2401.07529")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2401.07529.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2401.07529)
Connect an agent — have() is free.