Youngjae Yu, Seungju Han, Jiwan Chung, ♠ Junhyeok, Kim Jack, Hessel Liwei, Jiang ♢♡, ♣ Yejin, Son Yejin, Choi ♢♡
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Commonsense norms are defeasible by context: reading books is usually great, but not when driving a car. While contexts can be explicitly described in language, in embodied scenarios, contexts are often provided visually. This type of visually grounded reasoning about defeasible commonsense norms is generally easy for humans, but (as we show) poses a challenge for machines, as it necessitates both visual understanding and reasoning about commonsense norms. We construct a new multimodal benchmark for studying visual-grounded commonsense norms: NORMLENS. NORMLENS consists of 10K human judgments accompanied by freeform explanations covering 2K multimodal situations, and serves as a probe to address two questions: (1) to what extent can models align with average human judgment? and (2) how well can models explain their predicted judgments? We find that state-of-theart model judgments and explanations are not well-aligned with human annotation. Additionally, we present a new approach to better align models with humans by distilling social commonsense knowledge from large language models. The data and code are released at https://seungjuhan.me/normlens.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.10418")
get_code_for_paper("2310.10418")
have("2310.10418")
Connect an agent — have() is free.