Lin, Zheng Zhang, Yue Zhang, Yangqiu Song, Hongming Zhang, Weiqi Wang, Cheng Jiayang, Qiu Tsz, Ho Chan, ♠ Chunkit, Chan Dongyu, Ru Qipeng
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Analogy-making between narratives is crucial for human reasoning. In this paper, we evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus, STORYANALOGY, which contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory. We design a set of tests on STO-RYANALOGY, presenting the first evaluation of story-level analogy identification and generation. Interestingly, we find that the analogy identification tasks are incredibly difficult not only for sentence embedding models but also for the recent large language models (LLMs) such as ChatGPT and LLaMa. ChatGPT, for example, only achieved around 30% accuracy in multiple-choice questions (compared to over 85% accuracy for humans). Furthermore, we observe that the data in STORYANALOGY can improve the quality of analogy generation in LLMs, where a fine-tuned FlanT5-xxl model achieves comparable performance to zero-shot ChatGPT.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.12874")
get_code_for_paper("2310.12874")
have("2310.12874")
Connect an agent — have() is free.