Roger Wattenhofer, Yunpu Ma, Zhao Meng, Gerhard Lakemeyer, Yiran Xing, Zai Shi
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present Knowledge Enhanced Multimodal BART (KM-BART), which is a Transformerbased sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. We adapt the generative BART architecture (Lewis et al., 2020) to a multimodal model with visual and textual inputs. We further develop novel pretraining tasks to improve the model performance on the Visual Commonsense Generation (VCG) task. In particular, our pretraining task of Knowledge-based Commonsense Generation (KCG) boosts model performance on the VCG task by leveraging commonsense knowledge from a large language model pretrained on external commonsense knowledge graphs. To the best of our knowledge, we are the first to propose a dedicated task for improving model performance on the VCG task. Experimental results show that our model reaches state-of-the-art performance on the VCG task (Park et al., 2020) by applying these novel pretraining tasks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2101.00419")
get_code_for_paper("2101.00419")
have("2101.00419")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2101.00419.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2101.00419)
Connect an agent — have() is free.