Eduard Hovy, Yonatan Bisk, Artidoro Pagnoni, Evangelia Spiliopoulou
We lifted 16 functions out of this paper's own repositories and ran 10 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| spilioeve/eventsrealm | canonical | 10 of 16 |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally, Large Language models (LLM) have been exposed to procedural knowledge about how objects interact, yet our benchmarking shows they fail to reason about the world. Conversely, we also demonstrate that existing approaches often misrepresent the surprising abilities of LLMs via improper task encodings and that proper model prompting can dramatically improve performance of reported baseline results across multiple tasks. In particular, our results indicate that our prompting technique is especially useful for unseen attributes (out-of-domain) or when only limited data is available. 1 * Equal contribution. † Work completed before joining AWS AI Labs. 1 https://github.com/spilioeve/eventsrealm The robot holds a laptop. The robot forcefully throws the laptop. Pick up the yogurt, bananas, and sorbet. Place the ingredients in a blender. Blend the mixture until it's smooth in texture. Laptop is broken, picked-up and its location is different. 1. The cleanness, weight, volume and fullness of the blender changed. 2. The texture and appearance of the mixture changed. Query: "" Target: n-dim binary vector, n = #attributes Query each attribute in candidate list Query1: Is the location of the mug different? Target: The location of the mug is different. Query2: Is the temperature of the mug different? Target: The temperature of the mug is unchanged.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2211.05392")
get_code_for_paper("2211.05392")
have("2211.05392")
Connect an agent — have() is free.