Niranjan Balasubramanian, Raymond Mooney, Nathanael Chambers, Yash Lal
We lifted 6 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| StonyBrookNLP/tellmewhy | canonical | 5 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| create_inputs_for_rouge | Ran | StonyBrookNLP/tellmewhy/src/automatic_evaluation.py pointer only (licence: NONE) · get_code("5be895a93593b73e") |
| create_multi_reference_dictionary_for_gold_sentences | Ran | StonyBrookNLP/tellmewhy/src/automatic_evaluation.py pointer only (licence: NONE) · get_code("0e24b5aef615871d") |
| extract_true_annotations | Ran | StonyBrookNLP/tellmewhy/src/analyse_human_evaluation_output.py pointer only (licence: NONE) · get_code("16b36baf8c9bc5c8") |
| fleiss_kappa | Ran | StonyBrookNLP/tellmewhy/src/analyse_human_evaluation_output.py pointer only (licence: NONE) · get_code("cb3b5d6e87cb16b5") |
| sentence_level_multi_bertscore | Ran | StonyBrookNLP/tellmewhy/src/automatic_evaluation.py pointer only (licence: NONE) · get_code("f3025cd8b5ffa609") |
| weighted_fleiss_kappa | Not yet run | StonyBrookNLP/tellmewhy/src/analyse_human_evaluation_output.py pointer only (licence: NONE) · get_code("0eec09974e44d9dd") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Answering questions about why characters perform certain actions is central to understanding and reasoning about narratives. Despite recent progress in QA, it is not clear if existing models have the ability to answer "why" questions that may require commonsense knowledge external to the input narrative. In this work, we introduce TellMeWhy, a new crowd-sourced dataset that consists of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. For a third of this dataset, the answers are not present within the narrative. Given the limitations of automated evaluation for this task, we also present a systematized human evaluation interface for this dataset. Our evaluation of state-of-the-art models show that they are far below human performance on answering such questions. They are especially worse on questions whose answers are external to the narrative, thus providing a challenge for future QA and narrative understanding research.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2106.06132")
get_code_for_paper("2106.06132")
have("2106.06132")
Connect an agent — have() is free.