Graham Neubig, Emmy Liu, Chenxuan Cui, Kenneth Zheng
We lifted 14 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| nightingal3/fig-qa | canonical | 1 of 2 |
| simran-khanuja/multilingual-fig-qa | pwc_unofficial | 4 of 12 |
| Function | Status | Where it lives |
|---|---|---|
| eval_model | Ran | simran-khanuja/multilingual-fig-qa/run_baselines.py code served (permissive licence) · get_code("31e5d4061c0165ee") |
| evaluate_model | Ran | nightingal3/fig-qa/src/models/gpt_score.py code served (permissive licence) · get_code("2a5615f598f3af2b") |
| is_subarray | Ran | simran-khanuja/multilingual-fig-qa/get_syntax_chunks.py code served (permissive licence) · get_code("21007e567ee7726b") |
| main_train_loop | Ran | simran-khanuja/multilingual-fig-qa/run_baselines.py code served (permissive licence) · get_code("454507be28be36dd") |
| substring_between | Ran | simran-khanuja/multilingual-fig-qa/get_syntax_chunks.py code served (permissive licence) · get_code("62a67c92331edd10") |
| convert_examples_to_features | Not yet run | simran-khanuja/multilingual-fig-qa/utils.py code served (permissive licence) · get_code("4f74b4458496f1dc") |
| convert_multiple_choice_examples_to_features | Not yet run | simran-khanuja/multilingual-fig-qa/utils.py code served (permissive licence) · get_code("a7ffc645f7b1c747") |
| get_hypernyms | Not yet run | simran-khanuja/multilingual-fig-qa/plot_parsed_data.py code served (permissive licence) · get_code("10b84b6e2e00afe1") |
| get_pos_tags | Not yet run | simran-khanuja/multilingual-fig-qa/plot_parsed_data.py code served (permissive licence) · get_code("16090c8ed2178c61") |
| lemmatize_words | Not yet run | simran-khanuja/multilingual-fig-qa/plot_parsed_data.py code served (permissive licence) · get_code("792128008f28fb12") |
| remove_substrings | Not yet run | simran-khanuja/multilingual-fig-qa/get_syntax_chunks.py code served (permissive licence) · get_code("581a4523d21cb660") |
| sent_scoring | Not yet run | nightingal3/fig-qa/src/models/gpt_score.py code served (permissive licence) · get_code("39cf438241a8d1dd") |
| simple_accuracy | Not yet run | simran-khanuja/multilingual-fig-qa/utils.py code served (permissive licence) · get_code("3c241ecfe3749a6d") |
| train_model | Not yet run | simran-khanuja/multilingual-fig-qa/run_baselines.py code served (permissive licence) · get_code("e7b3e41b3b961fb4") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero-or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2204.12632")
get_code_for_paper("2204.12632")
have("2204.12632")
Connect an agent — have() is free.