James Thorne, Noah Lee, Na An
We lifted 28 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| xfactlab/emnlp2023-LLM-Disagreement | canonical | 1 of 17 |
| google-research/FLAN | canonical | 0 of 10 |
| xfactlab/emnlp2023-llm-disagreement | canonical | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| str2bool | Ran | xfactlab/emnlp2023-LLM-Disagreement/preprocess.py code served (permissive licence) · get_code("7c508037b40522af") |
| collapse_related_tasks | Not yet run | google-research/FLAN/flan/task_splits.py code served (permissive licence) · get_code("206789c8dcfa3c19") |
| dict_postprocess | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/sample.py code served (permissive licence) · get_code("6c55f5bfb0365f70") |
| dict_postprocess | Not yet run | xfactlab/emnlp2023-llm-disagreement/utils/processing.py code served (permissive licence) · get_code("5c81be5eb68c1209") |
| entropy_sample | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/sample.py code served (permissive licence) · get_code("a536795ec1c148df") |
| enumerate_items | Not yet run | google-research/FLAN/flan/tasks.py code served (permissive licence) · get_code("fc3cbdfde1b3cc50") |
| format_from_feature_dictionary | Not yet run | google-research/FLAN/flan/preprocessors.py code served (permissive licence) · get_code("48f4ea4298ef92ff") |
| generate_openai | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/generation.py code served (permissive licence) · get_code("e5b0490b427152f2") |
| generate_prompt_template | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/prompts.py code served (permissive licence) · get_code("e31192c49a763711") |
| is_superglue_task | Not yet run | google-research/FLAN/flan/task_splits.py code served (permissive licence) · get_code("f92e9e42a889c8dd") |
| jensen_shannon | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/metrics.py code served (permissive licence) · get_code("40756590747738d4") |
| kl_divergence | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/metrics.py code served (permissive licence) · get_code("cfd7ee739cccf02d") |
| load_data | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/preprocess.py code served (permissive licence) · get_code("3bf33362230ed1a5") |
| load_json | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/preprocess.py code served (permissive licence) · get_code("7eb40d424f4306a5") |
| load_model | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/models.py code served (permissive licence) · get_code("df8175311a751726") |
| negate | Not yet run | google-research/FLAN/flan/preprocessors.py code served (permissive licence) · get_code("e435cf490873e109") |
| option_postprocess | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/processing.py code served (permissive licence) · get_code("003e5dada7af2bae") |
| parse_glm_qa_answer | Not yet run | google-research/FLAN/flan/postprocessors.py code served (permissive licence) · get_code("486ad3a49d33036e") |
| preprocess_data | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/visualize.py code served (permissive licence) · get_code("c501787fde562c3c") |
| preprocess_data_Nie | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/visualize.py code served (permissive licence) · get_code("444acbf63a2aed61") |
| prompt_fs_generator | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/prompts.py code served (permissive licence) · get_code("425c2228b8849976") |
| prompt_generator | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/prompts.py code served (permissive licence) · get_code("6e27415d7d2bdc18") |
| random_sample | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/sample.py code served (permissive licence) · get_code("2163e5e3c4bcebcd") |
| relative_entropy | Not yet run | xfactlab/emnlp2023-LLM-Disagreement/utils/metrics.py code served (permissive licence) · get_code("7e736694212e7c19") |
| remove_input_patterns_options | Not yet run | google-research/FLAN/flan/utils.py code served (permissive licence) · get_code("252eaeffae9c5998") |
| remove_leading_quotes_and_spaces | Not yet run | google-research/FLAN/flan/postprocessors.py code served (permissive licence) · get_code("7d7424269c38356a") |
| t_name_to_flan_pattern_name | Not yet run | google-research/FLAN/flan/utils.py code served (permissive licence) · get_code("0688ceac63ea3074") |
| tokenize | Not yet run | google-research/FLAN/flan/preprocessors.py code served (permissive licence) · get_code("a0c365753489d524") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population. 1 * Equal contribution 1 The source code for the experiments is available at https://github.com/xfactlab/emnlp2023-LLM-Disagreement. Read the following and determine if the hypothesis can be inferred from the premise. Premise: She smiled back. Hypothesis: She was so happy she couldn't stop smiling.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.13788")
get_code_for_paper("2305.13788")
have("2305.13788")
Connect an agent — have() is free.