Thomas Müller, Momchil Hardalov, Lluís Màrquez, Matéo Mahaut, Laura Aina, Paula Czarnowska
We lifted 16 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| amazon-science/factual-confidence-of-llms | canonical | 9 of 16 |
| Function | Status | Where it lives |
|---|---|---|
| check_relation | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sf_graph_cleanup.py code served (permissive licence) · get_code("64cac13c7712a758") |
| compute_on_device | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/evaluation/consistency_utils.py code served (permissive licence) · get_code("38bcad1abd42a39a") |
| extract_ids | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sf_graph_cleanup.py code served (permissive licence) · get_code("96ff3da37ed8fb76") |
| format_context_trex | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sentence_extractors.py code served (permissive licence) · get_code("f6be3e2860c3bb80") |
| get_lpaqa_paraphrase | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sentence_extractors.py code served (permissive licence) · get_code("5649462c5d6fa11b") |
| get_model | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/arch.py code served (permissive licence) · get_code("c6aa1425713fcf1e") |
| get_tokenizer | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/arch.py code served (permissive licence) · get_code("5bcc936c4c8aef88") |
| sequence_log_score | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/evaluation/logit_scorer_utils.py code served (permissive licence) · get_code("a4baff91a17627e1") |
| surrogate_logit_score | Ran | amazon-science/factual-confidence-of-llms/src/self_knowledge/evaluation/logit_scorer_utils.py code served (permissive licence) · get_code("4a8e5481004d91f3") |
| compute_all_scores | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/evaluation/consistency_utils.py code served (permissive licence) · get_code("94dd1ade2ee9847a") |
| get_lama | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sampling.py code served (permissive licence) · get_code("5161229b89b4fad7") |
| get_nli_scoring_fun | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/evaluation/consistency_utils.py code served (permissive licence) · get_code("0e0da0e698e5347a") |
| get_popQA | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sampling.py code served (permissive licence) · get_code("298e3e6ffc4b1483") |
| log_to_pandas_dataframe | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/slot_filling.py code served (permissive licence) · get_code("67ebd03e36b5d144") |
| lpaqa_paraphrases | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sentence_extractors.py code served (permissive licence) · get_code("33ab7dabeec5a3a2") |
| o_pop_matching | Not yet run | amazon-science/factual-confidence-of-llms/src/self_knowledge/data_gen/sf_graph_cleanup.py code served (permissive licence) · get_code("cfdd2155ad385a81") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large Language Models (LLMs) tend to be unreliable in the factuality of their answers. To address this problem, NLP researchers have proposed a range of techniques to estimate LLM's confidence over facts. However, due to the lack of a systematic comparison, it is not clear how the different methods compare to one another. To fill this gap, we present a survey and empirical comparison of estimators of factual confidence. We define an experimental framework allowing for fair comparison, covering both fact-verification and question answering. Our experiments across a series of LLMs indicate that trained hidden-state probes provide the most reliable confidence estimates, albeit at the expense of requiring access to weights and training data. We also conduct a deeper assessment of factual confidence by measuring the consistency of model behavior under meaningpreserving variations in the input. We find that the confidence of LLMs is often unstable across semantically equivalent inputs, suggesting that there is much room for improvement of the stability of models' parametric knowledge. Our code is available at https://github.com/amazonscience/factual-confidence-of-llms.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.13415")
get_code_for_paper("2406.13415")
have("2406.13415")
Connect an agent — have() is free.