Nihar Shah, Isabelle Guyon, Zhen Xu, Alexander Goldberg, Ihsan Ullah, Thanh Gia, Hieu Khuong, Kent Rachmat
We lifted 6 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ihsaan-ullah/neurips-checklist-assistant | canonical | 0 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| clean | Not yet run | ihsaan-ullah/neurips-checklist-assistant/CheckPaperFormat/check_paper_format.py code served (permissive licence) · get_code("231eeb3c0401afe1") |
| clean_paper | Not yet run | ihsaan-ullah/neurips-checklist-assistant/CheckPaperFormat/check_paper_format.py code served (permissive licence) · get_code("61725cb61c0b533a") |
| clean_title | Not yet run | ihsaan-ullah/neurips-checklist-assistant/CheckPaperFormat/check_paper_format.py code served (permissive licence) · get_code("6683daa7f5a711d1") |
| extract_checklist | Not yet run | ihsaan-ullah/neurips-checklist-assistant/AssistantAnalysis/annotate_responses_public.py code served (permissive licence) · get_code("4de4b37bc6a31ca7") |
| extract_paper_parse | Not yet run | ihsaan-ullah/neurips-checklist-assistant/AssistantAnalysis/annotate_responses_public.py code served (permissive licence) · get_code("5d35a2c7da58137c") |
| q_to_file | Not yet run | ihsaan-ullah/neurips-checklist-assistant/AssistantAnalysis/annotate_responses_public.py code served (permissive licence) · get_code("d4d0d87078a34d26") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards. We conduct an experiment at the 2024 Neural Information Processing Systems (NeurIPS) conference, where 234 papers were voluntarily submitted to an "LLMbased Checklist Assistant." This assistant validates whether papers adhere to the author checklist used by NeurIPS, which includes questions to ensure compliance with research and manuscript preparation standards. Evaluation of the assistant by NeurIPS paper authors suggests that the LLM-based assistant was generally helpful in verifying checklist completion. In post-usage surveys, over 70% of authors found the assistant useful, and 70% indicate that they would revise their papers or checklist responses based on its feedback. While causal attribution to the assistant is not definitive, qualitative evidence suggests that the LLM contributed to improving some submissions. Survey responses and analysis of resubmissions indicate that authors made substantive revisions to their submissions in response to specific feedback from the LLM. The experiment also highlights common issues with LLMs-inaccuracy (20/52) and excessive strictness (14/52) were the most frequent issues flagged by authors. We also conduct experiments to understand potential gaming of the system, which reveal that the assistant could be manipulated to enhance scores through fabricated justifications, highlighting potential vulnerabilities of automated review tools.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.03417")
get_code_for_paper("2411.03417")
have("2411.03417")
Connect an agent — have() is free.