We lifted 17 functions out of this paper's own repositories and ran 15 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| dhananjayashok/littlehumandata | canonical | 15 of 17 |
| Function | Status | Where it lives |
|---|---|---|
| compute_inclusion | Ran | dhananjayashok/littlehumandata/compute_qa_metrics.py code served (permissive licence) · get_code("100237b0db550ff3") |
| compute_mode_ngrams | Ran | dhananjayashok/littlehumandata/analysis.py code served (permissive licence) · get_code("dca52e1f7f4a7dbc") |
| create_conversation | Ran | dhananjayashok/littlehumandata/llm-utils/data_script.py code served (permissive licence) · get_code("29bacdfb84b03880") |
| do_evaluation | Ran | dhananjayashok/littlehumandata/llm-utils/common_utils.py code served (permissive licence) · get_code("5c01635c26b2b6f5") |
| get_df | Ran | dhananjayashok/littlehumandata/llm-utils/predict.py code served (permissive licence) · get_code("c4bde9e27f99624d") |
| get_label_list | Ran | dhananjayashok/littlehumandata/llm-utils/classification.py code served (permissive licence) · get_code("adb7541f2edf078e") |
| get_metric_report_str | Ran | dhananjayashok/littlehumandata/llm-utils/causal_lm_conditional_generation.py code served (permissive licence) · get_code("29596155303849f9") |
| get_metric_report_str | Ran | dhananjayashok/littlehumandata/llm-utils/classification.py code served (permissive licence) · get_code("3a484a15b2187487") |
| get_metric_report_str | Ran | dhananjayashok/littlehumandata/llm-utils/common_utils.py code served (permissive licence) · get_code("dec1cca2d250dd20") |
| get_n_gram | Ran | dhananjayashok/littlehumandata/analysis.py code served (permissive licence) · get_code("18bd2e3ef915c4a4") |
| guarded_lower | Ran | dhananjayashok/littlehumandata/add_info_predfile.py code served (permissive licence) · get_code("2b29adfbbb6ee76b") |
| guarded_split | Ran | dhananjayashok/littlehumandata/analysis.py code served (permissive licence) · get_code("cb42513ddf067d63") |
| remove_urls | Ran | dhananjayashok/littlehumandata/prompt_generation/prompt_models.py code served (permissive licence) · get_code("b726f185fd16d31d") |
| tok_len | Ran | dhananjayashok/littlehumandata/prompt_generation/prompt_models.py code served (permissive licence) · get_code("019975ab5eef2a08") |
| truncate_context | Ran | dhananjayashok/littlehumandata/prompt_generation/prompt_models.py code served (permissive licence) · get_code("859b0c517f44d1ae") |
| compute_exact_match | Not yet run | dhananjayashok/littlehumandata/compute_qa_metrics.py code served (permissive licence) · get_code("d046034742d3fdba") |
| get_model | Not yet run | dhananjayashok/littlehumandata/llm-utils/predict.py code served (permissive licence) · get_code("795fca5aa8f88621") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Faced with an expensive human annotation process, creators of NLP systems increasingly turn to synthetic data generation. While this method shows promise, the extent to which synthetic data can replace human annotation is poorly understood. We investigate the use of synthetic data in Fact Verification (FV) and Question Answering (QA) by studying the effects of incrementally replacing human generated data with synthetic points on eight diverse datasets. Strikingly, replacing up to 90% of the training data only marginally decreases performance, but replacing the final 10% leads to severe declines. We find that models trained on purely synthetic data can be reliably improved by including as few as 125 human generated data points. We show that matching the performance gain of just a little additional human data (only 200 points) requires an order of magnitude more synthetic data and estimate price ratios at which human annotation would be a more cost-effective solution. Our results suggest that even when human annotation at scale is infeasible, there is great value to having a small proportion of the dataset being human generated.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.13098")
get_code_for_paper("2410.13098")
have("2410.13098")
Connect an agent — have() is free.