Austin Wang, Aman Patel, Arpita Singhal, Anusri Pampari, Maya Kasowski, Anshul Kundaje, Nnnelementnnn Nnnenlmteennn, Peak Sequences
We lifted 16 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| kundajelab/DART-Eval | canonical | 1 of 1 |
| kundajelab/dart-eval | canonical | 0 of 15 |
| Function | Status | Where it lives |
|---|---|---|
| make_region_df | Ran | kundajelab/DART-Eval/src/dnalm_bench/task_1_paired_control/dataset_generators/encode_ccre.py pointer only (licence: NONE) · get_code("853d8aee53d39296") |
| dinucleotide_shuffle | Not yet run | kundajelab/dart-eval/src/dnalm_bench/utils.py pointer only (licence: NONE) · get_code("44434c209290ce5b") |
| evaluate_finetuned_classifier | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_1_paired_control/finetune.py pointer only (licence: NONE) · get_code("f8b8d077d64721d3") |
| generate_latex_table_per_cell_type | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_1_paired_control/generate_latex.py pointer only (licence: NONE) · get_code("b59c3fbf640d6503") |
| get_accuracies | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_likelihoods.py pointer only (licence: NONE) · get_code("8d44605ff38fcc40") |
| get_confidence_interval | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_1_paired_control/generate_latex.py pointer only (licence: NONE) · get_code("3f5f87a9d18060a3") |
| get_distances | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_embeddings.py pointer only (licence: NONE) · get_code("ba9e7c54f2991e6a") |
| get_likelihoods | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_likelihoods.py pointer only (licence: NONE) · get_code("129082c04db6596b") |
| get_pvals | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_likelihoods.py pointer only (licence: NONE) · get_code("8dcd29ed5e28d719") |
| load_embeddings | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_embeddings.py pointer only (licence: NONE) · get_code("b05bd25300b743b3") |
| load_embeddings_and_labels | Not yet run | kundajelab/dart-eval/src/dnalm_bench/embedding_clustering.py pointer only (licence: NONE) · get_code("ebb6d12a7eaca88b") |
| load_embeddings_and_labels_subset | Not yet run | kundajelab/dart-eval/src/dnalm_bench/embedding_clustering.py pointer only (licence: NONE) · get_code("11d6f70d2317495f") |
| one_hot_encode | Not yet run | kundajelab/dart-eval/src/dnalm_bench/utils.py pointer only (licence: NONE) · get_code("a32416b3ff7b6896") |
| onehot_to_chars | Not yet run | kundajelab/dart-eval/src/dnalm_bench/utils.py pointer only (licence: NONE) · get_code("75f2134f65415576") |
| relate_embeddings_to_motifs | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_2_5_single/experiments/task_2_transcription_factor_binding/footprint_eval_embeddings.py pointer only (licence: NONE) · get_code("ae9202aca7e7c370") |
| underline_max_values | Not yet run | kundajelab/dart-eval/src/dnalm_bench/task_1_paired_control/generate_latex.py pointer only (licence: NONE) · get_code("51187133335595b3") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advances in self-supervised models for natural language, vision, and protein sequences have catalyzed the development of genomic DNA language models (DNALMs). These models aim to learn generalizable representations of diverse DNA elements, potentially enabling various downstream genomic prediction, interpretation and design tasks. However, existing benchmarks do not adequately assess the capabilities of DNALMs on an important class of non-coding DNA elements critical for regulating gene activity. Here, we introduce DART-Eval, a suite of representative benchmarks focused on regulatory DNA to evaluate performance of DNALMs across zero-shot, probed, and fine-tuned settings against contemporary ab initio models as baselines. DART-Eval addresses biologically relevant tasks including sequence motif discovery, cell-type specific regulatory activity prediction, and counterfactual prediction of regulatory genetic variants. Our systematic evaluations reveal that current annotation-agnostic DNALMs exhibit inconsistent performance and do not offer compelling gains over alternative baseline models for most tasks, despite requiring significantly more computational resources. We discuss potentially promising modeling, data curation, and evaluation strategies for the next generation of DNALMs. Our benchmark datasets and evaluation framework are available at https://github.com/kundajelab/DART-Eval 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2412.05430")
get_code_for_paper("2412.05430")
have("2412.05430")
Connect an agent — have() is free.