We lifted 14 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ise-uiuc/xft | canonical | 11 of 14 |
| Function | Status | Where it lives |
|---|---|---|
| detect_codeblocks | Ran | ise-uiuc/xft/src/magicoder/clean_data.py code served (permissive licence) · get_code("fd880e76becf098a") |
| filter_same_seed_problem_solution | Ran | ise-uiuc/xft/src/magicoder/clean_data.py code served (permissive licence) · get_code("5a60645429ce510e") |
| get_docstring_summary | Ran | ise-uiuc/xft/experiments/extract_function_comment_pairs.py code served (permissive licence) · get_code("ebc8b39c9a484052") |
| get_language | Ran | ise-uiuc/xft/experiments/avg_multipl_e.py code served (permissive licence) · get_code("52a867386608a548") |
| map_humaneval_problem | Ran | ise-uiuc/xft/experiments/text2code.py code served (permissive licence) · get_code("917a729e9d2282f6") |
| postprocess | Ran | ise-uiuc/xft/experiments/ds_1000.py code served (permissive licence) · get_code("1173d57fe1b4d158") |
| preprocess | Ran | ise-uiuc/xft/experiments/train_comment_code_pairs.py code served (permissive licence) · get_code("f8e7806484bcc9a6") |
| preprocess_completion_prompt | Ran | ise-uiuc/xft/experiments/ds_1000.py code served (permissive licence) · get_code("956cd0aa1cec0821") |
| process_data | Ran | ise-uiuc/xft/experiments/train_comment_code_pairs.py code served (permissive licence) · get_code("61194aba1f41bd36") |
| remove_all_whitespaces | Ran | ise-uiuc/xft/src/magicoder/clean_data.py code served (permissive licence) · get_code("c31a9474a257aaab") |
| strip_c_style_comment_delimiters | Ran | ise-uiuc/xft/experiments/extract_function_comment_pairs.py code served (permissive licence) · get_code("29080cd7e5b82ee7") |
| extract_methods | Not yet run | ise-uiuc/xft/experiments/extract_function_comment_pairs.py code served (permissive licence) · get_code("d711ab82d48bbdd3") |
| get_dataset | Not yet run | ise-uiuc/xft/experiments/collect_seed_documents.py code served (permissive licence) · get_code("2157c279da567204") |
| map_mbpp_problem | Not yet run | ise-uiuc/xft/experiments/text2code.py code served (permissive licence) · get_code("f95fb439168f8d49") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2404.15247")
get_code_for_paper("2404.15247")
have("2404.15247")
Connect an agent — have() is free.