Song Han, Tianle Cai, Kai Li, Tri Dao, Guangxuan Xiao, James Liu, Jason D. Lee
We lifted 9 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| FasterDecoding/BitDelta | canonical | 6 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| find_corr_stddev | Ran | FasterDecoding/BitDelta/bitdelta/misc.py code served (permissive licence) · get_code("788def677abe4a95") |
| get_dataloader | Ran | FasterDecoding/BitDelta/bitdelta/data.py code served (permissive licence) · get_code("c39b5bb17824dd6e") |
| pack | Ran | FasterDecoding/BitDelta/bitdelta/binary_gemm_kernel.py code served (permissive licence) · get_code("3857e2268fd2be5c") |
| parse_device | Ran | FasterDecoding/BitDelta/bitdelta/utils.py code served (permissive licence) · get_code("9dcbdcf4fe815b37") |
| parse_dict | Ran | FasterDecoding/BitDelta/bitdelta/utils.py code served (permissive licence) · get_code("c85ef5aa3bf5041e") |
| unpack | Ran | FasterDecoding/BitDelta/bitdelta/binary_gemm_kernel.py code served (permissive licence) · get_code("7b1afaa65397beea") |
| dequantize_model | Not yet run | FasterDecoding/BitDelta/bitdelta/misc.py code served (permissive licence) · get_code("ca14e14c001835fe") |
| get_dataset | Not yet run | FasterDecoding/BitDelta/bitdelta/data.py code served (permissive licence) · get_code("3ab4b98edc5243cf") |
| get_model | Not yet run | FasterDecoding/BitDelta/bitdelta/utils.py code served (permissive licence) · get_code("f08bdafbdaf3bee1") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large Language Models (LLMs) are typically trained in two phases: pre-training on large internet-scale datasets, and fine-tuning for downstream tasks. Given the higher computational demand of pre-training, it is intuitive to assume that finetuning adds less new information to the model, and is thus more compressible. We explore this assumption by decomposing the weights of fine-tuned models into their pre-trained components and an additional delta. We introduce a simple postfine-tuning method, BitDelta, which successfully quantizes this delta down to 1 bit without compromising performance. This interesting finding not only highlights the potential redundancy of information added during fine-tuning, but also has significant implications for the multi-tenant serving and multi-tenant storage of fine-tuned models. By enabling the use of a single high-precision base model accompanied by multiple 1-bit deltas, BitDelta dramatically reduces GPU memory requirements by more than 10×, thus reducing per-user generation latency by more than 10× in multi-tenant settings. We validate BitDelta through experiments across Llama-2, Mistral and MPT model families, and on models up to 70B parameters, showcasing minimal performance degradation in all tested settings.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.10193")
get_code_for_paper("2402.10193")
have("2402.10193")
Connect an agent — have() is free.