Zhiru Zhang, Mohamed Abdelfattah, Sheng Li, Jordan Dotzel, Bahaa Kotb, Yuzong Chen, Sushma Prasad, Gang Wu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The increasing size of large language models (LLMs) traditionally requires low-precision integer formats to meet strict latency and power demands. Yet recently, alternative formats such as Normal Float (NF4) have increased model accuracy at the cost of increased chip area. In this work, we first conduct a large-scale analysis of LLM weights and activations across 30 networks and conclude that most distributions follow a Student's t-distribution. We then derive a new theoretically optimal format, Student Float (SF4), that improves over NF4 across modern LLMs, for example increasing the average accuracy on LLaMA2-7B by 0.76% across tasks. Using this format as a high-accuracy reference, we then propose augmenting E2M1 with two variants of supernormal support for higher model accuracy. Finally, we explore the quality and efficiency frontier across 11 datatypes by evaluating their model accuracy and hardware complexity. We discover a Pareto curve composed of INT4, E2M1, and E2M1 with supernormal support, which offers a continuous tradeoff between model accuracy and chip area. For example, E2M1 with supernormal support increases the accuracy of Phi-2 by up to 2.19% with 1.22% area overhead, enabling more LLM-based applications to be run at four bits. The supporting code is hosted at https://github.com/cornell-zhang/llm-datatypes.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.03103")
get_code_for_paper("2405.03103")
have("2405.03103")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2405.03103.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2405.03103)
Connect an agent — have() is free.