Quoc Le, Orhan Firat, Kun Zhang, Adams Yu, Barret Zoph, Yu Wang, Yonghui Wu, Andrew Dai, Yanping Huang, Yanqi Zhou, Zhifeng Chen, Yuanzhong Xu, and 15 more
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero, one and few-shot performance across 29 NLP tasks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2112.06905")
get_code_for_paper("2112.06905")
have("2112.06905")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2112.06905.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2112.06905)
Connect an agent — have() is free.