Hao Peng, Roy Schwartz, Dianqi Li, A Smith, ♣ Noah
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks. Evidence has shown that they are overparameterized; attention heads can be pruned without significant performance loss. In this work, we instead "reallocate" them-the model learns to activate different heads on different inputs. Drawing connections between multi-head attention and mixture of experts, we propose the mixture of attentive experts model (MAE). MAE is trained using a block coordinate descent algorithm that alternates between updating (1) the responsibilities of the experts and (2) their parameters. Experiments on machine translation and language modeling show that MAE outperforms strong baselines on both tasks. Particularly, on the WMT14 English to German translation dataset, MAE improves over "transformer-base" by 0.8 BLEU, with a comparable number of parameters. Our analysis shows that our model learns to specialize different experts to different inputs. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2005.06537")
get_code_for_paper("2005.06537")
have("2005.06537")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2005.06537.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2005.06537)
Connect an agent — have() is free.