SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2204.13353 · ACL Findings · 2022

Attention Mechanism with Energy-Friendly Operations

Boxing Chen, Derek Wong, Lidia Chao, Dayiheng Liu, Baosong Yang, Rong Xiao, Yu Wan, Haibo Zhang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Attention mechanism has become the dominant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multiplications. In this paper, we rethink variants of attention mechanism from the energy consumption aspects. After reaching the conclusion that the energy costs of several energyfriendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. Code is available at: https://github.com/NLP2CT/E-Att.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2204.13353")
get_code_for_paper("2204.13353")
have("2204.13353")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2204.13353.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2204.13353.svg)](https://syntology.ai/paper/2204.13353)

Connect an agent — have() is free.