Xiaodong He, Yifan Wang, Junwei Bao, Youzheng Wu, Jing Zhao
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine-and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in selfattention and the uninformative tokens are replaced with one or several clusters as the coarsegranularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers significantly better trade-off between accuracy and FLOPs compared to prior methods 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2203.09055")
get_code_for_paper("2203.09055")
have("2203.09055")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2203.09055.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2203.09055)
Connect an agent — have() is free.