Xuanjing Huang, Xipeng Qiu, Yuxin Wang, Chu-Tak Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformers have made progress in miscellaneous tasks, but suffer from quadratic computational and memory complexities. Recent works propose sparse Transformers with attention on sparse graphs to reduce complexity and remain strong performance. While effective, the crucial parts of how dense a graph needs to be to perform well are not fully explored. In this paper, we propose Normalized Information Payload (NIP), a graph scoring function measuring information transfer on graph, which provides an analysis tool for trade-offs between performance and complexity. Guided by this theoretical analysis, we present Hypercube Transformer, a sparse Transformer that models token interactions in a hypercube and shows comparable or even better results with vanilla Transformer while yielding O(N log N ) complexity with sequence length N . Experiments on tasks requiring various sequence lengths lay validation for our graph function well 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2205.14014")
get_code_for_paper("2205.14014")
have("2205.14014")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2205.14014.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2205.14014)
Connect an agent — have() is free.