Renjie Liao, Christos Thrampoulidis, Sadegh Mahdavi
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformers have become the go-to architecture for language and vision tasks, yet their theoretical properties, especially memorization capacity, remain elusive. This paper investigates the memorization abilities of multi-head attention mechanisms, examining how many example sequences they can memorize, as a function of the number of heads and sequence length. Motivated by experimental findings on vision transformers, we introduce novel assumptions about the linear independence of input data, distinct from the commonly used general-position assumption. Under these assumptions, we demonstrate that an attention layer with H heads, dimension d, and context size n < d, featuring Θ(Hd 2 ) parameters, can memorize Ω(Hn) examples. Our analysis sheds light on how different attention heads handle various example sequences, aided by the softmax operator's saturation property. We validate our findings through experiments on synthetic data.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.02010")
get_code_for_paper("2306.02010")
have("2306.02010")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2306.02010.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2306.02010)
Connect an agent — have() is free.