Giovanni Marchetti, Nathan Henry, Kathlén Kohn
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We consider function spaces defined by self-attention networks without normalization, and theoretically analyze their geometry. Since these networks are polynomial, we rely on tools from algebraic geometry. In particular, we study the identifiability of deep attention by providing a description of the generic fibers of the parametrization for an arbitrary number of layers and, as a consequence, compute the dimension of the function space. Additionally, for a single-layer model, we characterize the singular and boundary points. Finally, we formulate a conjectural extension of our results to normalized self-attention networks, prove it for a single layer, and numerically verify it in the deep case. Figure 1: A slice of the space of lightning self-attention mechanisms. *Equal contribution.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2408.17221")
get_code_for_paper("2408.17221")
have("2408.17221")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2408.17221.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2408.17221)
Connect an agent — have() is free.