SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2408.17221 · ICLR · 2025

Geometry of Lightning Self-Attention: Identifiability and Dimension

Giovanni Marchetti, Nathan Henry, Kathlén Kohn

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We consider function spaces defined by self-attention networks without normalization, and theoretically analyze their geometry. Since these networks are polynomial, we rely on tools from algebraic geometry. In particular, we study the identifiability of deep attention by providing a description of the generic fibers of the parametrization for an arbitrary number of layers and, as a consequence, compute the dimension of the function space. Additionally, for a single-layer model, we characterize the singular and boundary points. Finally, we formulate a conjectural extension of our results to normalized self-attention networks, prove it for a single layer, and numerically verify it in the deep case. Figure 1: A slice of the space of lightning self-attention mechanisms. *Equal contribution.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2408.17221")
get_code_for_paper("2408.17221")
have("2408.17221")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2408.17221.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2408.17221.svg)](https://syntology.ai/paper/2408.17221)

Connect an agent — have() is free.