We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformers are one of the most successful architectures of modern neural networks. At their core there is the so-called attention mechanism, which recently interested the physics community as it can be written as the derivative of an energy function in certain cases: while it is possible to write the cross-attention layer as a modern Hopfield network, the same is not possible for the self-attention, which is used in the GPT architectures and other autoregressive models. In this work we show that it is possible to obtain the self-attention layer as the derivative of local energy terms, which resemble a pseudo-likelihood. We leverage the analogy with pseudo-likelihood to design a recurrent model that can be trained without backpropagation: the dynamics shows transient states that are strongly correlated with both train and test examples. Overall we present a novel framework to interpret self-attention as an attractor network, potentially paving the way for new theoretical approaches inspired from physics to understand transformers.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.16112")
get_code_for_paper("2409.16112")
have("2409.16112")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2409.16112.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2409.16112)
Connect an agent — have() is free.