Nicolas Flammarion, Aditya Varre, Mark Rofin
We lifted 10 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| tml-epfl/softmax | canonical | 0 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| evaluate | Not yet run | tml-epfl/softmax/exp_classification.py pointer only (licence: NONE) · get_code("de1a6b54bb6d7eb2") |
| evaluate | Not yet run | tml-epfl/softmax/exp_induction.py pointer only (licence: NONE) · get_code("195ef9d2094e5fa7") |
| get_metrics_for_batch | Not yet run | tml-epfl/softmax/exp_classification.py pointer only (licence: NONE) · get_code("dbc815742c417081") |
| get_metrics_for_batch | Not yet run | tml-epfl/softmax/exp_induction.py pointer only (licence: NONE) · get_code("179263c917ddf20b") |
| get_metrics_for_dataset | Not yet run | tml-epfl/softmax/exp_induction.py pointer only (licence: NONE) · get_code("c50658a8f12d5f40") |
| linear_attention | Not yet run | tml-epfl/softmax/src/attention.py pointer only (licence: NONE) · get_code("a91bbe5c96744514") |
| print_induction_sequence | Not yet run | tml-epfl/softmax/src/datasets.py pointer only (licence: NONE) · get_code("dea3bf88d257b9b2") |
| sigmoid_attention | Not yet run | tml-epfl/softmax/src/attention.py pointer only (licence: NONE) · get_code("05229e47182eaaf7") |
| softmax_attention | Not yet run | tml-epfl/softmax/src/attention.py pointer only (licence: NONE) · get_code("cac10aee0417a8c9") |
| train | Not yet run | tml-epfl/softmax/exp_classification.py pointer only (licence: NONE) · get_code("0dbeea554881e4bd") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Understanding the intricate non-convex training dynamics of softmax-based models is crucial for explaining the empirical success of transformers. In this article, we analyze the gradient flow dynamics of the value-softmax model, defined as L(Vσ(a)), where V and a are a learnable value matrix and attention vector, respectively. As the matrix times softmax vector parameterization constitutes the core building block of self-attention, our analysis provides direct insight into transformer's training dynamics. We reveal that gradient flow on this structure inherently drives the optimization toward solutions characterized by low-entropy outputs. We demonstrate the universality of this polarizing effect across various objectives, including logistic and square loss. Furthermore, we discuss the practical implications of these theoretical results, offering a formal mechanism for empirical phenomena such as attention sinks and massive activations.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2603.06248")
get_code_for_paper("2603.06248")
have("2603.06248")
Connect an agent — have() is free.