Ievgen Redko, Giuseppe Paolo, Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Themis Palpanas
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used realworld multivariate time series datasets. In particular, SAMformer surpasses current state-ofthe-art methods and is on par with the biggest foundation model MOIRAI while having significantly fewer parameters. The code is available at https://github.com/romilbert/samformer.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.10198")
get_code_for_paper("2402.10198")
have("2402.10198")
Connect an agent — have() is free.