SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2011.02886 · NeurIPS · 2020

Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization

Davide Bacciu, Antonio Carta, Alessandro Sperduti

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Training RNNs to learn long-term dependencies is difficult due to vanishing gradients. We explore an alternative solution based on explicit memorization using linear autoencoders for sequences, which allows to maximize the short-term memory and that can be solved with a closed-form solution without backpropagation. We introduce an initialization schema that pretrains the weights of a recurrent neural network to approximate the linear autoencoder of the input sequences and we show how such pretraining can better support solving hard classification tasks with long sequences. We test our approach on sequential and permuted MNIST. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2011.02886")
get_code_for_paper("2011.02886")
have("2011.02886")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2011.02886.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2011.02886.svg)](https://syntology.ai/paper/2011.02886)

Connect an agent — have() is free.