Equivariant Architectures for Learning in Deep Weight Spaces

Gal Chechik, Aviv Navon, Idan Achituve, Haggai Maron, Ethan Fetaya, Aviv Shamsian

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 34 functions out of this paper's own repositories and ran 17 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• AvivNavon/DWSNets — marked official in Papers with Code; named in the paper
• jkalogero/scalegmn

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.12780")
get_code_for_paper("2301.12780")
have("2301.12780")

Connect an agent — have() is free.