来自今天的爱可可AI前沿推介

[LG] Equivariant Architectures for Learning in Deep Weight Spaces

A Navon, A Shamsian, I Achituve, E Fetaya, G Chechik, H Maron
[Bar-Ilan University & NVIDIA]

深度权重空间学习的等变架构

要点:

  1. 提出一种用于在深度权重空间学习的新的网络架构，与MLP权重的自然置换对称性等变；
  2. 指出深度权重空间之间的仿射等变层空间的第一个特征；
  3. 对所提架构的表达能力进行了分析。

一句话总结:
提出一种用于深度权重空间学习的新的网络架构，与MLP的权重的自然置换对称性等变，表征了仿射等变层的空间，分析了其表达能力，并在各种学习任务中展示了优势。

摘要：
设计机器学习架构，以处理神经网络的原始权重矩阵形式，是一个新提出的研究方向。不幸的是，深度权重空间的独特对称结构使得这种设计非常具有挑战性。如果成功的话，这样的架构将能执行各种有趣的任务，从使预训练网络适应新域到编辑以函数(INRs或NeRFs)表示的对象。作为实现这一目标的第一步，本文提出一种用于在深度权重空间学习的新的网络架构。将预训练MLP的权重和偏置串联作为输入，用与MLP权重的自然置换对称等变的层组成来处理它。改变MLP中间层的神经元顺序并不影响它所代表的函数。本文为这些对称性提供了所有仿生等变层和不变层的完整特征，并展示了如何用三种基本操作来实现这些层：池化、广播和以适当方式应用于输入的全连接层。在各种学习任务中，证明了所提出架构的有效性及其对自然基线的优势。

Designing machine learning architectures for processing neural networks in their raw weight matrix form is a newly introduced research direction. Unfortunately, the unique symmetry structure of deep weight spaces makes this design very challenging. If successful, such architectures would be capable of performing a wide range of intriguing tasks, from adapting a pre-trained network to a new domain to editing objects represented as functions (INRs or NeRFs). As a first step towards this goal, we present here a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trained MLP and processes it using a composition of layers that are equivariant to the natural permutation symmetry of the MLP's weights: Changing the order of neurons in intermediate layers of the MLP does not affect the function it represents. We provide a full characterization of all affine equivariant and invariant layers for these symmetries and show how these layers can be implemented using three basic operations: pooling, broadcasting, and fully connected layers applied to the input in an appropriate manner. We demonstrate the effectiveness of our architecture and its advantages over natural baselines in a variety of learning tasks.

论文链接：https://arxiv.org/abs/2301.12780
图片
图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除