爱可可AI前沿推介(4.6)

LG - 机器学习 CV - 计算机视觉 CL - 计算与语言 AS - 音频与语音 RO - 机器人

转自爱可可爱生活

摘要：基于李代数卷积网络的对称性自动发现、多模态多任务掩码自编码器、基于路径的语言建模扩展、基于人工智能的科学理解、面向高效准确训练的表达结构矩阵、让老电影恢复生气、从人类和动物行为中学习可复用机器人运动技能、面向声调语言的歌曲自动翻译、基于语言模型的无提示高效少次学习

1、[LG] Automatic Symmetry Discovery with Lie Algebra Convolutional Network

N Dehmamy, R Walters, Y Liu, D Wang, R Yu

[Northwestern University & University of California San Diego]

基于李代数卷积网络的对称性自动发现。现有的等变神经网络，需要事先了解对称群和连续群的离散化。本文提出用李代数(无限小生成器)代替李群来实现。所提出的模型，李代数卷积网络(L-conv)，可自动发现对称性，并且不需要对群进行离散化。L-conv可作为构建任何群等变前馈架构的基础。CNN和图卷积网络都通过适当的群用L-conv进行表达。L-conv和物理学之间具有直接联系：(1)群不变损失泛化了场论 (2)Euler-Lagrange方程衡量了鲁棒性，以及 (3)等变性导致了守恒定律和Noether流。这些联系为设计更普遍的等变网络并将其应用于物理科学中的重要问题开辟了新途径。

Existing equivariant neural networks require prior knowledge of the symmetry group and discretization for continuous groups. We propose to work with Lie algebras (infinitesimal generators) instead of Lie groups. Our model, the Lie algebra convolutional network (L-conv) can automatically discover symmetries and does not require discretization of the group. We show that L-conv can serve as a building block to construct any group equivariant feedforward architecture. Both CNNs and Graph Convolutional Networks can be expressed as L-conv with appropriate groups. We discover direct connections between L-conv and physics: (1) group invariant loss generalizes field theory (2) Euler-Lagrange equation measures the robustness, and (3) equivariance leads to conservation laws and Noether current. These connections open up new avenues for designing more general equivariant networks and applying them to important problems in physical sciences.

https://arxiv.org/abs/2109.07103

2、[CV] MultiMAE: Multi-modal Multi-task Masked Autoencoders

R Bachmann, D Mizrahi, A Atanov, A Zamir

[Swiss Federal Institute of Technology Lausanne (EPFL)]

MultiMAE：多模态多任务掩码自编码器。本文提出一种名为多模态多任务掩码自编码器(MultiMAE)的预训练策略。在两个关键方面与标准掩码自编码不同：I）可以选择接受RGB图像以外的其他模态的输入信息("多模态")，以及 II）训练目标相应地包括预测RGB图像以外的多个输出("多任务")。利用掩码(跨图像块和输入模态)使MultiMAE的训练具有可操作性，并确保跨模态预测编码确实被网络学习。这种预训练策略得到了一个灵活、简单、高效的框架，改善了对下游任务的迁移结果。当RGB图像以外的其他信息可用时，或者当RGB以外的信息不可用时，可以灵活地使用相同的预训练网络——在所有配置中，产生与基线竞争或明显更好的结果。为避免需要多种模态和任务的训练数据集，本文完全用伪标签来训练MultiMAE，使得该框架广泛适用于任何RGB数据集。实验在多个迁移任务(图像分类、语义分割、深度估计)和数据集(ImageNet、ADE20K、Taskonomy、Hypersim、NYUv2)上进行。实验结果显示，该模型在跨模态/任务预测编码和迁移方面的能力令人印象深刻。

We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE). It differs from standard Masked Autoencoding in two key aspects: I) it can optionally accept additional modalities of information in the input besides the RGB image (hence "multi-modal"), and II) its training objective accordingly includes predicting multiple outputs besides the RGB image (hence "multi-task"). We make use of masking (across image patches and input modalities) to make training MultiMAE tractable as well as to ensure cross-modality predictive coding is indeed learned by the network. We show this pre-training strategy leads to a flexible, simple, and efficient framework with improved transfer results to downstream tasks. In particular, the same exact pre-trained network can be flexibly used when additional information besides RGB images is available or when no information other than RGB is available - in all configurations yielding competitive to or significantly better results than the baselines. To avoid needing training datasets with multiple modalities and tasks, we train MultiMAE entirely using pseudo labeling, which makes the framework widely applicable to any RGB dataset. The experiments are performed on multiple transfer tasks (image classification, semantic segmentation, depth estimation) and datasets (ImageNet, ADE20K, Taskonomy, Hypersim, NYUv2). The results show an intriguingly impressive capability by the model in cross-modal/task predictive coding and transfer.

https://arxiv.org/abs/2204.01678

3、[CL] PaLM: Scaling Language Modeling with Pathways

A Chowdhery, S Narang, J Devlin...

[Google Research]

PaLM：基于路径的语言建模扩展。大型语言模型已被证明在各种自然语言任务中用少次学习取得了显著的性能，大大减少了使模型自适应特定应用所需的特定任务训练样本的数量。为进一步了解规模对少次学习的影响，本文训练了一个5400亿参数、稠密激活的Transformer语言模型，称为Pathways语言模型(PaLM)。用Pathways在6144个TPU v4芯片上训练了PaLM，一个新的机器学习系统，能在多个TPU Pods上进行高效训练。通过在数百个语言理解和生成基准上实现最先进的少次学习结果来证明扩展的持续收益。在其中一些任务上，PaLM 540B实现了突破性的性能，在一套多步骤推理任务上超过了经过微调的最先进技术，并在最近发布的BIG-bench基准上超过了人类的平均表现。大量的BIG-bench任务显示了模型规模的不连续改进，这意味着当我们扩展到最大的模型时，性能陡然提高。PaLM在多语言任务和源码生成方面也有很强的能力，在一系列基准测试中得到了证明。还对偏见和毒性进行了全面分析，研究了与模型规模有关的训练数据记忆程度。讨论了与大型语言模型有关的伦理考虑，并讨论了潜在的缓解策略。

Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model (PaLM). We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-ofthe-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned stateof-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies.

https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf

4、[LG] On scientific understanding with artificial intelligence

M Krenn, R Pollice, S Y Guo...

[Max Planck Institute for the Science of Light (MPL) & University of Toronto & Weill-Cornell Medical College]

基于人工智能的科学理解。想象一下，有一个神谕能正确预测每个粒子物理实验的结果、每个化学反应的产物、每个蛋白质的功能。这样一个神谕，将彻底改变我们所知道的科学和技术。然而，作为科学家，我们不会满足于神谕本身，我们想要更多。我们想了解神谕是如何设想这些预测的。这一壮举被称为科学理解，经常被认为是科学的基本目标。现在，计算机和人工智能不断增长的力量带来了一个终极问题。先进的人工系统如何为科学理解做出贡献或自主地实现它？我们相信，这不是一个单纯的技术问题，而是处于科学的核心。因此，在这里，我们着手回答我们所处的位置以及我们将去向何方。我们首先从科学哲学中寻求建议来理解科学理解。然后，我们回顾了当前的技术状况，包括从文献中以及通过收集科学家关于他们如何在计算机帮助下获得新的概念性理解的数十个事实。这些综合的见解帮助我们定义了机器人辅助科学理解的三个层面。机器人作为 I）计算的显微镜，II）灵感和终极的，尚未存在的来源 III）理解的代理。对于每个维度，我们都解释了超越现状的新途径，并释放出人工智能对科学中心目标的全部贡献。希望我们的观点能激励和集中研究，使机器人获得新的科学理解，并最终使我们更接近真正的人工科学家。

Imagine an oracle that correctly predicts the outcome of every particle physics experiment, the products of every chemical reaction, or the function of every protein. Such an oracle would revolutionize science and technology as we know them. However, as scientists, we would not be satisfied with the oracle itself. We want more. We want to comprehend how the oracle conceived these predictions. This feat, denoted as scientific understanding, has frequently been recognized as the essential aim of science. Now, the ever-growing power of computers and artificial intelligence poses one ultimate question: How can advanced artificial systems contribute to scientific understanding or achieve it autonomously? We are convinced that this is not a mere technical question but lies at the core of science. Therefore, here we set out to answer where we are and where we can go from here. We first seek advice from the philosophy of science to understand scientific understanding. Then we review the current state of the art, both from literature and by collecting dozens of anecdotes from scientists about how they acquired new conceptual understanding with the help of computers. Those combined insights help us to define three dimensions of android-assisted scientific understanding: The android as a I) computational microscope, II) resource of inspiration and the ultimate, not yet existent III) agent of understanding. For each dimension, we explain new avenues to push beyond the status quo and unleash the full power of artificial intelligence's contribution to the central aim of science. We hope our perspective inspires and focuses research towards androids that get new scientific understanding and ultimately bring us closer to true artificial scientists.

https://arxiv.org/abs/2204.01467

5、[LG] Monarch: Expressive Structured Matrices for Efficient and Accurate Training

T Dao, B Chen, N Sohoni, A Desai, M Poli, J Grogan, A Liu, A Rao...

[Stanford University & University at Buffalo & University of Michigan]

Monarch：面向高效准确训练的表达结构矩阵。大型神经网络在许多领域都很出色，但其训练和微调费用很高。减少计算/内存需求的一个流行方法，是用结构化矩阵(如稀疏、低秩、傅里叶变换)取代密集的权重矩阵。这些方法还没有被广泛采用：（1）由于不利的效率-质量权衡，在端到端训练中；（2）在稠密到稀疏微调中缺乏近似给定稠密权重矩阵的可行算法。为解决这些问题，本文提出了一类矩阵Monarch，具有硬件效率(为更好地利用硬件，它们被参数化为两个块对角矩阵的乘积)和表现力(可以代表许多常用的变换)。令人惊讶的是，用Monarch矩阵近似稠密权重矩阵的问题，尽管是非凸的，但有一个分析性最佳解。Monarch矩阵的这些特性开启了训练和微调稀疏和稠密模型的新途径。通过经验验证，Monarch在几个端到端的稀疏训练应用中可以实现有利的精度-效率权衡：在ImageNet分类和Wikitext-103语言建模中，ViT和GPT-2的训练速度提高了2倍，模型质量相当，在PDE求解和MRI重建任务中，误差降低了40%。在从稀疏到稠密的训练中，通过一种"反向稀疏化"的简单技术，Monarch矩阵作为一种有用的中间表示，将OpenWebText的GPT-2预训练速度提高了2倍而质量没有下降。同样的技术带来了23%的BERT预训练速度，甚至比创造MLPerf 1.1记录的Nvidia的非常优化的实现还要快。在稠密到稀疏的微调中，作为概念验证，Monarch近似算法将GLUE上的BERT微调速度提高了1.7倍，而且精度相当。

Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute/memory requirements is to replace dense weight matrices with structured ones (e.g., sparse, low-rank, Fourier transform). These methods have not seen widespread adoption (1) in end-to-end training due to unfavorable efficiency–quality tradeoffs, and (2) in denseto-sparse fine-tuning due to lack of tractable algorithms to approximate a given dense weight matrix. To address these issues, we propose a class of matrices (Monarch) that is hardware-efficient (they are parameterized as products of two block-diagonal matrices for better hardware utilization) and expressive (they can represent many commonly used transforms). Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution. These properties of Monarch matrices unlock new ways to train and fine-tune sparse and dense models. We empirically validate that Monarch can achieve favorable accuracy–efficiency tradeoffs in several end-to-end sparse training applications: speeding up ViT and GPT-2 training on ImageNet classification and Wikitext-103 language modeling by 2× with comparable model quality, and reducing the error on PDE solving and MRI reconstruction tasks by 40%. In sparse-to-dense training, with a simple technique called “reverse sparsification,” Monarch matrices serve as a useful intermediate representation to speed up GPT-2 pretraining on OpenWebText by 2× without quality drop. The same technique brings 23% faster BERT pretraining than even the very optimized implementation from Nvidia that set the MLPerf 1.1 record. In dense-to-sparse fine-tuning, as a proof-of-concept, our Monarch approximation algorithm speeds up BERT fine-tuning on GLUE by 1.7× with comparable accuracy.

https://arxiv.org/abs/2204.00595