Phase Transition from Random Features to Fully Trainable Networks

Xin He ,

Huan Lao ,

Xiliang Lu ,

Shijun Zhang

ML NNGTB NN

2026年10月22日

构建精确且计算上可行的近似空间，是科学计算中的一项基础性任务。部分可训练神经网络架起了固定随机特征与完全自适应表示之间的桥梁。本文定量刻画了近似误差界中的一种相变模式，并在训练误差中观察到类似行为：初始阶段性能迅速但平滑地提升，随后收益逐渐衰减；当初始化失配程度更大时，这一转变更为陡峭。在核函数谱衰减服从多项式律（衰减速率指数为 $\alpha > 1$）且满足杠杆兼容性（leverage compatibility）的条件下，我们为宽度为 $m$、可训练比例为 $P$ 的两层网络建立了平方 $L_2$ 近似误差界：$\mathcal{O}\big((\kappa_P \log(m/\delta)/m)^\alpha\big)$。该界在再生核希尔伯特空间（RKHS）单位球面上一致成立，且以至少 $1-\delta$ 的概率成立。其中，常数 $\kappa_0^*$ 和 $\kappa_1^*$ 分别刻画了固定特征组与可训练特征组相对于目标 RKHS 的采样失配程度，而 $\kappa_P = \big(P / \kappa_1^* + (1-P) / \kappa_0^*\big)^{-1}$。对于 $d$ 维球面上的带偏置 ReLU（biased-ReLU）特征，我们验证了上述假设成立，并得到 $\alpha = (d+2)/(d-1)$；当初始化尺度 $\tau \geq 1$ 时，有 $\kappa_0^* = 4\sqrt{2}\,\tau(d+1)/d$；而在采用岭杠杆采样（ridge leverage sampling）时，$\kappa_1^* = 1$。以 $d=5$、$\tau=10$ 为例，仅需约 $4.1\%$ 的神经元可训练，即可实现从 $P=0$ 到 $P=1$ 所能带来的近似误差界降幅的 $90\%$。一个结构化的三层网络示例进一步说明：少量第一层可训练神经元如何通过非线性复合方式，有效利用下游固定的网络模块。大量数值实验——涵盖多种设置，包括深度达 15 层的网络——均展现出训练误差中类似的相变现象。

Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $α>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((κ_P\log(m/δ)/m)^α)$ for two-layer networks of width $m$ and trainable proportion $P$, where $κ_P=(P/κ_1^*+(1-P)/κ_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-δ$. The constants $κ_0^*$ and $κ_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $α=(d+2)/(d-1)$ and $κ_0^*=4\sqrt{2}τ(d+1)/d$ for initialization scale $τ\ge1$, and $κ_1^*=1$ under ridge leverage sampling. For $d=5$ and $τ=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.

许愿