Mean Activation Curvature for Scalable Second-Order Optimization in Deep Networks

초록

Second-order methods can accelerate deep neural network training, but their adoption is limited by the cost and instability of estimating and inverting curvature matrices. We revisit Kronecker-factored Fisher approximations via an empirical structural analysis of activa- tion and gradient statistics in modern architectures. Across a range of models, we find that activation statistics capture most of the effective curvature directions, while gradient statis- tics mainly act as a global or diagonal rescaling. Based on this observation, we propose mean activation curvature, a scalable curvature surrogate that yields two optimizers, MAC and SMAC, offering different trade-offs between expressiveness and efficiency. We further extend the construction to self-attention layers with a structured approximation that retains the role of attention scores, and provide a convergence analysis under standard assumptions. Experiments on vision and language models show that MAC/SMAC matches or improves the accuracy of existing second-order baselines while reducing training time and memory usage. Code: github.com/hseung88/mac.

키워드