Questions and Answers for Flat-Minima Optimization of Modern Neural Networks

Today's neural networks have complex training loss landscapes, where different global minima yield models with different generalization abilities. A well-known hypothesis explaining this phenomenon is that the minima of neural networks can be characterized by their flatness, i.e., how strongly the loss varies in the minimum's neighborhood. It implies that flatter minima tend to generalize better than sharp minima, if the test loss is shifted from the training loss due to random perturbations (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017). Recent work observes that the loss surfaces of several neural networks are locally asymmetric rather than simply flat or sharp (He et al., 2019). This means that, starting from a local minimum, there exist directions in which the loss increases abruptly along one side, while growing slowly along the opposite side. As a consequence, one should bias the solution towards the flat side of the valley to generalize better than the exact minimizer. Two methods for finding flat minima stand out: 1. Averaging methods (i.e., Stochastic Weight Averaging, SWA), and 2. Minimax methods (i.e., Sharpness Aware Minimization, SAM). However, despite similar motivations, there is still limited understanding into their properties. We will (a) develop comprehensive evaluations, visualisations, and benchmarks across computer vision, natural language processing, and graph learning tasks to understand these approaches; (b) derive new results in implicit differentiation to design a more principled approach to flat minimization: (c) investigate efficient numerical computation methods for making this approach scalable to modern neural networks.