Survey of Loss Landscape Surfaces: Theory, Applications and Algorithms

Abstract

This paper presents a survey of loss landscape surface (LLS) research in machine learning optimization. A three-stage filtering methodology was applied to over 300 research papers to select mathematically rigorous studies, yielding 50 seminal papers with analysis of the top 10 key contributions. The survey provides chronological analysis spanning 1951-2024, showing evolution from 4 foundational papers (2007) to 21 active contributions (2018-2021). A taxonomical framework categorizes 32 stochastic gradient descent (SGD) algorithms from classical SGD (1951) to recent variants. Analysis reveals six research themes: visualization techniques, minima-generalization relationships, step size selection, SGD extensions, batch size effects, and adaptive learning rates. Research questions are categorized into resolved issues (flat minima correlation), ongoing debates (batch size selection), and emerging challenges (transformer landscapes). Results demonstrate that despite algorithmic proliferation, only limited fundamentally novel continuous LLS algorithms exist. The survey identifies theoretical-practical gaps and provides a standardized framework for key algorithms. Contributions include tables documenting research chronology and algorithm evolution, providing foundation for future high-dimensional optimization research.