A new algorithm tackles bilevel optimization with multiple inner minima.
problem Challenges in bilevel optimization with multiple inner minima.
method Reformulated as constrained optimization, solved via primal-dual bilevel optimization (PDBO) algorithm.
result First non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima.
A new stochastic method tackles bi-level optimization problems in deep learning.
problem Bi-level optimization problems in deep learning, including hyperparameter optimization and meta learning.
method Turning a BLO problem into a stochastic optimization, using SGLD MCMC and a recurrent algorithm to compute MC-estimated hypergradient.
result Our method is more robust to suboptimal inner optimization and non-unique inner minima, leading to more reliable solutions.
Study reveals structure of local minima in GMMs, identifying key cluster centers.
problem Identifying optimal cluster centers in non-convex GMM landscapes.
method Analyzing the negative log-likelihood function of GMMs in the population limit.
result Local minima share a common structure that partially identifies true cluster centers.
We present a novel optimization method, named the Combined Optimization Method (COM), for the joint optimization of two or more cost functions. Unlike the conventional joint optimization schemes, which try to find minima in a weighted sum of cost functions, the COM explores search space for common minima shared by all …
We introduce the Genetic-Gated Networks (G2Ns), simple neural networks that combine a gate vector composed of binary genetic genes in the hidden layer(s) of networks. Our method can take both advantages of gradient-free optimization and gradient-based optimization methods, of which the former is effective for problems …
In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It bec…
SGD can jump from high rank minima to low rank minima in DLNs, but not back.
problem SGD's tendency to get stuck in high rank minima in DLNs.
method Analysis of the L2-regularized loss function of DLNs and the definition of absorbing sets. result SGD has a non-zero probability to jump from high rank minima to low rank minima but zero probability to jump back.
Study minima of geodesic lengths for specific curves on surfaces.
problem Finding the shortest geodesic paths on surfaces.
method Using curves related to dessins d'enfants and Grothendieck-Belyi surfaces.
result Minima of geodesic lengths are achieved on Riemann surfaces defined over number fields.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
This paper constructs quandles with abelian inner automorphism groups from graphs, proving their homogeneity.
problem Finding quandles with specific automorphism properties.
method Starting from simple graphs, the paper constructs quandles with abelian inner automorphism groups and proves their homogeneity.
result Homogeneous quandles with abelian inner automorphism groups are constructed from vertex-transitive graphs.
Ensemble learning is a method of combining multiple trained models to improve model accuracy. We propose the usage of such methods, specifically ensemble average, inside Convolutional Neural Network (CNN) architectures by replacing the single convolutional layers with Inner Average Ensembles (IEA) of multiple convoluti…
Study of Gaussian distributions using entropic Gromov-Wasserstein and inner product Gromov-Wasserstein.
problem Optimal transportation between Gaussian distributions with different dimensions.
method Entropic Gromov-Wasserstein and inner product Gromov-Wasserstein, with closed-form expressions and von Neumann's trace inequality.
result Closed-form expressions for the entropic IGW and its unbalanced variant between Gaussian distributions.
We propose a quantization based approach for fast approximate Maximum Inner Product Search (MIPS). Each database vector is quantized in multiple subspaces via a set of codebooks, learned directly by minimizing the inner product quantization error. Then, the inner product of a query to a database vector is approximated …
New insights into spurious local minima in k-means clustering.
problem Understanding and mitigating spurious local minima in k-means clustering.
method Investigating spurious local minima under a probabilistic generative model.
result Proven structures of spurious local minima for k-means clustering.
DualAdam improves generalization of Adam by integrating its update mechanisms.
problem Adam's tendency to converge to sharp minima leading to suboptimal generalization.
method DualAdam combines Adam and inverse Adam's update mechanisms to enhance generalization.
result DualAdam outperforms Adam and state-of-the-art variants in generalization performance.
mSAM improves generalization by making models flatter.
problem Over-parameterized models can have varying generalization performance.
method mSAM modifies loss function to favor flatter minima through adversarial perturbations.
result mSAM achieves flatter minima than SAM and SGD, improving generalization.
This work reveals symmetries in quantum circuits and develops a noise-aware optimization method.
problem Understanding and optimizing the cost landscape of parametrized quantum circuits.
method Analytical proof of symmetries and their resilience to noise, followed by the development of SYMH optimization method.
result Symmetries in PQCs lead to degeneracy in the cost landscape and can be exploited to improve optimization under noise.
The paper extends kernel ridge regression to product kernels and reveals new convergence behaviors.
problem Understanding kernel ridge regression in large dimensions with various kernels.
method Established a broad family of large dimensional kernels and derived convergence rates.
result Revealed new phenomena including minimax optimality, saturation effect, and multiple descent behavior.
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trai…
In this paper, we determine the automorphism group of the p-cones (p=2) in dimension greater than two. In particular, we show that the automorphism group of those p-cones are the positive scalar multiples of the generalized permutation matrices that fix the main axis of the cone. Next, we take a look at a pro…
Transformers with multiple layers learn to estimate bigram distributions, while single-layer models often get stuck in unigram local minima.
problem Understanding the sequential modeling capabilities of transformers using Markov chains.
method Introducing a new framework to analyze transformers via Markov chains, characterizing their loss landscapes.
result Single-layer transformers often get stuck in local minima representing the unigram distribution, while deeper models reliably converge to the ground-truth bigram.
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively…
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
New framework learns sufficient invariant features robustly across distribution shifts.
problem Learning robust models under distribution shifts between training and test datasets.
method Sufficient Invariant Learning (SIL) framework and Adaptive Sharpness-aware Group Distributionally Robust Optimization (ASGDRO) algorithm.
result Empirical evaluations confirm ASGDRO's robustness against distribution shifts.
Study reveals properties of local minima in ReLU networks.
problem Understanding the loss landscape of neural networks.
method Theoretical analysis of one-hidden-layer ReLU networks.
result All differentiable local minima are global within certain regions.
Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block …
Paper finds wide minima are better for generalization and proposes a new learning rate schedule.
problem The challenge of finding optimal learning rates for model training.
method The paper introduces a new hypothesis about the density of wide minima and designs an explore-exploit learning rate schedule.
result The explore-exploit learning rate schedule improves model performance and reduces training time.
Gradient descent in deep networks tends to find flat minima, which are nearly balanced.
problem Understanding the effect of gradient descent on the structure of minima in deep neural networks.
method Characterized flat minima in linear neural networks trained with a quadratic loss.
result Flat minima correspond to nearly balanced networks where the gain from input to intermediate representations is nearly constant.
Truncated SGD with heavy-tailed noise eliminates sharp local minima.
problem Avoiding sharp local minima in deep learning models.
method Truncated SGD with heavy-tailed gradient noise.
result Truncated SGD can eliminate sharp local minima entirely from its training trajectory.
Optimizers find approximate global minima in non-convex problems.
problem Understanding why local methods solve non-convex optimization problems.
method Formalizing the hypothesis that many local minima are approximately global minima.
result Most local minima of practical non-convex objectives are approximately global minima.
The study analyzes local minima in ReLU networks and finds low probability of bad local minima.
problem Understanding the existence and probability of local minima in ReLU networks.
method Theoretical analysis combined with linear programming and experiments on MNIST and CIFAR-10 datasets.
result No bad differentiable local minima found almost everywhere in weight space.
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatl…
A new multi-kernel RBFNN design improves performance and speed.
problem Improving the performance and speed of RBFNNs.
method Proposes a novel multi-kernel RBFNN where each base kernel has its own weight.
result Better performance gains including faster convergence, better local minima, and resilience against poor local minima.
GPU-accelerates multiuser detection for 5G URLLC systems.
problem Efficiently detecting payloads in OFDM frames with ultra-low latency.
method Implemented partially linear multiuser detection in RKHSs on a GPU-accelerated platform.
result Sub-millisecond latency detection in 5G URLLC systems.
Paper describes links of mixed polynomials with specific properties.
problem Understanding the links of mixed polynomials with nice Newton boundaries.
method Analyzes links constructed from sequences of links associated with compact 1-faces of the Newton boundary.
result Links of singularities of inner non-degenerate mixed polynomials can be described using a specific procedure.
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. Global minima found for multidimensional scaling with penalties.
problem Finding global minima in multidimensional scaling.
method Combining stress loss function with a quadratic penalty term to find minimizers.
result Trajectory of minimizers leads to global minima.
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss o…
Proposes NRS to find flat minima in deep neural networks.
problem Finding optimal solutions in deep neural networks with overparameterization.
method NRS leverages the concept of flat minima and uses Kullback-Leibler divergence to regularize the neighborhood region in weight space.
result NRS drives optimizers towards flat minima, improving generalization ability across various model architectures.
The notion of flat minima has played a key role in the generalization studies of deep learning models. However, existing definitions of the flatness are known to be sensitive to the rescaling of parameters. The issue suggests that the previous definitions of the flatness might not be a good measure of generalization, b…
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
GRAIN: Group Aggregation via Min-Norm Objective
problem Learning instability in large models
method Replacing mean aggregation with min-norm convex combination
result Improves performance and reduces variance
The paper proposes a thermodynamic potential to guide training of generative models, breaking ergodicity to improve functionality.
problem Improving generative model functionality while limiting access to underrepresented patterns.
method Constructing a thermodynamic potential that guides training, leading to multiple minima in the free energy.
result Training a generative model breaks ergodicity, preventing escape into the high-temperature phase.
VAE global minima can learn correct manifold dimensions, even with conditioning variables.
problem Understanding VAE behavior on manifolds and adapting to varying dimensions.
method Proving VAE global minima can learn correct manifold dimensions and extending to CVAEs.
result Proven that VAE global minima can learn correct manifold dimensions and adapted to CVAEs.
New insights into hidden minima in neural networks.
problem Identifying hidden minima in two-layer ReLU networks.
method Analyzing curves along which loss is minimized, focusing on eigenvalue contributions.
result Distinctive structural and symmetry properties of arcs emanating from hidden minima.
In "Width complexes for knots and 3-manifolds," Jennifer Schultens defines the width complex for a knot in order to understand the different positions a knot can occupy in the 3-sphere and the isotopies between these positions. She poses several questions about these width complexes; in particular, she asks whether the…
DGSAM improves domain generalization by minimizing individual sharpness.
problem Improving domain generalization models that perform well on unseen target domains.
method Shifts DG paradigm toward minimizing individual sharpness across source domains.
result DGSAM reduces performance variance across domains with less computational overhead.