arXiv research

Regularizers change the geometric properties of loss functions in neural networks.

PINNs solve differential geometry problems in complex shapes.

Geodesics connect model modes in neural network loss landscapes.

The pursuit of explaining and improving generalization in deep learning has elicited efforts both in regularization techniques as well as visualization techniques of the loss surface geometry. The latter is related to the intuition prevalent in the community that flatter local optima leads to lower generalization error…

Paper proposes fitting loss functions to data using source functions from information geometry.

Researchers improve visualization of neural network loss landscapes.

A new learning method using hyperbolic geometry for class labels.

This study explores star-shaped regularizers learned from critic-based losses.

This paper finds ReLU restores symmetry in SCL under class imbalances.

New method proves asymptotic normality for matrix sensing problems.

Unified framework connects different neural network models.

By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous sy…

Monotonic Linear Interpolation property in neural networks persists despite non-convexity.

Monge SAM improves deep learning by making sharpness-aware minimization invariant to reparametrizations.

New framework for data-driven hyperparameter tuning with structured loss.

Toolbox for stochastic Euler equations using Ebin-Marsden theory.

EpiMer merges models by solving Fréchet mean on a Riemannian manifold.

Looped transformers outperform standard transformers in complex reasoning tasks due to a specific loss landscape geometry.

Deep learning dynamics and NTK evolution studied through diverse measures.

New study shows deep networks generalize well due to loss surface geometry.

Ricci flow stability on manifolds with bounded geometry ensures convergence to hyperbolic metrics.

New neural network solves Nirenberg problem for curvature on sphere.

A simple graphical model for correlated defaults is proposed, with explicit formulas for the loss distribution. Algebraic geometry techniques are employed to show that this model is well posed for default dependence: it represents any given marginal distribution for single firms and pairwise correlation matrix. These t…

Study compares geometric approaches for shape and deformation statistics.

This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.

Develops a new theory of loss functions for statistical machine learning.

Unified framework for non-negative matrices and tensors using Wasserstein loss.

Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statistically accurate models. In this paper we study how the choice of optimization algorithm influences the robustness of the resulting classifie…

This work characterizes the fundamental limit of network pruning using statistical dimension and convex geometry.

SLERP interpolation optimizes dynamic weight rebalancing in AMMs.

We develop efficient algorithms to train $\ell_1$-regularized linear classifiers with large dimensionality $d$ of the feature space, number of classes $k$, and sample size $n$. Our focus is on a special class of losses that includes, in particular, the multiclass hinge and logistic losses. Our approach combines several…

Spectral measurements reveal hidden representation geometry in language model training.

Improves model generalization by minimizing loss sharpness.

This work tackles Bayesian neural networks by addressing loss landscape symmetries.

Paper finds exact Hessian sharpness in deep matrix factorization.

Deep neural networks are workhorse models in machine learning with multiple layers of non-linear functions composed in series. Their loss function is highly non-convex, yet empirically even gradient descent minimisation is sufficient to arrive at accurate and predictive models. It is hitherto unknown why are deep neura…

Adaptive optimization methods bias neural network trajectories towards regions of lower local geometry.

This paper presents a novel CNN-based approach for synthesizing high-resolution LiDAR point cloud data. Our approach generates semantically and perceptually realistic results with guidance from specialized loss-functions. First, we utilize a modified per-point loss that addresses missing LiDAR point measurements. Secon…

In a graph convolutional network, we assume that the graph $G$ is generated wrt some observation noise. During learning, we make small random perturbations $ΔG$ of the graph and try to improve generalization. Based on quantum information geometry, $ΔG$ can be characterized by the eigendecomposition of the graph Laplaci…

Optimization geometry affects deep learning performance.

The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed 4 striking local properties of neural loss landscapes on classification tasks: …

OrthoGrad improves neural calibration by constraining gradient updates orthogonally.

Study geometric properties of loss functions to understand neural network performance.

The paper analyzes how noise geometry influences the performance of SGD in machine learning.

A new framework for structured prediction on non-vectorial spaces.

Benguria and Loss have conjectured that, amongst all smooth closed curves of length $2π$ in the plane, the lowest possible eigenvalue of the operator $L=-Δ+κ^2$ was one. They observed that this value was achieved on a two-parameter family, $\mathcal{O}$, of geometrically distinct ovals containing the round circle and c…

The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, …

The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.