arXiv research

Study geometric properties of loss functions to understand neural network performance.

Focal loss reduces model curvature for better calibration.

Study on scalar curvature minimizability loss and saddle point solutions.

New framework to understand and exploit curvature in deep learning loss landscapes.

Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.

New neural network solves Nirenberg problem for curvature on sphere.

New approach ties loss curvature to model performance in deep learning.

New method approximates curvature from symmetries in deep networks.

Deep Curvature Suite offers a PyTorch package for neural network curvature analysis.

We introduce a scalable measure of curvature for analyzing training dynamics of large language models.

Proposes a new network to improve nuclei segmentation in histopathology images.

Paper introduces a new $G^\star$ regret measure for online convex optimization with smooth losses.

New algorithm exploits curvature of feasible sets for fast online convex optimization.

Graph networks struggle with multi-task learning due to varying property loss surface curvatures.

This paper establishes minimax rates for online regression with arbitrary classes of functions and general losses. We show that below a certain threshold for the complexity of the function class, the minimax rates depend on both the curvature of the loss function and the sequential complexities of the class. Above this…

New insights into SGD and generalization via shift-curvature and bias-curvature mechanisms.

Improved online convex optimization with delayed feedback using curvature.

The speed at which one can minimize an expected loss using stochastic methods depends on two properties: the curvature of the loss and the variance of the gradients. While most previous works focus on one or the other of these properties, we explore how their interaction affects optimization speed. Further, as the ulti…

The total curvature of complex hypersurfaces in $\bC^{n+1}$ and its variation in families appear to depend not only on singularities but also on the behaviour in the neighbourhood of infinity. We find the asymptotic loss of total curvature towards infinity and we express the total curvature and the Gauss-Bonnet defect …

We explore geometric aspects of bubble convergence for harmonic maps. More precisely, we show that the formation of bubbles is characterised by the local excess of curvature on the target manifold. We give a universal estimate for curvature concentration masses at each bubble point and show that there is no curvature l…

Wider neural networks have predominantly positive curvature, aiding optimization.

This work justifies neural collapse under MSE loss and analyzes the optimization landscape.

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce mi…

EpiMer merges models by solving Fréchet mean on a Riemannian manifold.

Enhances deep learning robustness to noise without sacrificing clean data accuracy.

The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed 4 striking local properties of neural loss landscapes on classification tasks: …

State-of-the-art classifiers have been shown to be largely vulnerable to adversarial perturbations. One of the most effective strategies to improve robustness is adversarial training. In this paper, we investigate the effect of adversarial training on the geometry of the classification landscape and decision boundaries…

This paper improves HNNs by learning optimal curvature for better generalization.

CurvSSL improves SSL by aligning local manifold curvature.

The classical asymptotic theory for parametric $M$-estimators guarantees that, in the limit of infinite sample size, the excess risk has a chi-square type distribution, even in the misspecified case. We demonstrate how self-concordance of the loss allows to characterize the critical sample size sufficient to guarantee …

Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.

This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side). The heart of the…

DEO uses gradient information to escape saddle points in neural networks.

New algorithms for online learning without boundedness or Lipschitz loss assumptions.

New algorithm tackles heterogeneous curvature in online convex optimization.

In this paper, we establish the non-positivity of the second eigenvalue of the Schrödinger operator $-\textrm{div}\big( P_r \nabla\cdot\big) - W_r^2$ on a closed hypersurface $Σ^n$ of $\mathbb{R}^{n+1}$, where $W_r$ is a power of the $(r+1)$-th mean curvature of $Σ^n$. In the case that this eigenvalue is null we have a…

There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of the loss landscape that incorporates many of them. High dimensionality plays a key role in our model. Our core idea is to model the loss la…

Automatic segmentation of auditory ossicles from CT images using Ricci curvature.

SGD without replacement decouples into curvature-following and flatness-regularizing steps.

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly und…

This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the outliers in the data. The analysis of the role of a particular non-convex loss for prediction accur…

Burq-Gérard-Tzvetkov and Hu established $L^p$ estimates ($2\le p\le \infty$) for the restriction of eigenfunctions to submanifolds. The estimates are sharp, except for the log loss at the endpoint $L^2$ estimates for submanifolds of codimension 2. It has long been believed that the log loss at the endpoint can be remov…

In [LW], we construct examples of two-dimensional Hamiltonian stationary self-shrinkers and self-expanders for Lagrangian mean curvature flows, which are asymptotic to the union of two Schoen-Wolfson cones. These self-shrinkers and self-expanders can be glued together to yield solutions of the Brakke flow - a weak form…

This paper examines how different loss functions affect neural network features and performance.

A new method improves adversarial robustness and interpretability with reduced training time.

Gradient-based methods find saddle points, not critical points, in neural networks.

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spa…

Efficient regularization mitigates catastrophic overfitting in single-step adversarial training.