arXiv research

Survey of SDR methods for high-dimensional regression and embedding.

Study geometric properties of loss functions to understand neural network performance.

Differential quantities, including normals, curvatures, principal directions, and associated matrices, play a fundamental role in geometric processing and physics-based modeling. Computing these differential quantities consistently on surface meshes is important and challenging, and some existing methods often produce …

New algorithms estimate Hessians using random directions for faster stochastic optimization.

A new dimension reduction method based on Gaussian finite mixtures is proposed as an extension to sliced inverse regression (SIR). The model-based SIR (MSIR) approach allows the main limitation of SIR to be overcome, i.e., failure in the presence of regression symmetric relationships, without the need to impose further…

Study eigenvalues of a nonlinear operator and apply to submanifolds with bounded mean curvature.

Large learning rates cause parameter instability, leading to better generalization.

Establishes log-concavity estimates for convex domains' first Dirichlet eigenfunctions.

A new optimisation method efficiently scales Hessian-vector products for neural networks.

New method accelerates neural network training by focusing on flat directions.

The study examines principal directions and curvatures of Lagrangian submanifolds.

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one o…

We analyze the Hessian spectra of large models up to 100B parameters.

Improper affine spheres have played an important role in the development of geometric methods for the study of the Hessian one equation. Here, we review most of the advances we have made in this direction during the last twenty years.

New method shows Hessian estimator from random samples converges to true Hessian on complex manifolds.

New method computes affine normal directions efficiently for sparse polynomials.

In this paper, we characterize and classify all surfaces endowed with canonical principal direction relative to a space-like and light-like, constant direction in Minkowski 3-spaces.

Given a vector field $X$ in a Riemannian manifold, a hypersurface is said to have a canonical principal direction relative to $X$ if the projection of $X$ onto the tangent space of the hypersurface gives a principal direction. We give different ways for building these hypersurfaces, as well as a number of useful charac…

Study extends Hausdorff dimension Hessian results to new hyperconvex representations.

Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.

In this paper, we introduce canonical principal direction (CPD) submanifolds with higher codimension in Euclidean spaces. We obtain the complete classification of surfaces endowed with CPD in the Euclidean 4-space.

Study on discrete surfaces with constant principal curvature for nanocarbon applications.

In the present paper we classify all surfaces in $\E^3$ with a canonical principal direction. Examples of these type of surfaces are constructed. We prove that the only minimal surface with a canonical principal direction in the Euclidean space ${\mathbb{E}}^3$ is the catenoid.

Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed on the same order of time as gradients. In this paper we exploit this property a…

Autoencoders are a deep learning model for representation learning. When trained to minimize the distance between the data and its reconstruction, linear autoencoders (LAEs) learn the subspace spanned by the top principal directions but cannot learn the principal directions themselves. In this paper, we prove that $L_2…

CWGD measures gradient diversity weighted by curvature, improving SGD convergence.

In this paper we characterize and classify surfaces in ${\mathbb{H}}^2\times{\mathbb{R}}$ which have a canonical principal direction. Here ${\mathbb{H}}^2$ denotes the hyperbolic plane. We study some geometric properties such as minimality and flatness. Some examples are given to complete the study.

We establish a classification of cubic minimal cones in case of the so-called radial eigencubics. Our principal result states that any radial eigencubic is either a member of the infinite family of eigencubics of Clifford type, or belongs to one of 18 exceptional families. We prove that at least 12 of the 18 families a…

Study Neumann problem for special Lagrangian type equations.

Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.

We consider surfaces in Euclidean space parametrized on an annular domain such that the first fundamental form and the principal curvatures are rotationally invariant, and the principal curvature directions only depend on the angle of rotation (but not the radius). Such surfaces generalize the Enneper surface. We show …

A new PCR method using SVD with sparse regularization.

Approximate Newton methods are a standard optimization tool which aim to maintain the benefits of Newton's method, such as a fast rate of convergence, whilst alleviating its drawbacks, such as computationally expensive calculation or estimation of the inverse Hessian. In this work we investigate approximate Newton meth…

Attention learns PCA on Gaussian data, proving its connection to principal component analysis.

Noise injection regularizes Hessian, improving neural network training and generalization.

A parallel optimization method for convex functions using Hessian sketching and debiasing.

The paper analyzes stability and convergence rates of entropic and Sinkhorn potentials.

We introduce a scalable measure of curvature for analyzing training dynamics of large language models.

We explore the geometrical interpretation of the PCA based clustering algorithm Principal Direction Divisive Partitioning (PDDP). We give several examples where this algorithm breaks down, and suggest a new method, gap partitioning, which takes into account natural gaps in the data between clusters. Geometric features …

A neural network models pressure-Hessian from local velocity gradients in turbulent flows.

We study submanifolds whose principal curvatures, counted with multiplicities, do not depend on the normal direction. Such submanifolds, which we briefly call CPC submanifolds, are always austere, hence minimal, and have constant principal curvatures. Well-known classes of examples include totally geodesic submanifolds…

In this paper we introduce a notion of parallel transport for principal bundles with connections over differentiable stacks. We show that principal bundles with connections over stacks can be recovered from their parallel transport thereby extending the results of Barrett, Caetano and Picken, and Schreiber and Waldof f…

Gradient descent forces neural network eigenvalues to a specific threshold.

A class of surfaces-graphs in a Riemannian 3-space with a prescribed projection of one field of principal directions onto a surface $Π$ is considered. A problem of determination of such surfaces when both principal curvatures are given over a line in $Π$ is formulated and studied. The geometric problem is reduced to th…

A new algorithm reduces the time and space complexity for multinomial logistic bandits.

Study on null hypersurfaces with constant angle in Lorentzian manifolds.

New superintegrable systems derived from Frobenius structures.

SAM optimizes deep networks by oscillating between sides of the minimum.