arXiv research

Study complete gradient Ricci solitons with zero radial Weyl curvature.

Gradient descent struggles to achieve zero loss in deep learning models due to non-generic data distributions.

Gradient methods converge exponentially in concave network games.

Zero loss is achievable in overparametrized DL networks under specific conditions.

Gradient descent learns ReLU functions with non-zero bias efficiently.

We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…

Given a 3-manifold M with no spherical boundary components, and a primitive class φin H^1(M;Z), we show that the following are equivalent: (1) φis a fibered class, (2) the rank gradient of (M,φ) is zero, (3) the Heegaard gradient of (M,φ) is zero.

The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.

Paper proposes a mean-field gradient descent for zero-sum games, proving convergence to Nash equilibrium.

Gradient Descent Ascent converges to von-Neumann solution in hidden zero-sum games.

Deep ResNets can achieve zero loss with enough layers and weights.

This work proposes using zero-variance control variates to reduce variance in pathwise gradient estimators for variational inference.

A new gradient estimator for online optimization with two function evaluations.

The infimal Heegaard gradient of a compact 3-manifold was defined and studied by Marc Lackenby in an approach toward the well-known virtually Haken conjecture. As instructive examples, we consider Seifert fibered 3-manifolds, and show that a Seifert fibered 3-manifold has zero infimal Heegaard gradient if and only if i…

A new algorithm optimizes softmax units in large language models.

Embeds sparsity in deep neural networks, allowing exact zero parameters.

Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data. In this…

Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, while understanding the dynamics of gradient algorithms for solving such formulations has remained a grand challenge. As a first step, we restrict to bilinear zero-sum games and giv…

We show that gradient shrinking, expanding or steady Ricci solitons have potentials leading to suitable reference probability measures on the manifold. For shrinking solitons, as well as expanding soltions with nonnegative Ricci curvature, these reference measures satisfy sharp logarithmic Sobolev inequalities with low…

Gradient descent finds halfspaces with low error for agnostic learning.

Cyclic Data Parallelism reduces memory usage and balances gradient communications.

LLMs predict SGD convergence without training on specific data.

A method predicts GNS of transformer layers using normalization layer norms.

Overparameterized deep networks have the capacity to memorize training data with zero \emph{training error}. Even after memorization, the \emph{training loss} continues to approach zero, making the model overconfident and the test performance degraded. Since existing regularizers do not directly aim to avoid zero train…

Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…

It is well known that Markov chain Monte Carlo (MCMC) methods scale poorly with dataset size. A popular class of methods for solving this issue is stochastic gradient MCMC. These methods use a noisy estimate of the gradient of the log posterior, which reduces the per iteration computational cost of the algorithm. Despi…

Randomly sampled interpolators achieve zero generalization error with enough data.

Introduces a differentiable approximation to the zero-one loss.

Method generates time-series attribution maps with identifiability guarantees.

Gradient-based methods for games suffer from discrete update steps that cause drift, affecting performance.

We consider regular surfaces $M$ that are given as the zeros of a polynomial function $p:R^3\rightarrow R$, where the gradient of $p$ vanishes nowhere. We assume that $M$ has non-zero mean curvature and prove that there exist only two examples of such surfaces, namely the sphere and the circular cylinder.

Gradient-free optimization for additive models achieves optimal error.

Graph manifolds are manifolds that decompose along tori into pieces with a tame $S^1$-structure. In this paper, we prove that the simplicial volume of graph manifolds (which is known to be zero) can be approximated by integral simplicial volumes of their finite coverings. This gives a uniform proof of the vanishing of …

Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…

We prove that if the fundamental group of an orientable finite volume hyperbolic 3-manifold has finite index in the reflection group of a right-angled ideal polyhedra in $\mathbb{H}^3$ then it has a co-final tower of finite sheeted covers with positive rank gradient. The manifolds we provide are also known to have co-f…

This paper provides a mathematical foundation for deep neural networks solving PDEs.

Gradient descent converges linearly for neural networks with specific conditions.

Adam optimizer converges to zeros of a new vector field, not just gradient zeros.

A new method for distributed optimization with noisy function evaluations.

Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.

Extends neural network training framework to handle noise and uncertainty.

New method trains neural networks with threshold activation functions efficiently.

We propose a simple and general variant of the standard reparameterized gradient estimator for the variational evidence lower bound. Specifically, we remove a part of the total derivative with respect to the variational parameters that corresponds to the score function. Removing this term produces an unbiased gradient …

Study optimizes zero-order strongly convex function minimization with higher order smoothness.

This paper explains how low-precision arithmetic causes loss spikes in deep learning models.

In this paper, we show that steady or shrinking complete gradient Yamabe solitons with finite total scalar curvature and non-positive Ricci curvature are Ricci flat. Moreover, under certain pinching condition for Ricci curvature, we show that steady or shrinking complete gradient Yamabe solitons with finite total scala…

In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning…