arXiv research

Study complete gradient Ricci solitons with zero radial Weyl curvature.

Gradient descent struggles to achieve zero loss in deep learning models due to non-generic data distributions.

Gradient methods converge exponentially in concave network games.

Zero loss is achievable in overparametrized DL networks under specific conditions.

Gradient descent learns ReLU functions with non-zero bias efficiently.

We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…

Given a 3-manifold M with no spherical boundary components, and a primitive class φin H^1(M;Z), we show that the following are equivalent: (1) φis a fibered class, (2) the rank gradient of (M,φ) is zero, (3) the Heegaard gradient of (M,φ) is zero.

The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.

Paper proposes a mean-field gradient descent for zero-sum games, proving convergence to Nash equilibrium.

Gradient Descent Ascent converges to von-Neumann solution in hidden zero-sum games.

Deep ResNets can achieve zero loss with enough layers and weights.

This work proposes using zero-variance control variates to reduce variance in pathwise gradient estimators for variational inference.

A new gradient estimator for online optimization with two function evaluations.

The infimal Heegaard gradient of a compact 3-manifold was defined and studied by Marc Lackenby in an approach toward the well-known virtually Haken conjecture. As instructive examples, we consider Seifert fibered 3-manifolds, and show that a Seifert fibered 3-manifold has zero infimal Heegaard gradient if and only if i…

Flooding prevents deep networks from achieving zero training loss, improving test performance.

A new algorithm optimizes softmax units in large language models.

Embeds sparsity in deep neural networks, allowing exact zero parameters.

Uncertainty sampling, a popular active learning algorithm, is used to reduce the amount of data required to learn a classifier, but it has been observed in practice to converge to different parameters depending on the initialization and sometimes to even better parameters than standard training on all the data. In this…

Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, while understanding the dynamics of gradient algorithms for solving such formulations has remained a grand challenge. As a first step, we restrict to bilinear zero-sum games and giv…

We show that gradient shrinking, expanding or steady Ricci solitons have potentials leading to suitable reference probability measures on the manifold. For shrinking solitons, as well as expanding soltions with nonnegative Ricci curvature, these reference measures satisfy sharp logarithmic Sobolev inequalities with low…

Gradient descent finds halfspaces with low error for agnostic learning.

Cyclic Data Parallelism reduces memory usage and balances gradient communications.

LLMs predict SGD convergence without training on specific data.

Method identifies mixed Nash equilibria in high dimensions for training mixtures of GANs.

A method predicts GNS of transformer layers using normalization layer norms.

Gradient descent stagnates in low-precision, but unbiased rounding schemes improve convergence.

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non…

It is well known that Markov chain Monte Carlo (MCMC) methods scale poorly with dataset size. A popular class of methods for solving this issue is stochastic gradient MCMC. These methods use a noisy estimate of the gradient of the log posterior, which reduces the per iteration computational cost of the algorithm. Despi…

Randomly sampled interpolators achieve zero generalization error with enough data.

Introduces a differentiable approximation to the zero-one loss.

Method generates time-series attribution maps with identifiability guarantees.

Gradient-based methods for games suffer from discrete update steps that cause drift, affecting performance.

We consider regular surfaces $M$ that are given as the zeros of a polynomial function $p:R^3\rightarrow R$, where the gradient of $p$ vanishes nowhere. We assume that $M$ has non-zero mean curvature and prove that there exist only two examples of such surfaces, namely the sphere and the circular cylinder.

Gradient-free optimization for additive models achieves optimal error.

Graph manifolds are manifolds that decompose along tori into pieces with a tame $S^1$-structure. In this paper, we prove that the simplicial volume of graph manifolds (which is known to be zero) can be approximated by integral simplicial volumes of their finite coverings. This gives a uniform proof of the vanishing of …

Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…

We prove that if the fundamental group of an orientable finite volume hyperbolic 3-manifold has finite index in the reflection group of a right-angled ideal polyhedra in $\mathbb{H}^3$ then it has a co-final tower of finite sheeted covers with positive rank gradient. The manifolds we provide are also known to have co-f…

This paper provides a mathematical foundation for deep neural networks solving PDEs.

Proposes a new training algorithm for zero-sum games to avoid convergence issues.

Gradient descent converges linearly for neural networks with specific conditions.

Adam optimizer converges to zeros of a new vector field, not just gradient zeros.

A new method for distributed optimization with noisy function evaluations.

Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.

Extends neural network training framework to handle noise and uncertainty.

New method trains neural networks with threshold activation functions efficiently.

We propose a simple and general variant of the standard reparameterized gradient estimator for the variational evidence lower bound. Specifically, we remove a part of the total derivative with respect to the variational parameters that corresponds to the score function. Removing this term produces an unbiased gradient …

Study optimizes zero-order strongly convex function minimization with higher order smoothness.

This paper explains how low-precision arithmetic causes loss spikes in deep learning models.