arXiv research

Proposes Coherent Gradients to explain and reduce overfitting in neural networks.

Proposes a method to compare neural networks using feature and gradient vectors.

In optimization, the natural gradient method is well-known for likelihood maximization. The method uses the Kullback-Leibler divergence, corresponding infinitesimally to the Fisher-Rao metric, which is pulled back to the parameter space of a family of probability distributions. This way, gradients with respect to the p…

Method generates visual explanations for similarity models without classification.

Task-specific scores are often used to optimize for and evaluate the performance of conditional text generation systems. However, such scores are non-differentiable and cannot be used in the standard supervised learning paradigm. Hence, policy gradient methods are used since the gradient can be computed without requiri…

New framework monitors neural network training and reveals regularisation mechanisms.

Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.

Cosine similarity can force points to grow in magnitude, causing convergence issues.

Study evaluates relevance metrics for similarity-based model explanations.

Algorithm learns similarity metrics for individual fairness.

Optimizes shapes of curves using Möbius energy gradients.

In this paper we present CatBoost, a new open-sourced gradient boosting library that successfully handles categorical features and outperforms existing publicly available implementations of gradient boosting in terms of quality on a set of popular publicly available datasets. The library has a GPU implementation of lea…

New hashing method improves document retrieval precision.

Charmer improves character-level adversarial attacks for language models.

The paper establishes gradient estimates for harmonic and heat equation solutions on manifolds with boundary.

New model-free algorithm achieves similar LQR regret guarantees.

We show that the linear trace Harnack quadratic on a steady gradient Ricci soliton satisfies the heat equation. Similar result holds for shrinkers. We also present an interpolation between Perelman's and Cao--Hamilton's Harnacks on a steady soliton.

DVGS identifies low-quality data quickly and accurately.

Generative Adversarial Networks create realistic brain MRI images.

Survey on gradient Ricci solitons in 4D, focusing on geometry and classification.

Optimizes LLM prompts using logged user feedback.

In this paper, we prove that the $L^p$ essential spectra of the Laplacian on functions are $[0,+\infty)$ on a non-compact complete Riemannian manifold with non-negative Ricci curvature at infinity. The similar method applies to gradient shrinking Ricci soliton, which is similar to non-compact manifold with non-negative…

We characterize complete nonnegatively curved steady gradient soliton with curvature in L^1. We show that there are isometric to a product (R^2,g_{cigar}) times(R^{n-2}, eucl))/Gamma where Gamma is a Bieberbach group of rank n-2. We prove also a similar local splitting result under weaker curvature assumptions.

In this paper we propose a method to build a neural network that is similar to an ensemble of decision trees. We first illustrate how to convert a learned ensemble of decision trees to a single neural network with one hidden layer and an input transformation. We then relax some properties of this network such as thresh…

Proposes a semi-implicit back propagation method for neural networks.

New method for bidirectional generative modeling using adversarial gradient estimation.

We study the problem of meta-learning through the lens of online convex optimization, developing a meta-algorithm bridging the gap between popular gradient-based meta-learning and classical regularization-based multi-task transfer methods. Our method is the first to simultaneously satisfy good sample efficiency guarant…

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity be…

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two terms: (i) a stochastic term which is not affected by the delay, and (ii) a higher …

COMP-AMS optimizes distributed training with compressed gradients, achieving similar accuracy with less communication.

Method measures weight similarity in neural networks using normalization and statistical inference.

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). Our analysis relies on the p…

Stochastic gradient descent (SGD) has been widely used in machine learning due to its computational efficiency and favorable generalization properties. Recently, it has been empirically demonstrated that the gradient noise in several deep learning settings admits a non-Gaussian, heavy-tailed behavior. This suggests tha…

Proves well-posedness of gradient solitons on bundle gerbe.

Paper introduces a new, tractable measure of model complexity.

We give a simple proof that the Frank-Wolfe algorithm obtains a stationary point at a rate of $O(1/\sqrt{t})$ on non-convex objectives with a Lipschitz continuous gradient. Our analysis is affine invariant and is the first, to the best of our knowledge, giving a similar rate to what was already proven for projected gra…

Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…

New GPU algorithm boosts machine learning with larger datasets.

We consider the problem of minimizing the sum of two convex functions: one is the average of a large number of smooth component functions, and the other is a general convex function that admits a simple proximal mapping. We assume the whole objective function is strongly convex. Such problems often arise in machine lea…

New convergence guarantees for learning with unknown nuisance parameters.

UMAP's true loss function differs from what was previously thought, focusing on nearest neighbor graph similarities.

Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…

Short note on soft-max and policy gradients in bandit problems using Lyapunov functions.

Adam's hyperparameters implicitly regularize solutions, penalizing or impeding loss gradients' norms.

Study on curvature decay in steady Ricci solitons, proving dichotomy.

Continuing our previous work (arXiv:1509.07981v1), we derive another global gradient estimate for positive functions, particularly for positive solutions to the heat equation on finite or locally finite graphs. In general, the gradient estimate in the present paper is independent of our previous one. As applications, i…

ES and FD gradients converge as optimization dimension grows.

A new method reduces complexity and uncertainty in neural networks.