arXiv research

Develops a mathematical model for automatic differentiation in machine learning.

We introduce a proximal subdifferential and develop a calculus for nonsmooth functions defined on any Riemannian manifold $M$. We give several applications of this theory, concerning: 1) differentiability and geometrical properties of the distance function to a closed subset $C$ of $M$; 2) solvability and implicit func…

New theory for nonsmooth systems helps optimize and control complex functions.

We establish some perturbed minimization principles, and we develop a theory of subdifferential calculus, for functions defined on Riemannian manifolds. Then we apply these results to show existence and uniqueness of viscosity solutions to Hamilton-Jacobi equations defined on Riemannian manifolds.

We discuss in which sense general metric measure spaces possess a first order differential structure. Building on this, we then see that on spaces with Ricci curvature bounded from below a second order calculus can be developed, permitting to define Hessian, covariant/exterior derivatives and Ricci curvature.

We introduce novel equations, in the spirit of rough path theory, that parametrize level sets of intrinsically regular maps on the Heisenberg group with values in $\mathbb{R}^2$. These equations can be seen as a sub-Riemannian counterpart to classical ODEs arising from the implicit function theorem. We show that they e…

In this paper we present a new theory of calculus over $k$-dimensional domains in a smooth $n$-manifold, unifying the discrete, exterior, and continuum theories. The calculus begins at a single point and is extended to chains of finitely many points by linearity, or superposition. It converges to the smooth continuum w…

We discuss smooth nonlinear control systems with symmetry. For a free and proper action of the symmetry group, the reduction of symmetry gives rise to a reduced smooth nonlinear control system. If the action of the symmetry group is only proper, the reduced nonlinear control system need not be smooth. Using the smooth …

PPGD solves nonconvex nonsmooth optimization problems without KL property.

Study on Adam-family methods for nonsmooth optimization with convergence guarantees.

This paper tackles nonsmooth optimization in machine learning.

In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits the structure of common nonsmooth loss functions to achieve optimal convergence ra…

Study on nonsmooth contractive SA with constant stepsize and Q-learning.

Study compares nonsmooth spaces with integrable Ricci bounds.

New stability bounds for SGD on nonsmooth convex losses.

Regularity results for geodesic X-ray transform on nonsmooth manifolds

New methods help escape strict saddle points in nonsmooth optimization.

In this paper, we study the Kurdyka-Łojasiewicz (KL) exponent, an important quantity for analyzing the convergence rate of first-order methods. Specifically, we develop various calculus rules to deduce the KL exponent of new (possibly nonconvex and nonsmooth) functions formed from functions with known KL exponents. In …

New Langevin Monte Carlo algorithms for sampling from nonsmooth distributions.

We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…

New method solves nonsmooth low-rank matrix optimization problems efficiently.

Study on numerical reliability of AD for MaxPool in neural nets.

Paper develops privacy-preserving federated learning for nonsmooth objectives.

The paper defines measures acting in duality with sections of bundles in curved spaces.

Paper develops algorithms for nonsmooth, nonconvex statistical learning problems.

Theory of parallel transport on non-collapsed RCD spaces established.

We construct some nonsmoothable actions of Z2 * Z2 on spin four-manifolds by using an equivariant version of Furuta' s 10/8inequality. The examples satisfy following property: any proper subgroup of Z2 * Z2 is smoothable for some smooth structure.

In regularized risk minimization, the associated optimization problem becomes particularly difficult when both the loss and regularizer are nonsmooth. Existing approaches either have slow or unclear convergence properties, are restricted to limited problem subclasses, or require careful setting of a smoothing parameter…

We construct a nonsmoothable Z\times Z-action on the connected sum of an Enriques surface and S^2\times S^2, such that each of generators is smoothable. We also construct a nonsmoothable self-homeomorphism on an Enriques surface.

Paper develops an online covariance estimator for nonsmooth stochastic approximation problems.

Unified Lagrangian-based methods for nonsmooth nonconvex optimization.

We consider in this paper a class of composite optimization problems whose objective function is given by the summation of a general smooth and nonsmooth component, together with a relatively simple nonsmooth term. We present a new class of first-order methods, namely the gradient sliding algorithms, which can skip the…

Nonconvex and nonsmooth problems have recently attracted considerable attention in machine learning. However, developing efficient methods for the nonconvex and nonsmooth optimization problems with certain performance guarantee remains a challenge. Proximal coordinate descent (PCD) has been widely used for solving opti…

We show that every closed, simply connected, spin topological 4-manifold except $S^4$ and $S^2\times S^2$ admits a homologically trivial, pseudofree, locally linear action of $\mathbb{Z}_p$ for any sufficiently large prime number $p$ which is nonsmoothable for any possible smooth structure.

The paper guarantees global stability for stochastic subgradient methods in nonsmooth nonconvex optimization.

We consider a class of nonconvex nonsmooth optimization problems whose objective is the sum of a smooth function and a finite number of nonnegative proper closed possibly nonsmooth functions (whose proximal mappings are easy to compute), some of which are further composed with linear maps. This kind of problems arises …

Nesterov's extrapolation improves convergence in nonsmooth optimization.

Improved shuffling gradient methods converge faster for nonsmooth convex optimization.

Two new methods solve nonsmooth optimization on Riemannian Stiefel manifold.

Study Langevin Monte Carlo for sampling non-log-concave distributions.

Adam's convergence rate in nonsmooth nonconvex optimization is analyzed.

This paper concerns dictionary learning, i.e., sparse coding, a fundamental representation learning problem. We show that a subgradient descent algorithm, with random initialization, can provably recover orthogonal dictionaries on a natural nonsmooth, nonconvex $\ell_1$ minimization formulation of the problem, under mi…

We present a stochastic setting for optimization problems with nonsmooth convex separable objective functions over linear equality constraints. To solve such problems, we propose a stochastic Alternating Direction Method of Multipliers (ADMM) algorithm. Our algorithm applies to a more general class of nonsmooth convex …

Study shows limitations of Lie bracket commutation for nonsmooth vector fields.

In this paper, we consider the problem of minimizing the average of a large number of nonsmooth and convex functions. Such problems often arise in typical machine learning problems as empirical risk minimization, but are computationally very challenging. We develop and analyze a new algorithm that achieves robust linea…

New solver SR2 tackles deep neural network training with nonsmooth regularization.

By the Thurston stability theorem, a group of C^1 orientation-preserving diffeomorphisms of the closed unit interval is locally indicable. We show that the local order structure of orbits gives a stronger criterion for nonsmoothability that can be used to produce new examples of locally indicable groups of homeomorphis…

Much of the vast literature on the integral during the last two centuries concerns extending the class of integrable functions. In contrast, our viewpoint is akin to that taken by Hassler Whitney [{\it Geometric integration theory}, Princeton Univ. Press, Princeton, NJ, 1957] and by geometric measure theorists because …