arXiv research

New algorithms optimize non-smooth, non-convex objectives with improved complexity.

Paper tackles private optimization for non-smooth objectives efficiently.

Averaged SGD optimizes a smoothed objective, leading to better generalization.

This paper studies continuum-armed bandits under Besov smoothness conditions and derives minimax rates.

Hierarchical randomized smoothing improves model robustness for complex data.

We present a new perspective on the celebrated Sinkhorn algorithm by showing that is a special case of incremental/stochastic mirror descent. In order to see this, one should simply plug Kullback-Leibler divergence in both mirror map and the objective function. Since the problem has unbounded domain, the objective func…

AsylADMM improves gossip-based learning for non-smooth objectives.

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipsc…

In this paper, we consider efficient differentially private empirical risk minimization from the viewpoint of optimization algorithms. For strongly convex and smooth objectives, we prove that gradient descent with output perturbation not only achieves nearly optimal utility, but also significantly improves the running …

New method tackles bilevel optimization with polyhedral constraints.

Paper analyzes convergence of stochastic methods under heavy-tailed noise.

New algorithms optimize without knowing problem parameters.

AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.

A body of recent work has focused on constructing a variational family of filtered distributions using Sequential Monte Carlo (SMC). Inspired by this work, we introduce Particle Smoothing Variational Objectives (SVO), a novel backward simulation technique and smoothed approximate posterior defined through a subsampling…

New method smooths optimization for sparse regularization.

ConquerNet smooths quantile regression for deep learning with minimax guarantees.

We introduce the notion of a symplectic hopfoid, which is a "groupoid-like" object in the category of symplectic manifolds where morphisms are given by canonical relations. Such groupoid-like objects arise when applying a version of the cotangent functor to the structure maps of a Lie groupoid. We show that such object…

We compare various different definitions of "the category of smooth objects". The definitions compared are due to Chen, Frölicher, Sikorski, Smith, and Souriau. The method of comparison is to construct functors between the categories that enable us to see how the categories relate to each other. This produces a diagram…

We propose a stochastic approximation method for approximating the efficient frontier of chance-constrained nonlinear programs. Our approach is based on a bi-objective viewpoint of chance-constrained programs that seeks solutions on the efficient frontier of optimal objective value versus risk of constraint violation. …

New geometric objects generalize Lie groupoids, with nontrivial tangent bundle properties.

A new biased gradient descent method for conditional stochastic optimization.

If $X$ is a smooth manifold then the $\mathbb R$-algebra $C^\infty(X)$ of smooth functions $c:X\to\mathbb R$ is a $C^\infty$-$ring$. That is, for each smooth function $f:{\mathbb R}^n\to\mathbb R$ there is an $n$-fold operation $Φ_f:C^\infty(X)^n\to C^\infty(X)$ acting by $Φ_f:(c_1,\ldots,c_n)\mapsto f(c_1,...,c_n)$, a…

New algorithms minimize noisy, irregular functions without gradients.

Smooth Kahler-Einstein metrics have been studied for the past 80 years. More recently, singular Kahler-Einstein metrics have emerged as objects of intrinsic interest, both in differential and algebraic geometry, as well as a powerful tool in better understanding their smooth counterparts. This article is mostly a surve…

Armijo line-search speeds up gradient descent for various functions.

We propose Top-N-Rank, a novel family of list-wise Learning-to-Rank models for reliably recommending the N top-ranked items. The proposed models optimize a variant of the widely used discounted cumulative gain (DCG) objective function which differs from DCG in two important aspects: (i) It limits the evaluation of DCG …

New method improves robustness of large models without sacrificing accuracy.

This paper proposes a novel proximal-gradient algorithm for a decentralized optimization problem with a composite objective containing smooth and non-smooth terms. Specifically, the smooth and nonsmooth terms are dealt with by gradient and proximal updates, respectively. The proposed algorithm is closely related to a p…

Bayesian optimization on networks using Gaussian process models.

Hypergraph is a general way of representing high-order relations on a set of objects. It is a generalization of graph, in which only pairwise relations can be represented. It finds applications in various domains where relationships of more than two objects are observed. On a hypergraph, as a generalization of graph, o…

We consider the problem of finding local minimizers in non-convex and non-smooth optimization. Under the assumption of strict saddle points, positive results have been derived for first-order methods. We present the first known results for the non-smooth case, which requires different analysis and a different algorithm…

SIXO improves inference by learning smoothing distributions from all observations.

A new clustering algorithm considers data smoothness for better performance.

Many real-world objects are designed by smooth curves, especially in the domain of aerospace and ship, where aerodynamic shapes (e.g., airfoils) and hydrodynamic shapes (e.g., hulls) are designed. To facilitate the design process of those objects, we propose a deep learning based generative model that can synthesize sm…

We consider the fundamental problem in non-convex optimization of efficiently reaching a stationary point. In contrast to the convex case, in the long history of this basic problem, the only known theoretical results on first-order non-convex optimization remain to be full gradient descent that converges in $O(1/\varep…

MGDA converges under generalized smoothness for neural network optimization.

PPOS improves PPO by smoothing the surrogate objective function.

New DRO algorithm finds robust models without strong assumptions.

FeDualEx tackles saddle point optimization in federated learning with composite objectives.

We establish a relation between smooth 2-functors defined on the path 2-groupoid of a smooth manifold and differential forms on this manifold. This relation can be understood as a part of a dictionary between fundamental notions from category theory and differential geometry. We show that smooth 2-functors appear in se…

New method tackles non-smooth tensor data for better recovery.

We review the basic definition of a stack and apply it to the topological and smooth settings. We then address two subtleties of the theory: the correct definition of a ``stack over a stack'' and the distinction between small stacks (which are algebraic objects) and large stacks (which are generalized spaces).

We present a framework to train a structured prediction model by performing smoothing on the inference algorithm it builds upon. Smoothing overcomes the non-smoothness inherent to the maximum margin structured prediction objective, and paves the way for the use of fast primal gradient-based optimization algorithms. We …

Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner p…

This paper proposes a method for machine learning from unlabeled data in the form of a time-series. The mapping that is learned is shown to extract slowly evolving information that would be useful for control applications, while efficiently filtering out unwanted, higher-frequency noise. The method consists of training…

Auto-regressive models improve smoothing efficiency with exponentially tapered windows.

In unsupervised learning, there is no apparent straightforward cost function that can capture the significant factors of variations and similarities. Since natural systems have smooth dynamics, an opportunity is lost if an unsupervised objective function remains static during the training process. The absence of concre…

Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…