arXiv research

Paper develops momentum schemes with variance reduction for non-convex composition optimization.

Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The objective function is the composition of two expectations of stochastic functions, and is more challenging to optimize than vanilla stochastic o…

AdaGrad fails to adapt to Hölder-smoothness in composite optimization problems.

New method tackles composite optimization with error feedback.

FeDualEx tackles saddle point optimization in federated learning with composite objectives.

The stochastic gradient descent has been widely used for solving composite optimization problems in big data analyses. Many algorithms and convergence properties have been developed. The composite functions were convex primarily and gradually nonconvex composite functions have been adopted to obtain more desirable prop…

Consider the stochastic composition optimization problem where the objective is a composition of two expected-value functions. We propose a new stochastic first-order method, namely the accelerated stochastic compositional proximal gradient (ASC-PG) method, which updates based on queries to the sampling oracle using tw…

Develops consistent approximations for composite optimization problems.

Improved subgradient method tackles ill-conditioned composite optimization problems.

This paper advances FL algorithms for composite optimization and statistical recovery.

Paper analyzes stability and generalization of SCO algorithms.

Unified algorithm for minimizing composite functions with flexible design.

Here we study non-convex composite optimization: first, a finite-sum of smooth but non-convex functions, and second, a general function that admits a simple proximal mapping. Most research on stochastic methods for composite optimization assumes convexity or strong convexity of each function. In this paper, we extend t…

CARE method estimates precision matrix for compositional data, achieving optimality in high dimensions.

Paper proposes iLPA for solving DC composite optimization problems, with applications to matrix completion with outliers.

Optimizes natural frequencies of cellular composites with various microstructures.

New approach for distributed online optimization of non-convex losses with sublinear regret.

Adaptive sampling method solves constrained and composite optimization problems.

Many machine learning, statistical inference, and portfolio optimization problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG)…

We consider multi-level composite optimization problems where each mapping in the composition is the expectation over a family of random smooth mappings or the sum of some finite number of smooth mappings. We present a normalized proximal approximate gradient (NPAG) method where the approximate gradients are obtained v…

Paper solves robust convex problems with heavy-tailed noise.

AutoBayes simplifies variational inference by composing models and optimizing them.

This paper studies robust estimation methods in high dimensions, comparing model-averaged and composite quantile estimators.

We consider the composition optimization with two expected-value functions in the form of $\frac{1}{n}\sum\nolimits_{i = 1}^n F_i(\frac{1}{m}\sum\nolimits_{j = 1}^m G_j(x))+R(x)$, { which formulates many important problems in statistical learning and machine learning such as solving Bellman equations in reinforcement l…

We consider in this paper a class of composite optimization problems whose objective function is given by the summation of a general smooth and nonsmooth component, together with a relatively simple nonsmooth term. We present a new class of first-order methods, namely the gradient sliding algorithms, which can skip the…

We consider optimization of composite objective functions, i.e., of the form $f(x)=g(h(x))$, where $h$ is a black-box derivative-free expensive-to-evaluate function with vector-valued outputs, and $g$ is a cheap-to-evaluate real-valued function. While these problems can be solved with standard Bayesian optimization, we…

Optimizes convergence rate of stochastic proximal algorithms for composite convex problems.

New scalable methods for robust model learning from large datasets.

There is tremendous interest in precision medicine as a means to improve patient outcomes by tailoring treatment to individual characteristics. An individualized treatment rule formalizes precision medicine as a map from patient information to a recommended treatment. A treatment rule is defined to be optimal if it max…

BOIS optimizes complex systems by leveraging structural knowledge.

In this paper, we develop a convolutional neural network model to predict the mechanical properties of a two-dimensional checkerboard composite quantitatively. The checkerboard composite possesses two phases, one phase is soft and ductile while the other is stiff and brittle. The ground-truth data used in the training …

New algorithm tackles nested bi-level optimization problems for robust feature learning.

In the past few years, off-policy reinforcement learning methods have shown promising results in their application for robot control. Deep Q-learning, however, still suffers from poor data-efficiency and is susceptible to stochasticity in the environment or reward functions which is limiting with regard to real-world a…

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that…

We propose a novel probabilistic approach to multilevel clustering problems based on composite transportation distance, which is a variant of transportation distance where the underlying metric is Kullback-Leibler divergence. Our method involves solving a joint optimization problem over spaces of probability measures t…

Unified framework for aligning and composing diffusion models to satisfy multiple constraints.

A portfolio of different stocks and a risk-less security whose composition is dynamically maintained stable by trading shares at any time step leads to a growth of the capital with a nonrandom rate. This is the key for the theory of optimal-growth investment formulated by Kelly. In presence of transaction costs, the op…

New algorithm solves complex optimization problems without needing projections.

Many machine learning models, such as logistic regression~(LR) and support vector machine~(SVM), can be formulated as composite optimization problems. Recently, many distributed stochastic optimization~(DSO) methods have been proposed to solve the large-scale composite optimization problems, which have shown better per…

SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms, and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in smooth nonconvex optimization. However, SPIDER uses an accuracy-dependent stepsize that slows down the convergence in practice, and cannot handle objec…

BL learns interpretable optimization structures from data.

Paper tackles distributed linear regression with compositional covariates.

Our objective is to estimate the unknown compositional input from its output response through an unknown system after estimating the inverse of the original system with a training set. The proposed methods using artificial neural networks (ANNs) can compete with the optimal bounds for linear systems, where convex optim…

Gaussian processes struggle with compositional functions, but deep Gaussian processes can outperform.

Classical stochastic gradient methods are well suited for minimizing expected-value objective functions. However, they do not apply to the minimization of a nonlinear function involving expected values or a composition of two expected-value functions, i.e., problems of the form $\min_x \mathbf{E}_v [f_v\big(\mathbf{E}_…

The paper accelerates ISTA and FISTA algorithms for composite optimization problems.

The ERI is a new index for measuring exam readiness.

Composite likelihood inference of fractional Gaussian processes with sequentially optimal subset selection