arXiv research

In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization. To the best of our knowledg…

This paper proves, in very general settings, that convex risk minimization is a procedure to select a unique conditional probability model determined by the classification problem. Unlike most previous work, we give results that are general enough to include cases in which no minimum exists, as occurs typically, for in…

New algorithms help machines forget old data efficiently.

Optimizes exp-concave losses with a new risk bound.

New method turns optimization algorithms into uniformly stable learning algorithms for non-Euclidean norms.

Differential privacy is concerned about the prediction quality while measuring the privacy impact on individuals whose information is contained in the data. We consider differentially private risk minimization problems with regularizers that induce structured sparsity. These regularizers are known to be convex but they…

The paper tackles performative risk optimization under weak convexity assumptions.

Deep neural networks reduce portfolio tail-risk by 99% in crisis-era simulations.

In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …

Adversarial consistency depends on the uniqueness of adversarial Bayes classifiers.

Equivalent characterizations of multiportfolio time consistency are deduced for closed convex and coherent set-valued risk measures on $L^p(Ω,\mathcal F, P; R^d)$ with image space in the power set of $L^p(Ω,\mathcal F_t,P;R^d)$. In the convex case, multiportfolio time consistency is equivalent to a cocycle condition on…

New framework for optimizing machine learning risks.

We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framewo…

Improved ADMM for convex distributed learning with differential privacy.

The paper analyzes local minima in high-dimensional empirical risk minimization.

The paper characterizes dynamic return and star-shaped risk measures via BSDEs.

New study shows ERMs can fail in convex optimization with high dimensionality.

We study Spectral Measures of Risk from the perspective of portfolio optimization. We derive exact results which extend to general Spectral Measures M_phi the Pflug--Rockafellar--Uryasev methodology for the minimization of alpha--Expected Shortfall. The minimization problem of a spectral measure is shown to be equivale…

The paper analyzes elicitability of return risk measures and their scoring functions.

Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.

We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empiric…

We consider the optimization of active extension portfolios. For this purpose, the optimization problem is rewritten as a stochastic programming model and solved using a clever multi-start local search heuristic, which turns out to provide stable solutions. The heuristic solutions are compared to optimization results o…

Optimizes portfolios with GM returns using convex optimization.

We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…

Graphical models trained using maximum likelihood are a common tool for probabilistic inference of marginal distributions. However, this approach suffers difficulties when either the inference process or the model is approximate. In this paper, the inference process is first defined to be the minimization of a convex f…

Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.

The study uncovers the breakdown of Gaussian universality in high-dimensional empirical risk minimization.

In this work we consider adversarial contextual bandits with risk constraints. At each round, nature prepares a context, a cost for each arm, and additionally a risk for each arm. The learner leverages the context to pull an arm and then receives the corresponding cost and risk associated with the pulled arm. In additi…

Improved Frank-Wolfe method reduces dependence on data size for empirical risk minimization.

A new RL framework for risk-sensitive decision-making using convex scoring functions.

The paper bounds payoffs and option prices in discrete models.

The paper generalizes offset Rademacher complexities to convex and non-convex problems.

Dual averaging-type methods are widely used in industrial machine learning applications due to their ability to promoting solution structure (e.g., sparsity) efficiently. In this paper, we propose a novel accelerated dual-averaging primal-dual algorithm for minimizing a composite convex function. We also derive a stoch…

In regularized risk minimization, the associated optimization problem becomes particularly difficult when both the loss and regularizer are nonsmooth. Existing approaches either have slow or unclear convergence properties, are restricted to limited problem subclasses, or require careful setting of a smoothing parameter…

Optimal private ERM and SCO with subquadratic gradient complexity.

New DP algorithm improves privacy and efficiency for convex optimization.

We consider the problem of minimizing a sum of clipped convex functions; applications include clipped empirical risk minimization and clipped control. While the problem of minimizing the sum of clipped convex functions is NP-hard, we present some heuristics for approximately solving instances of these problems. These h…

Stochastic approximation (SA) is a classical approach for stochastic convex optimization. Previous studies have demonstrated that the convergence rate of SA can be improved by introducing either smoothness or strong convexity condition. In this paper, we make use of smoothness and strong convexity simultaneously to boo…

A wide array of machine learning problems are formulated as the minimization of the expectation of a convex loss function on some parameter space. Since the probability distribution of the data of interest is usually unknown, it is is often estimated from training sets, which may lead to poor out-of-sample performance.…

Risk measures for multivariate financial positions are studied in a utility-based framework. Under a certain incomplete preference relation, shortfall and divergence risk measures are defined as the optimal values of specific set minimization problems. The dual relationship between these two classes of multivariate ris…

This paper aims to provide a better understanding of a symmetric loss. First, we emphasize that using a symmetric loss is advantageous in the balanced error rate (BER) minimization and area under the receiver operating characteristic curve (AUC) maximization from corrupted labels. Second, we prove general theoretical p…

Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …

We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss. In contrast, we develop a co…

New method calibrates diffusion models for image regression tasks.

We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…

In incomplete financial markets not every contingent claim can be replicated by a self-financing strategy. The risk of the resulting shortfall can be measured by convex risk measures, recently introduced by Föllmer, Schied (2002). The dynamic optimization problem of finding a self-financing strategy that minimizes the …

We propose a general approach for supervised learning with structured output spaces, such as combinatorial and polyhedral sets, that is based on minimizing estimated conditional risk functions. Given a loss function defined over pairs of output labels, we first estimate the conditional risk function by solving a (possi…

New bounds show polyhedral surrogates are optimal for generalization.