Improved sample complexity for diffusion models without needing empirical risk minimizers.
problem Theoretical limitations in sample complexity for diffusion models.
method Structured decomposition of score estimation error, eliminating dependence on neural network parameters.
result Achieved sample complexity bound of O(ε^(-4)) without empirical risk minimizer access.
Paper analyzes time series prediction using empirical risk minimization.
problem Optimizing 1-step-ahead prediction for time series.
method Empirical risk minimization applied to recursive algorithms for time series forecasting.
result Empirical risk minimization achieves optimal predictive performance.
The paper analyzes the performance of empirical risk minimization for p-norm linear regression.
problem Empirical risk minimization on p-norm linear regression. method Analyzes performance under various conditions and moment assumptions.
result High probability excess risk bounds for empirical risk minimizer, matching asymptotic rates.
In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization. To the best of our knowledg…
New learning algorithm for real analytic functions without gradient descent.
problem Learning real analytic functions without gradient descent.
method Taylor approximation and sampling data distribution.
result Nonuniform learning result for real analytic functions.
New method estimates Schrödinger bridge potentials via empirical risk minimization.
problem Estimating Schrödinger bridge potentials from samples.
method Rewriting Schrödinger system as a fixed-point equation and estimating the potential via empirical risk minimization.
result Uniform concentration of empirical risk around population counterpart under sub-Gaussian assumptions.
Prove non-asymptotic bounds for minimal risk in statistical learning
problem Estimating minimal risk in statistical learning
method Using concentration inequalities
result Non-asymptotic bounds for minimal risk
Paper shows robust estimators converge to true risk minimizers at optimal rates.
problem Understanding asymptotic properties of robust risk minimizers.
method Investigates robust analogues of empirical risk minimization, focusing on median of means estimator.
result Robust minimizers converge to true minimizers at optimal rates and have similar asymptotic variance.
Paper tackles heavy-tailed data without finite variance, proposing robust risk minimization.
problem Empirical risk minimization under heavy-tailed data with finite p-th moment. method Minimizes risk values robustly estimated via Catoni's method, using generalized generic chaining.
result Shows better performance of optimizer based on empirical risks via Catoni-style estimation.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
A new DP algorithm for weighted ERM protects sensitive data in predictive models.
problem Protecting sensitive personal information in predictive models trained via ERM.
method Proposes the first differentially private algorithm for weighted ERM with formal privacy guarantees.
result Demonstrates strong DP guarantees while maintaining robust performance in real-world data.
STORM enables edge computing for empirical risk minimization.
problem Training models on edge devices for streaming data.
method Online sketching for empirical risk minimization.
result STORM can estimate least-squares objective accurately.
Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.
problem Convergence rate of empirical minimizer for unbounded functions with fast growth.
method Analyzes L1-distance convergence rate of the empiric minimizer for coercive functions sampled with noise. result Convergence rate is bounded above by ann−1/q, where q is the dimension and an=o(nε) for every ε>0. Sharp bounds on ERM's minimal error in regression.
problem Understanding ERM's performance in regression tasks.
method Sharp lower bounds for ERM in random and fixed design settings.
result ERM's performance depends on the global or local complexity of the model.
The paper evaluates biased methods for alpha-divergence minimization.
problem The impact of bias on solutions found for alpha-divergence minimization.
method Empirical evaluation of biased methods for alpha-divergence minimization, focusing on bias effects and dimensionality.
result Solutions are biased towards KL-divergence minimizers and require impractical computation in high dimensions to minimize alpha-divergence.
Recently theoretical guarantees have been obtained for matrix completion in the non-uniform sampling regime. In particular, if the sampling distribution aligns with the underlying matrix's leverage scores, then with high probability nuclear norm minimization will exactly recover the low rank matrix. In this article, we…
We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framewo…
New regularization method reduces support of empirical risk minimization solutions.
problem Regularization in empirical risk minimization with relative entropy.
method Introduces Type-II regularization, characterizes solutions, analyzes properties of relative entropy.
result Type-II regularization collapses solution support into reference measure's support.
We consider distributed convex optimization problems originated from sample average approximation of stochastic optimization, or empirical risk minimization in machine learning. We assume that each machine in the distributed computing system has access to a local empirical loss function, constructed with i.i.d. data sa…
ERM with f-divergence regularization yields unique solution.
problem Optimizing empirical risk with f-divergence. method Mild conditions on f lead to unique optimal measure. result Equivalence of ERM-fDR to different f-divergence regularization. Optimizes bilevel empirical risk minimization with improved oracle calls.
problem Optimizing bilevel empirical risk minimization problems.
method Proposes a bilevel extension of the SARAH algorithm.
result Demonstrates improved oracle calls to achieve stationarity.
Improved multi-group learning with group-realizable concepts.
problem Enhancing multi-group learning efficiency.
method Empirical risk minimization over group-realizable concepts.
result Improved sample complexity in group-realizable settings.
Noise-ignorant empirical risk minimization achieves state-of-the-art performance on noisy data.
problem Learning with noisy labels in multi-class classification problems.
method Introducing relative signal strength (RSS) to quantify transferability and applying Noise Ignorant Empirical Risk Minimization (NI-ERM).
result NI-ERM achieves state-of-the-art performance on CIFAR-N data challenge.
Dual averaging-type methods are widely used in industrial machine learning applications due to their ability to promoting solution structure (e.g., sparsity) efficiently. In this paper, we propose a novel accelerated dual-averaging primal-dual algorithm for minimizing a composite convex function. We also derive a stoch…
We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empiric…
New approach avoids excess empirical risk in domain generalization.
problem Learning models that generalize to unseen distributions from diverse data sets.
method Minimizes penalty under constraint of optimal empirical risk, leveraging rate-distortion theory.
result Significant improvements in domain generalization performance across multiple methods.
The paper proves deep learning can be robust with certain loss functions.
problem The robustness of deep learning models under flawed data.
method Empirical-risk minimization with unbounded, Lipschitz-continuous loss functions.
result These loss functions provide efficient prediction under minimal data assumptions.
Online TERM improves robustness and fairness in streaming data.
problem Streaming data's lack of worst-case fairness and robustness in ERM.
method Proposes an online TERM formulation to balance average-case accuracy with worst-case fairness and robustness.
result Negative tilting effectively suppresses outlier influence, positive tilting improves recall with minimal precision loss.
Analyzes empirical risk minimization in finance, showing effectiveness and generalization issues.
problem Analyzing empirical risk minimization in finance for optimal hedging and investment decisions.
method Classical statistical machine learning techniques and non-asymptotic estimates based on Rademacher complexity.
result Over-training leads to anticipative decisions, but non-asymptotic estimates show convergence for large training sets.
Empirical study of IRMv1, an invariant risk minimization framework.
problem Learning predictors invariant to spurious correlations across different training environments.
method Extending ColoredMNIST experiment to various settings.
result IRMv1 performs better as spurious correlation varies more widely.
The paper analyzes local minima in high-dimensional empirical risk minimization.
problem Understanding local minima in high-dimensional data models.
method Using Kac-Rice formula and proportional asymptotics, the paper derives bounds on local minima.
result Sharp asymptotics on estimation and prediction errors are derived.
We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss. In contrast, we develop a co…
New method uses robust estimators for Newton's method in empirical risk minimization.
problem Improving robustness in empirical risk minimization.
method Robust Newton's method with gradient and Hessian replaced by robust estimators.
result Faster convergence rates in high-dimensional settings.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
New algorithms achieve uniform stability for empirical risk minimization.
problem Designing uniformly stable optimization algorithms for empirical risk minimization.
method Black-box conversion of smooth optimization algorithms and development of Mirror Descent for smooth optimization.
result Optimal algorithms with uniform stability and convergence rates for smooth optimization.
We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension. For common eigendecay patterns of the population covariance matrix, this quantity is significantly smaller than the ambient dimension. Our results reveal interesting connections…
Sharp-MAML improves MAML by reducing saddle points in few-shot learning.
problem Challenges in optimizing MAML due to complex loss landscape.
method Sharpness-aware minimization applied to MAML.
result Sharp-MAML and its variant outperform plain MAML on few-shot learning tasks.
Reweighting improves risk bounds in certain data regions.
problem Improving risk bounds in classification and heteroscedastic regression.
method Weighted empirical risk minimization with a data-dependent weight function.
result A weighted ERM estimator can achieve superior performance in specific sub-regions.
We consider statistical learning problems, when the distribution P′ of the training observations Z1′,…,Zn′ differs from the distribution P involved in the risk one seeks to minimize (referred to as the test distribution) but is still defined on the same measurable space as P and dominates it. In th…
We study in this paper the consequences of using the Mean Absolute Percentage Error (MAPE) as a measure of quality for regression models. We show that finding the best model under the MAPE is equivalent to doing weighted Mean Absolute Error (MAE) regression. We also show that, under some asumptions, universal consisten…
Mirror descent algorithm recovers low-rank matrices in matrix sensing.
problem Matrix sensing with low-rank matrices under certain conditions.
method Discrete-time mirror descent applied to empirical risk with Bregman divergence analysis.
result Mirror descent converges to a matrix minimizing a specific nuclear norm-related quantity.
New algorithms delete user data from machine learning models efficiently.
problem Deleting user data from machine learning models trained with empirical risk minimization.
method Developed an online unlearning algorithm using the infinitesimal jackknife, targeting non-smooth regularizers.
result Empirically improved runtime while maintaining memory requirements and test accuracy.
Paper presents ERM with f-divergence regularization and its properties.
problem Minimizing empirical risk with f-divergence constraints. method Introduces normalization function and solves ERM-fDR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.
Improved privacy and utility in machine learning with adaptive differential privacy.
problem Enhancing privacy in machine learning models while maintaining utility.
method Adaptive differentially private (ADP) learning method that optimally adapts noise to stepsize.
result ADP method significantly improves utility compared to standard differentially private methods.
We consider a generic convex optimization problem associated with regularized empirical risk minimization of linear predictors. The problem structure allows us to reformulate it as a convex-concave saddle point problem. We propose a stochastic primal-dual coordinate (SPDC) method, which alternates between maximizing ov…
In stochastic optimization, the population risk is generally approximated by the empirical risk. However, in the large-scale setting, minimization of the empirical risk may be computationally restrictive. In this paper, we design an efficient algorithm to approximate the population risk minimizer in generalized linear …
Differentially private method for estimating individualized treatment rules.
problem Estimating individualized treatment rules while preserving privacy.
method Differentially private two-stage empirical risk minimization (DP-2ERM).
result Improved privacy-utility trade-off demonstrated through simulations and applications.
A new method sorts models to find the best one with minimal risk.
problem Finding the best model with minimal risk among nested models.
method Nested Empirical Risk (NER) and Sorted NER (S-NER) methods.
result The S-NER method selects the true model order and the most parsimonious model.