Study on tradeoffs between mistakes and ERM oracle calls in online and transductive learning.
problem Analyzing online and transductive learning with limited ERM and weak consistency oracle access.
method Proves lower bounds and upper bounds on mistakes and oracle calls, considering realizable and agnostic cases.
result Achieves optimal mistake bounds with weak consistency queries for certain concept classes.
Algorithm solves online binary classification and infinite games using ERM oracle.
problem Online learning and solving infinite games with computationally inefficient oracles.
method Proposes an algorithm relying solely on ERM oracle calls for online binary classification and nonparametric games.
result Achieves finite and sublinearly growing regret in various settings.
New algorithm learns efficiently with a simple 'yes/no' oracle.
problem Can efficient learning be achieved with a simpler oracle than ERM?
method Developed an oracle that returns 'yes' or 'no' for realizable datasets.
result Learnability is possible with a polynomial price in VC dimension.
New algorithm reduces ERM problem size while maintaining accuracy.
problem Empirical risk minimization problem size reduction.
method Adaptive Deterministic Uniform-Weight Trimming (ADUWT) algorithm.
result Uniform ( 1 ± ε ) (1\pm\varepsilon) ( 1 ± ε ) relative-error approximation for ERM objective. New algorithm improves gradient-based ERM for smooth convex losses.
problem Empirical risk minimization of smooth, strongly convex loss functions.
method Iterative gradient-based method with local polynomial regression.
result Oracle complexity of O ( ( p ε − 1 ) d / ( 2 η ) ) O((p ε^{-1})^{d/(2η)}) O (( p ε − 1 ) d / ( 2 η ) ) for our algorithm. Accelerates ERM problems with LPI-GD and improved oracle complexity.
problem Empirical Risk Minimization (ERM) problems with strong convexity and smoothness.
method Local Polynomial Interpolation-based Gradient Descent (LPI-GD) and accelerated methods.
result Oracle complexity improved to $ ilde{O}\left(\sqrtσ m^d \log(1/\varepsilon)
ight)$ .
A new federated learning algorithm improves on existing methods by exploiting data smoothness.
problem Federated learning optimization with smooth loss functions.
method Federated Low Rank Gradient Descent (FedLRGD) algorithm.
result FedLRGD outperforms Federated Averaging (FedAve) in federated oracle complexity under certain conditions.
Piecewise polynomial interpolation-based gradient descent reduces oracle complexity for smooth loss functions.
problem Optimizing empirical risk minimization loss functions
method Piecewise polynomial interpolation-based gradient descent
result Oracle complexity is reduced for smooth loss functions
ERM performs well in feature learning with minimal feature maps.
problem Empirical risk minimization in feature learning with square loss.
method Asymptotic and non-asymptotic analysis of ERM performance.
result Excess risk quantiles of ERM match those of oracle procedure under certain conditions.
We present efficient algorithms for the problem of contextual bandits with i.i.d. covariates, an arbitrary sequence of rewards, and an arbitrary class of policies. Our algorithm BISTRO requires d calls to the empirical risk minimization (ERM) oracle per round, where d is the number of actions. The method uses unlabeled…
Paper addresses hybrid learning with constrained adversaries, achieving optimal performance.
problem Hybrid learning problem with i.i.d. features and adversarial labels.
method Structured adversarial setting, efficient algorithm with ERM oracle.
result Oracle-efficient algorithm with regret scaling with Rademacher complexity.
New method improves transductive learning predictions with multiplicative oracle inequalities.
problem Improving transductive learning predictions with known covariates.
method Median of Level-Set Aggregation (MLSA) for transductive LOO prediction.
result Proved multiplicative oracle inequality for LOO error.
The study provides theoretical guarantees for the statistical performance of optimal decision trees.
problem Theoretical limits on the statistical performance of globally optimal decision trees.
method Sharp oracle inequalities and uniform concentration framework based on Rademacher complexity.
result Derivation of minimax optimal rates for piecewise sparse heterogeneous anisotropic Besov space.
Study iterated ERM in active learning, deriving test error bounds.
problem Statistical dependencies in active learning with reused data.
method Sharp asymptotic analysis of test error for linear models.
result Double-descent behavior in test error due to data selection.
This guide simplifies high-probability regret bounds in empirical risk minimization.
problem High-probability regret bounds in empirical risk minimization.
method Modular presentation, three-step recipe, localized Rademacher complexity, local maximal inequalities, metric-entropy integrals.
result Recover familiar rates for various function classes and derive regret bounds for nuisance components.
New algorithm reduces online learning error for unknown feature distributions.
problem Oracle-efficient hybrid online learning with unknown feature and label distributions.
method Computational efficient online predictor using ERM oracle for finite-VC and fat-shattering classes.
result Oracle-efficient sublinear regret bounds for hybrid online learning with unknown feature generation.
This paper investigates the supervised learning problem with observations drawn from certain general stationary stochastic processes. Here by \emph{general}, we mean that many stationary stochastic processes can be included. We show that when the stochastic processes satisfy a generalized Bernstein-type inequality, a u…
We develop a new active learning algorithm for the streaming setting satisfying three important properties: 1) It provably works for any classifier representation and classification problem including those with severe noise. 2) It is efficiently implementable with an ERM oracle. 3) It is more aggressive than all previo…
One of the most effective algorithms for differentially private learning and optimization is objective perturbation. This technique augments a given optimization problem (e.g. deriving from an ERM problem) with a random linear term, and then exactly solves it. However, to date, analyses of this approach crucially rely …
Study robust linear regression with outliers, providing exact asymptotics for ERM performance.
problem Robust linear regression in high-dimension with outliers.
method Analyzes ℓ 2 \ell_2 ℓ 2 , ℓ 1 \ell_1 ℓ 1 , and Huber losses, providing asymptotic performance metrics. result Optimally-regularised ERM is asymptotically consistent with simple calibration, but Huber loss requires norm calibration.
NAPP-ERM improves ERM with differential privacy guarantees by iteratively achieving target regularization and delivering strong convexity.
problem Over-regularization in privacy-preserving ERM approaches.
method Noise-Augmented Privacy-Preserving Empirical Risk Minimization (NAPP-ERM) with a dual-purpose l2 regularizer and privacy budget retrieval strategy.
result Mitigates over-regularization and achieves strong convexity through a single regularizer.
Improved agnostic boosting with better sample efficiency.
problem Agnostic boosting's poor sample efficiency compared to Empirical Risk Minimization.
method Leverages sample reuse across rounds, guarantees better generalization.
result Substantially more sample-efficient agnostic boosting algorithm.
Logit correction improves model performance by correcting spurious correlations.
problem Spurious correlations lead to poor model performance during inference.
method Proposes logit correction (LC) loss to mitigate spurious correlations.
result LC loss outperforms state-of-the-art solutions by 5.5% absolute improvement.
Entropy asymmetry affects regularization in ERM, leading to biased solutions.
problem Analyzing the impact of relative entropy asymmetry in ERM regularization.
method Examined Type-I and Type-II ERM-RER, comparing their solutions and properties.
result Type-II ERM-RER regularization introduces a strong bias against training data.
Simple DP algorithms find approximate solutions for nonconvex ERM.
problem Finding approximate solutions to nonconvex ERM problems with privacy.
method Differential privacy, descent directions, line search, mini-batching, two-phase strategy.
result Effective algorithms for nonconvex ERM with privacy guarantees.
Study on Privileged ERM showing limitations and providing capacity analysis.
problem Improving classification accuracy with privileged information.
method Theoretical analysis of Privileged ERM using VC dimension and generalization bounds.
result Worst-case guarantees for Privileged ERM cannot improve over standard ERM unless privileged information capacity is similar or smaller.
New dual formulation reduces generalization error for ERM-fDR.
problem Generalization error in constrained optimization problems.
method Introduces a dual formulation of ERM-fDR using Legendre-Fenchel transform and implicit function theorem.
result Explicit characterizations of generalization error for algorithms under mild conditions.
Paper shows ERM's suboptimality due to bias, not variance.
problem Understanding why ERM fails to achieve optimal rates.
method Probabilistic and admissibility proofs for ERM in various settings.
result ERM's suboptimality is due to bias, not variance.
While many solutions for privacy-preserving convex empirical risk minimization (ERM) have been developed, privacy-preserving nonconvex ERM remains a challenge. We study nonconvex ERM, which takes the form of minimizing a finite-sum of nonconvex loss functions over a training set. We propose a new differentially private…
Empirical risk minimization (ERM) is ubiquitous in machine learning and underlies most supervised learning methods. While there has been a large body of work on algorithms for various ERM problems, the exact computational complexity of ERM is still not understood. We address this issue for multiple popular ERM problems…
Paper proposes a new method for SP with covariates using PADR and ERM.
problem Stochastic programming with covariate information.
method Empirical risk minimization (ERM) with nonconvex piecewise affine decision rules (PADR).
result The method provides theoretical consistency and computational tractability for nonconvex SP problems.
This paper studies universal rates of ERM for binary classification under agnostic learning.
problem The challenge of achieving universal rates of ERM for binary classification under agnostic learning.
method The paper explores the agnostic universal rates of ERM for binary classification, revealing three possible rates: e − n e^{-n} e − n , o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) , or arbitrarily slow. result The paper provides a complete characterization of which concept classes fall into each of the three categories of agnostic universal rates.
Default-ERM shortcut learning persists even without additional information.
problem Default-ERM shortcut learning in perception tasks despite stable feature sufficiency.
method Studied linear perception task; developed margin control (MARG-CTRL) loss functions.
result Margin control mitigates shortcut learning on various tasks.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
Paper presents ERM with f f f -divergence regularization and its properties.
problem Minimizing empirical risk with f f f -divergence constraints. method Introduces normalization function and solves ERM- f f f DR via ODE. result Characterizes difference between empirical risks and provides numerical algorithm.
Develops a new learning framework for dynamic data.
problem Poor performance of existing strategies in dynamic data and goals.
method Prospective Learning framework and Prospective ERM algorithm.
result Prospective ERM converges to Bayes risk under certain assumptions.
ERM struggles with synthetic data, but some algorithms can still learn correctly.
problem Learning from a mix of natural and synthetic data.
method Modeling the scenario as a sequence of learning tasks with oblivious algorithms, studying ERM and its limitations.
result ERM converges to the true mean but is outperformed by weighted algorithms. ERM does not always converge in the PAC setting, but there are algorithms capable of learning the correct hypothesis.
Simpler majority vote of three classifiers achieves optimal error bounds.
problem Developing an optimal PAC learning algorithm in the realizable setting.
method Returning the majority vote of three ERM classifiers.
result Achieves optimal in-expectation bound on error.
LOO prediction method improves generalization guarantees for arbitrary datasets.
problem Understanding LOO error guarantees in fully transductive settings for arbitrary datasets.
method Median of Level-Set Aggregation (MLSA) for empirical-risk level sets.
result Multiplicative oracle inequality for LOO error with complexity scaling.
Near-optimal algorithms for predicting across multiple loss functions efficiently.
problem Predicting optimally across various loss functions simultaneously.
method Developed near-optimal online and offline learning algorithms for omniprediction.
result Achieved near-optimal complexity for both online and offline settings.
Paper explores universal rates of ERM in machine learning.
problem Understanding universal learning rates for ERM.
method Analyzes realizable concept classes and ERM principles.
result Four possible universal learning rates by ERM.
Upper bounds and lower bounds show ERM outperforms DG methods in various settings.
problem Limitations of domain generalisation methods in various settings.
method Upper bounds and lower bounds on excess risk of ERM, and analysis of DG settings.
result It is not possible to significantly outperform ERM in DG settings.
Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.
problem Limited theoretical analysis for distributed ERM with general loss functions and hypothesis spaces.
method Derive tight risk bounds under assumptions on hypothesis space and loss function.
result Developed more general risk bound for distributed ERM without strong convexity restriction.
ERM with square loss achieves sublinear error for learnable function classes with smoothed data.
problem Statistical and computational hardness in sequential decision-making.
method Empirical Risk Minimization (ERM) with square loss, focusing on unknown base measure and smooth data.
result ERM achieves error scaling as i l d e O ( c o m p ( F ) ⋅ T ) ilde O( \sqrt{\mathrm{comp}(\mathcal F)\cdot T} ) i l d e O ( comp ( F ) ⋅ T ) for learnable function classes. Sharp bounds on ERM's minimal error in regression.
problem Understanding ERM's performance in regression tasks.
method Sharp lower bounds for ERM in random and fixed design settings.
result ERM's performance depends on the global or local complexity of the model.
New approach to robustly reliable learners against instance-targeted attacks.
problem Addressing data poisoning attacks on robustly reliable learners.
method Defining regularized robustly reliable learners and efficient algorithms.
result Efficient algorithms for robustly reliable learners with sublinear runtime.
We propose a new family of fairness definitions for classification problems that combine some of the best properties of both statistical and individual notions of fairness. We posit not only a distribution over individuals, but also a distribution over (or collection of) classification tasks. We then ask that standard …
FeAT improves OOD generalization by learning richer features.
problem Improving feature learning for out-of-distribution (OOD) generalization.
method Feature Augmented Training (FeAT) iteratively augments and retains features from different subsets of training data.
result FeAT effectively learns richer features, boosting OOD performance.