A new method approximates expected empirical loss for stochastic deep learning tasks.
problem Determining optimal step sizes for stochastic gradient descent in deep learning.
method Applying one-dimensional function fitting to noisy losses of vertical cross sections to approximate expected empirical loss.
result The method leads to a robust and straightforward optimization method that performs well across datasets and architectures.
The paper proves deep learning can be robust with certain loss functions.
problem The robustness of deep learning models under flawed data.
method Empirical-risk minimization with unbounded, Lipschitz-continuous loss functions.
result These loss functions provide efficient prediction under minimal data assumptions.
A new loss function α α α -loss bridges log-loss and 0 0 0 - 1 1 1 loss for binary classification.
problem Improving binary classification performance using a tunable loss function.
method Introducing α α α -loss, proving its margin-based form and classification-calibration, and providing an upper bound on empirical risk. result Empirical and expected risk difference upper bound for logistic regression-based classification.
We consider distributed convex optimization problems originated from sample average approximation of stochastic optimization, or empirical risk minimization in machine learning. We assume that each machine in the distributed computing system has access to a local empirical loss function, constructed with i.i.d. data sa…
A new SGD framework reduces empirical risk by favoring higher loss observations.
problem Minimizing empirical risk in machine learning problems.
method Develops a biased gradient estimator for stochastic optimization.
result Minimizes an ordered modification of the empirical average loss.
CcGAN tackles conditional image generation for continuous labels.
problem Mathematical challenges in conditioning on continuous, scalar labels.
method Proposes novel empirical losses and label input methods for continuous conditional GANs.
result CcGAN generates diverse, high-quality images from continuous labels.
Simplified screening tests for data points in optimization.
problem Discarding irrelevant data points in empirical risk minimization.
method Designing loss functions and regularizing convex losses to induce sparsity, using ellipsoidal approximations.
result Automatic discarding of data samples without losing optimization guarantees.
Paper studies convergence rates from surrogate risk minimizers to Bayes optimal classifier.
problem Analyzing the convergence rates of surrogate risk minimizers to the Bayes optimal classifier.
method Introducing consistency intensity to characterize surrogate loss functions and using it to derive convergence rates.
result Empirical surrogate risk minimizers converge faster to the Bayes optimal classifier under certain conditions.
New loss function calibrates WW-hinge loss for multiclass SVM.
problem WW-hinge loss not calibrated with 0-1 loss.
method Introduced ordered partition loss and proved WW-hinge loss is calibrated.
result WW-hinge loss is calibrated with ordered partition loss.
Paper proposes a new e e e -exponentiated transformation to make convex loss functions more robust to outliers.
problem Making convex loss functions robust to outliers in the presence of label noise.
method Introduces a novel e e e -exponentiated transformation for loss functions and proves its effectiveness through theoretical and empirical analysis. result The transformed loss function achieves tighter generalization error bounds and higher accuracy in noisy datasets.
Private algorithms minimize population loss with optimal rate.
problem Private optimization of convex functions with stochastic samples.
method Differentially private algorithms based on algorithmic stability.
result Optimal rate of 1 / n 1/\sqrt{n} 1/ n for population loss. New regularization techniques using mass transportation for better generalization.
problem Mitigating overfitting with scarce data.
method Distributionally robust optimization and worst-case expected loss.
result Generalization bounds and tractable learning problems.
Paper improves privacy in ERM with faster algorithms and broader applicability.
problem Privacy-preserving machine learning with empirical risk minimization.
method Develops faster algorithms for differentially private ERM in various settings.
result Achieves optimal or near-optimal utility bounds with less gradient complexity.
This paper analyzes output activation functions for adversarial losses.
problem Understanding which output activation functions form a well-behaved adversarial loss.
method Variational divergence minimization and a comparative framework for adversarial losses.
result There is no single winning combination of output activation functions and regularization approaches across all settings.
DLM for BNNs fails to improve over ELBO optimization.
problem Performance of DLM for Bayesian Neural Networks (BNNs).
method Direct Loss Minimization (DLM) compared to ELBO optimization.
result DLM does not significantly improve over ELBO optimization for BNNs.
Deep networks can memorize random labels; symmetric loss improves this.
problem Deep networks can memorize random labels, ignoring standard regularization.
method Empirical studies with MNIST and CIFAR-10 datasets, formal definition of robustness.
result Symmetric loss function improves network's ability to resist memorization.
The paper studies the landscape of empirical risk for non-convex losses.
problem Understanding the complexity of non-convex losses in high-dimensional estimation.
method Analyzing the landscape of the empirical risk, focusing on stationary points and their properties.
result Uniform convergence of the gradient and Hessian of the empirical risk to their population counterparts, ensuring good properties of the population risk can be carried to the empirical risk.
Non-parametric method for estimating aggregate loss distribution using empirical characteristic functions.
problem Estimating aggregate loss distribution in insurance applications.
method Combining empirical characteristic functions of frequency and severity, numerically inverting them, and applying the Gil-Pelaez formula.
result Efficient method for evaluating aggregate loss distribution and related measures like VaR.
Novel convex surrogate for non-modular loss functions.
problem Computational tractability for non-modular loss functions.
method Submodular-supermodular decomposition, slack-rescaling, Lov{á}sz hinge.
result First tractable solution for non-modular loss functions.
Paper provides uniform deviation bounds for unbounded loss functions, improving k-Means clustering bounds.
problem Uniform deviation bounds for unbounded loss functions, specifically k-Means clustering.
method Novel framework to obtain uniform deviation bounds for unbounded loss functions.
result Improved bounds for k-Means clustering under weak assumptions, achieving $\mathcal{O}\left(m^{-\frac12}
ight)$ rate.
Proposes a cross entropy loss for better ranking algorithms.
problem Improving the theoretical understanding and performance of ranking algorithms.
method Introduces a cross entropy-based loss function that is a convex bound on NDCG and consistent with NDCG.
result Empirically, the proposed method outperforms existing algorithms in quality and robustness.
Optimizes exp-concave losses with a new risk bound.
problem Optimizing exp-concave losses with stochastic convex optimization.
method Empirical Risk Minimization with a unified geometric assumption and local norms.
result Provides an O ( d / n + log ( 1 / δ ) / n ) O( d / n + \log( 1 / δ) / n ) O ( d / n + log ( 1/ δ ) / n ) excess risk bound. Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
Visualizes basins of attraction for neural network loss functions.
problem Understanding the nature of neural network loss surfaces and basins of attraction.
method Gradient-based random sampling to visualize basins of attraction and stationary points.
result Entropic loss has a more searchable landscape with fewer stationary points than quadratic loss.
New empirical Bayes estimator outperforms soft-thresholding for high-dimensional sparse vectors.
problem Estimating high-dimensional sparse vectors from noisy observations.
method Empirical Bayes shrinkage estimator using a Bernoulli-Gaussian prior.
result Hybrid estimator outperforms soft-thresholding in compressed sensing applications.
Clarifies model-based RL's theoretical issues and counterexamples for popular losses.
problem Model-based reinforcement learning's empirical performance vs. theoretical properties and popular loss functions.
method Analyzes empirical and theoretical aspects of model-based RL and constructs counterexamples for losses.
result MuZero loss fails in stochastic and deterministic environments, leading to exponential sample complexity.
Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.
problem Limited theoretical analysis for distributed ERM with general loss functions and hypothesis spaces.
method Derive tight risk bounds under assumptions on hypothesis space and loss function.
result Developed more general risk bound for distributed ERM without strong convexity restriction.
Paper shows non-convex loss functions can be optimized efficiently.
problem Optimizing non-convex loss functions is challenging.
method Uses stochastic variance reduction methods to find global optimal solutions.
result Stochastic variance reduction methods converge to global optimal with linear rate.
The article analyzes high-dimensional classification using empirical risk minimization with precise error predictions.
problem Classifying high-dimensional data with Gaussian mixture models.
method Theoretical analysis of ridge-regularized and unregularized empirical risk minimization for high-dimensional Gaussian mixture separation.
result The square loss is optimal for high-dimensional classification in both ridge-regularized and unregularized cases.
Optimizes hybrid insurance contracts for heavy-tailed losses.
problem Providing insurance against heavy-tailed losses with finite expected loss.
method Combines traditional and parametric insurance, using a Pareto-type criterion for optimization.
result The hybrid contract outperforms traditional contracts in simulations and real data.
The paper analyzes and proposes methods for privately sharing individual privacy losses using per-instance differential privacy.
problem The standard differential privacy framework provides a worst-case bound that may not accurately reflect individual privacy losses.
method The paper analyzes per-instance differential privacy and proposes methods to privately and accurately publish per-instance privacy losses.
result The methods privately and accurately publish per-instance differential privacy losses with minimal additional privacy cost.
Study analyzes landscape complexity of empirical loss functions with correlated data.
problem Understanding the complexity of loss landscapes in machine learning with structured data.
method Kac-Rice formula and random matrix theory applied to high-dimensional empirical loss functions.
result Characterizes the average number of critical points in loss functions with structured data.
EnsLoss combines multiple loss functions to prevent overfitting in classification.
problem Preventing overfitting in classification models.
method EnsLoss is an ensemble method that combines loss functions, ensuring calibration and consistency.
result EnsLoss improves classification accuracy compared to fixed loss methods.
Study on using random subspaces for ERM with various loss functions.
problem Improving learning accuracy with computational savings from random subspaces.
method Random subspaces of a hypothesis space, considering data-dependent subspaces.
result Unified analysis showing computational efficiency can be improved without performance loss.
The Nyström method improves learning efficiency for convex losses.
problem Improving computational efficiency in empirical risk minimization.
method Using random subspaces to approximate hypothesis spaces in convex loss functions.
result Computational gains can be achieved without sacrificing learning performance for general convex Lipschitz losses.
Study on estimating class probabilities using empirical risk minimization.
problem Estimating class probabilities within binary classification.
method Empirical risk minimization (ERM) for class probability estimation.
result The estimator converges to true class probabilities under certain conditions.
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
Deep neural networks' loss surfaces contain every low-dimensional pattern.
problem Finding arbitrary low-dimensional patterns in neural network loss surfaces.
method Empirical and theoretical analysis of loss landscapes of deep neural networks.
result Deep universal approximators exhibit a property where arbitrary smooth patterns exist in their loss surfaces.
Paper analyzes high-dimensional portfolio risks and finds empirical out-of-sample relative loss is more reliable.
problem Analyzing risks in high-dimensional portfolios using empirical variance.
method Derives asymptotic behavior of out-of-sample variance and relative loss in high-dimensional settings.
result Empirical out-of-sample relative loss is more reliable than variance in high-dimensional portfolios.
Peer loss functions learn from noisy labels without noise rate specification.
problem Learning from noisy labels without knowing noise rates.
method Introduced peer loss functions within ERM framework.
result Peer loss functions lead to optimal or near-optimal classifiers.
New algorithm optimizes PAC-Bayes bound without surrogate loss.
problem Mismatch between optimisation objective and generalisation bound in stochastic neural networks.
method Proposes a novel training algorithm that optimizes the PAC-Bayesian bound directly.
result Empirical results show improved performance over existing PAC-Bayesian training methods.
Develops a new framework for robust regression with EGM.
problem Addressing robust regression with heavy-tailed noise or outliers.
method Empirical gain maximization (EGM) to approximate noise density.
result Unified analysis of robust regression approaches.
Study on neural networks with quadratic activation functions, focusing on optimization and generalization.
problem Understanding the dynamics and generalization of neural networks with quadratic activation in the over-parametrized regime.
method Teacher-student scenario, empirical loss landscape analysis, gradient descent dynamics, numerical experiments.
result Conditions for the neural network to recover the teacher and achieve small generalization error.
Loss-guided training accelerates node embedding methods on graphs.
problem Training efficiency in graph learning methods with implicit positive examples.
method Dynamic adjustment of training distribution based on loss values.
result Significant acceleration in training and computation over static methods.
New algorithms and guarantees for multiple-source adaptation.
problem Improving model performance on target mixtures from multiple sources.
method Normalized solutions with theoretical guarantees, algorithms for distribution-weighted combination.
result Our algorithm outperforms competing approaches by producing a robust model.
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
Paper extends chaining technique for empirical risk minimization bounds.
problem Empirical risk minimization with unbounded noise and estimates.
method Chaining technique applied to random design settings, proving excess risk bounds.
result Proves upper bounds for empirical risk minimization with sub-Gaussian or subexponential noise.
A new method estimates the learning coefficient using empirical loss.
problem Estimating the learning coefficient in complex models.
method Numerical estimation using Empirical Loss.
result Our method outperforms existing techniques in terms of bias and variance.