We carefully study how well minimizing convex surrogate loss functions, corresponds to minimizing the misclassification error rate for the problem of binary classification with linear predictors. In particular, we show that amongst all convex surrogate losses, the hinge loss gives essentially the best possible bound, o…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “error loss”
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of -consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings.
result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
Optimal multiclass U-calibration error found to be Θ(√KT).
problem Online multiclass U-calibration with low regret for all bounded proper losses.
method Follow-the-Perturbed-Leader algorithm and lower bound construction.
result Optimal U-calibration error is Θ(√KT).
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.
We consider binary classification problems with positive definite kernels and square loss, and study the convergence rates of stochastic gradient methods. We show that while the excess testing loss (squared loss) converges slowly to zero as the number of observations (and thus iterations) goes to infinity, the testing …
New loss function improves accuracy of MRI parameter estimation.
problem Systematic errors in parameter estimates at low SNR.
method Developed and implemented negative log Rician likelihood (NLR) loss.
result NLR loss shows higher accuracy in parameter estimation than MSE loss at low SNR.
Robust Variable Selection for High-dimensional Regression with Missing Data and Measurement Errorsstat.ME
Robust variable selection for high-dimensional data with missing and measurement errors.
problem Missing data and measurement errors confound data distribution.
method Exponential loss function with inverse probability weighting and additive error models.
result The Atan punishment method improves robust variable selection.
This paper improves multi-label ranking by reweighting univariate losses, enhancing consistency and performance.
problem Improving multi-label ranking performance while maintaining consistency.
method Systematic study of consistency and generalization error bounds for learning algorithms, proposing a reweighted univariate loss.
result Inconsistent pairwise losses can lead to better performance than consistent univariate losses in practice.
A hybrid loss framework improves time series forecasting by balancing global and component errors.
problem Current time series methods may prioritize less significant sub-series, leading to forecasting bias.
method Proposes a hybrid loss framework combining global and component losses, dynamically adjusting weights.
result Improves time series forecasting performance by 0.5-2% on average.
New binary loss functions improve density ratio estimation accuracy.
problem Improving accuracy of density ratio estimators using binary classifiers.
method Characterized loss functions based on prescribed error measures in Bregman divergences.
result Novel loss functions prioritize accurate estimation of large density ratio values.
Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
Clarifies model-based RL's theoretical issues and counterexamples for popular losses.
problem Model-based reinforcement learning's empirical performance vs. theoretical properties and popular loss functions.
method Analyzes empirical and theoretical aspects of model-based RL and constructs counterexamples for losses.
result MuZero loss fails in stochastic and deterministic environments, leading to exponential sample complexity.
The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
Study shows how classifiers can approach Bayes error in high-dimensional settings.
problem Generalization error in high-dimensional perceptrons.
method Proved a formula for generalization error using convex optimization and observed that logistic and hinge regression can approach Bayes error closely.
result Logistic and hinge regression can approach Bayes-optimal generalization error closely in high-dimensional settings.
Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …
Study non-asymptotic bounds for robust estimators under misspecified models.
problem Evaluate performance of robust estimators under adversarial conditions.
method Propose a general approach to adversarial risk analysis, including investigations on generalization and approximation errors.
result Establish non-asymptotic upper bounds for adversarial excess risk under Lipschitz loss functions.
Adversarial training achieves optimal test error for shallow networks.
problem Achieving optimal adversarial test error for general data distributions.
method Applying new Rademacher complexity bounds and properties of optimal adversarial predictors.
result Adversarial training can achieve optimal adversarial test error for general data distributions.
Develops high-dimensional measurement error models for non-linear loss functions.
problem Measurement errors in ultrahigh-dimensional biomedical data.
method Lipschitz loss functions, L1 norm minimization, Lasso analog.
result Improved accuracy in classification and quantile regression problems.
Deep learning has become the method of choice in many application domains of machine learning in recent years, especially for multi-class classification tasks. The most common loss function used in this context is the cross-entropy loss, which reduces to the log loss in the typical case when there is a single correct r…
This paper compares deeper and wider neural networks for optimal generalization error in Sobolev losses.
problem The dilemma of choosing between deeper or wider neural networks for optimal generalization error.
method Analytical investigations into the influence of sample points, parameters, and loss function regularity on neural network architecture.
result A higher number of parameters favors wider neural networks, while more sample points and greater loss function regularity favor deeper neural networks.
This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.
problem Theoretical understanding of square loss in overparametrized neural networks.
method Systematic investigation of square loss in the NTK regime for both separable and non-separable classes.
result Square loss shows fast convergence rates and robustness guarantees for overparametrized neural networks.
This paper presents the hierarchical generalized linear model (HGLM) for loss reserving in a non-life insurance company. Because in this case the error of prediction is expressed by a complex analytical formula, the error bootstrap estimator is proposed instead. Moreover, the bootstrap procedure is used to obtain full …
In order to push the performance on realistic computer vision tasks, the number of classes in modern benchmark datasets has significantly increased in recent years. This increase in the number of classes comes along with increased ambiguity between the class labels, raising the question if top-1 error is the right perf…
Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…
New decision-theoretic calibration error metric improves prediction reliability.
problem Improving the reliability of predictions for decision-making.
method Proposed Calibration Decision Loss (CDL) and an efficient algorithm to achieve near-optimal CDL.
result Near-optimal CDL guarantees vanishing payoff loss from miscalibration.
Flooding prevents deep networks from achieving zero training loss, improving test performance.
problem Deep networks can achieve zero training error but often have zero training loss, leading to overconfidence and poor test performance.
method Flooding prevents training loss from reaching zero by applying gradient ascent when loss is below a preset flood level.
result Flooding improves test performance and induces a double descent curve of the test loss.
PACMAN provides bounds for classification tasks considering accuracy vs. negative log-loss mismatch.
problem Mismatch between accuracy and negative log-loss in classification tasks.
method Point-wise PAC approach over generalization gap, using likelihood ratio and concentration inequalities.
result PACMAN provides point-wise PAC bounds for the generalization problem.
In many scenarios of a language identification task, the user will specify a small set of languages which he/she can speak instead of a large set of all possible languages. We want to model such prior knowledge into the way we train our neural networks, by replacing the commonly used softmax loss function with a novel …
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to ind…
The paper explores MAE as a loss function for DNN vector-to-vector regression, proving its advantages over MSE.
problem Improving loss function for deep neural network based vector-to-vector regression.
method Presenting performance bounds and new properties of MAE, deriving generalized upper bounds, and interpreting MAE as a Laplacian distribution.
result MAE is a more suitable loss function than MSE for DNN based vector-to-vector regression, especially when errors follow a Laplacian distribution.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First -consistency bounds for comp-sum losses and smooth adversarial comp-sum losses.
New loss function equivalence reveals PER's uniform sampling can be improved.
problem Improving Prioritized Experience Replay (PER) for better learning efficiency.
method Transforming non-uniformly sampled data loss functions into uniformly sampled ones.
result Some environments can replace PER with a new loss function without performance loss.
We study prediction and estimation problems using empirical risk minimization, relative to a general convex loss function. We obtain sharp error rates even when concentration is false or is very restricted, for example, in heavy-tailed scenarios. Our results show that the error rate depends on two parameters: one captu…
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
Proposes a new loss function for learning with noisy labels.
problem Improving model learnability with noisy labels.
method Uses generalized Jensen-Shannon divergence as a noise-robust loss function.
result Shows state-of-the-art results on noisy data.
Sign information is the key to overcoming the inevitable saturation error in compressive sensing systems, which causes information loss and results in bias. For sparse signal recovery from saturation, we propose to use a linear loss to improve the effectiveness from existing methods that utilize hard constraints/hinge …
Finding hidden-feature depending laws inside a data set and classifying it using Neural Networkcs.LG
The logcosh loss function helps neural networks learn set-valued functions better.
problem Learning set-valued functions with neural networks.
method Using artificial neural networks with logcosh loss.
result Neural networks with logcosh loss can classify samples based on set-valued functions.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
New PAC-Bayes bound controls multiple error types simultaneously.
problem Current PAC-Bayes bounds are limited to scalar metrics.
method Bounding KL divergence between empirical and true probabilities of multiple error types.
result First PAC-Bayes bound for rich information-rich certificates.
Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points…
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
New bounds for neural networks without loss boundedness assumption.
problem Generalization error bounds for two-layer neural networks.
method Wasserstein distance estimates and moment bounds for stochastic gradient method.
result Dimension-free rate of order for independent test data.
Symmetric losses improve classifier robustness from corrupted labels.
problem Improving classifier performance from corrupted labels.
method Symmetric losses that satisfy a certain condition.
result Symmetric losses enhance robust classification from corrupted labels.
Study on -consistency bounds for machine learning surrogates.
problem Estimating target loss error relative to surrogate loss error in machine learning.
method Developed -consistency bounds for various surrogates and loss functions.
result Stronger guarantees than existing methods, offering distribution-dependent and -independent bounds.
New approach shapes error distribution in long-term forecasting.
problem Disparate error distributions in recent transformer models.
method Loss shaping constraints to respect upper bounds on loss at each time-step.
result Competitive average performance with shaped error distribution.
New method gives provable error bounds for neural nets under distribution shift.
problem Proving reliable error bounds for neural networks under distribution shift.
method Optimizing a classifier to disagree with another, using a new 'disagreement loss'.
result Valid error bounds with comparable accuracy to competitive methods.
Post-processing predictors reduces calibration errors for decision-making.
problem Predictors with low calibration error for machine learning may have high error for decision-making.
method Post-processing with ε distance to calibration adds noise to make predictions differentially private.
result Post-processing achieves O(√ε) ECE and CDL, asymptotically optimal.
Paper introduces new loss functions for multi-class abstention learning.
problem Learning with multi-class classification and the ability to abstain.
method Developed new families of surrogate losses for abstention.
result Proved strong consistency guarantees for new surrogate losses.