This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.
problem Theoretical understanding of square loss in overparametrized neural networks.
method Systematic investigation of square loss in the NTK regime for both separable and non-separable classes.
result Square loss shows fast convergence rates and robustness guarantees for overparametrized neural networks.
Exponential testing error reduction with stochastic gradient methods under low-noise conditions.
problem Binary classification with positive definite kernels and square loss.
method Stochastic gradient methods under low-noise conditions.
result Testing error converges exponentially fast, while testing loss converges slowly.
Global optimality conditions found for deep neural networks.
problem Understanding the error landscape of deep neural networks.
method Analyzing deep linear and nonlinear neural networks with squared error loss.
result Necessary and sufficient conditions for global optimality in deep linear networks, with an efficiently checkable test.
New algorithm improves regression error bounds and accelerates performance for low noise.
problem Nonparametric least square regression in RKHS with optimal error bounds.
method Kernel Truncated Randomized Ridge Regression (KTRRR) with optimal generalization error bounds.
result Faster finite-time and asymptotic rates on low noise problems.
The paper analyzes error bounds and KL properties for noisy matrix recovery problems.
problem Noisy low-rank matrix recovery problems.
method Squared F-norm regularization, accelerated alternating minimization method.
result Established error bounds and KL properties for critical points and global minimizers.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
Paper unifies bias and variance models for classification.
problem Different frameworks for bias and variance in classification.
method Unified Tumer & Ghosh and James approaches.
result Closed form relationships between 0/1 loss and squared error loss.
The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.
problem Understanding the loss landscape and minimizers of regularized deep matrix factorization problems.
method Theoretical analysis of ℓ2-regularized deep matrix factorization/deep linear network training problems with squared-error loss. result The unique end-to-end minimizer exists for all target matrices except for a set of Lebesgue measure zero.
This research finds that using mean-squared error and codeword targets improves adversarial robustness.
problem Evaluating and improving the robustness of neural networks against adversarial attacks.
method Training neural networks on mean-squared error and using codeword targets as representations.
result The modified models show up to 98.7% increase in accuracy against untargeted attacks and up to 99.8% decrease in targeted attack success rates.
New loss function improves accuracy of MRI parameter estimation.
problem Systematic errors in parameter estimates at low SNR.
method Developed and implemented negative log Rician likelihood (NLR) loss.
result NLR loss shows higher accuracy in parameter estimation than MSE loss at low SNR.
ALRC clips learning rates to stabilize neural network training.
problem Stability issues in training with small batch sizes and high learning rates.
method Adaptive learning rate clipping (ALRC) to limit losses.
result ALRC decreases errors in unstable training conditions.
Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
The logcosh loss function helps neural networks learn set-valued functions better.
problem Learning set-valued functions with neural networks.
method Using artificial neural networks with logcosh loss.
result Neural networks with logcosh loss can classify samples based on set-valued functions.
Transformer-based models overfit financial time series data, leading to increased prediction variance.
problem Forecast collapse of transformer-based models under squared loss in financial time series.
method Theoretical analysis and numerical experiments on high-frequency EUR/USD exchange rate data.
result Increased model expressivity in Transformer-based models leads to spurious fluctuations without reducing bias, resulting in higher prediction variance.
Improved speech enhancement using diffusion models with MSE loss.
problem Efficient incorporation of noisy speech in generative speech enhancement.
method Augmented diffusion-based generative model with a MSE loss for enhanced speech.
result Proposed method improves speech enhancement performance compared to original diffusion model.
We derive a mapping between MSE and CCC, revealing counterintuitive insights.
problem Missing mapping between mean square error and concordance correlation coefficient.
method Derive mathematical formula connecting MSE and CCC, analyze graphical implications.
result Formula uncovers counterintuitive insights and precise range for CCC given MSE.
Principal Component Analysis (PCA) is a very successful dimensionality reduction technique, widely used in predictive modeling. A key factor in its widespread use in this domain is the fact that the projection of a dataset onto its first K principal components minimizes the sum of squared errors between the original …
Cross-validation pitfalls in change-point regression are addressed with new approaches.
problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.
The paper analyzes cross-validation for correlated data and introduces a bias-corrected estimator.
problem Cross-validation with squared error loss assumes independent and identically distributed (i.i.d.) data, which is often violated in correlated data.
method The paper presents a criterion for standard CV suitability and introduces a bias-corrected estimator (CVc) for correlated data. result The bias-corrected estimator (CVc) yields an unbiased estimate of prediction error in settings where standard CV is invalid. This study explains gradient flow dynamics in neural networks for small initialisation.
problem Understanding the training dynamics of neural networks for small initialisation.
method Analysis of gradient flow dynamics for one-hidden layer ReLU networks with orthogonal inputs.
result Gradient flow converges to zero loss and characterizes implicit bias towards minimum variation norm.
New algorithm predicts tensor entries faster than best rank-r tensor.
problem Agnostic tensor completion with unknown rank.
method Sum-of-squares relaxation of tensor nuclear norm.
result Prediction error rate of O(r2d3/2/n), improving over matrix completion. 3D GAN improves MRI image quality from low-res scans.
problem Improving MRI image quality from low-resolution scans.
method Adversarial learning using 3D convolutions and least squares loss.
result 3D GAN generates high-quality 3D MRI images from low-resolution scans.
The Nyström method improves learning efficiency for convex losses.
problem Improving computational efficiency in empirical risk minimization.
method Using random subspaces to approximate hypothesis spaces in convex loss functions.
result Computational gains can be achieved without sacrificing learning performance for general convex Lipschitz losses.
Paper develops estimators for unbounded density ratios with applications in error control.
problem Estimating density ratios with unbounded domains and ranges.
method Least squares and logistic regression loss functions for density ratio estimation.
result Established upper bounds on estimation errors with optimal rates for unbounded density ratios.
Paper proposes a new loss function to improve image reconstruction quality.
problem Blurred images when using pixel loss for convolutional autoencoders.
method Introduces spatial frequency loss (SFL) to mitigate blurring.
result Reduced blurs in reconstructed images using SFL.
Deep networks often capture low frequency functions, improving generalization.
problem Understanding deep learning's generalization ability.
method Showed F-Principle holds for various loss functions and applied it to differential equations.
result Deep networks capture low frequency functions, leading to better generalization.
We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…
Super learner with Huber loss improves cost prediction and causal effect estimation in healthcare expenditure data.
problem Challenges in modeling healthcare expenditure distributions with standard super learning methods.
method Proposes a super learner using Huber loss, a robust loss function that down-weights outliers.
result Demonstrates appreciable finite-sample gains in cost prediction and causal effect estimation.
Analyzes double descent in binary classification models with different losses.
problem Understanding the double descent phenomenon in binary classification models.
method Analytic study of gradient descent with logistic and square losses on binary linear classification models.
result The double descent phenomenon persists but with differences compared to logistic loss.
The article analyzes high-dimensional classification using empirical risk minimization with precise error predictions.
problem Classifying high-dimensional data with Gaussian mixture models.
method Theoretical analysis of ridge-regularized and unregularized empirical risk minimization for high-dimensional Gaussian mixture separation.
result The square loss is optimal for high-dimensional classification in both ridge-regularized and unregularized cases.
In this paper, we explore ordinal classification (in the context of deep neural networks) through a simple modification of the squared error loss which not only allows it to not only be sensitive to class ordering, but also allows the possibility of having a discrete probability distribution over the classes. Our formu…
DEQGAN uses GANs to solve differential equations without supervision.
problem Solving differential equations with neural networks.
method Generative Adversarial Networks (GANs) to learn the loss function.
result DEQGAN achieves lower mean squared errors and competitive solution accuracy compared to traditional methods.
We prove a new generalization bound that shows for any class of linear predictors in Gaussian space, the Rademacher complexity of the class and the training error under any continuous loss ℓ can control the test error under all Moreau envelopes of the loss ℓ. We use our finite-sample bound to directly recover…
Two new algorithms improve Q* approximation in batch RL with linear error propagation.
problem Improving Q* approximation in batch reinforcement learning.
method Two novel algorithms that estimate Bellman error directly, without quadratic dependence.
result Linear-in-horizon error propagation for batch RL algorithms.
The paper explores MAE as a loss function for DNN vector-to-vector regression, proving its advantages over MSE.
problem Improving loss function for deep neural network based vector-to-vector regression.
method Presenting performance bounds and new properties of MAE, deriving generalized upper bounds, and interpreting MAE as a Laplacian distribution.
result MAE is a more suitable loss function than MSE for DNN based vector-to-vector regression, especially when errors follow a Laplacian distribution.
Mack's estimator improves chain ladder prediction for large exposure insurance models.
problem Uncertainty quantification in compound Poisson loss models.
method Large exposure asymptotics applied to Mack's estimator.
result Chain ladder prediction uncertainty can be quantified without model assumptions.
Cryptocurrency prices predicted using LSTM, SVM, and polynomial regression.
problem Uncertainty in crypto coin values.
method Long Short Term Memory, Support Vector Machine, Polynomial Regression models.
result Support Vector Machine with linear kernel had the smallest mean square error.
The paper analyzes generalization properties of scalable kernel methods.
problem Understanding the generalization of doubly stochastic learning algorithms.
method Theoretical analysis of different variants of doubly stochastic learning algorithms in nonparametric regression.
result Derivation of generalization error convergence results for the algorithms.
Hard to learn ReLU with Gaussian data, but can approximate efficiently.
problem Learning a ReLU with Gaussian marginals under arbitrary labels.
method Proved hardness and developed an efficient approximation algorithm.
result Efficient approximation algorithm for best-fitting ReLU with error O(opt2/3). Biased mean regression estimates factors exceeding expected loss or radiation release severity.
problem Estimating factors exceeding expected loss or radiation severity levels.
method Biased mean regression using superexpectation error minimization.
result Equivalent to quantile regression and CVaR optimization under specific conditions.
Convolutional neural network improves MRE image reconstruction.
problem Reconstructing MRE images from displacement data is computationally intensive and costly.
method Proposes a CNN architecture to directly map MRE displacement data into elastograms, introducing a secondary loss for training.
result CNN-generated images compare favorably with nonlinear inversion methods.
Proposes a new loss function for better super-resolution images.
problem Improving the quality of super-resolution images.
method Introduces a robust loss function based on edge preservation using the Canny operator.
result Enhanced performance in PSNR and SSIM metrics compared to MSE loss function.
Given a task of predicting Y from X, a loss function L, and a set of probability distributions Γ on (X,Y), what is the optimal decision rule minimizing the worst-case expected loss over Γ? In this paper, we address this question by introducing a generalization of the principle of maximum entropy. Applying t…
Least Squares Estimators are suboptimal for 5D convex functions.
problem Suboptimality of Least Squares Estimators in estimating multidimensional convex functions.
method Analysis of natural subclasses of convex functions in random and fixed design settings.
result Risk of LSE is n−2/d while minimax risk is n−4/(d+4) for d≥5. Visualizes basins of attraction for neural network loss functions.
problem Understanding the nature of neural network loss surfaces and basins of attraction.
method Gradient-based random sampling to visualize basins of attraction and stationary points.
result Entropic loss has a more searchable landscape with fewer stationary points than quadratic loss.
Chain-ladder reserving is sensitive to outliers, leading to unreliable estimates.
problem Sensitivity of loss reserving techniques to outliers.
method Derivation of impact functions for reserves and mean squared errors of prediction under Mack's Model.
result Impact of outliers varies widely in a loss triangle and depends on other cells.
We introduce a novel semi-supervised version of the least squares classifier. This implicitly constrained least squares (ICLS) classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-supervised method…
Extends matrix factorization for deviance-based losses with GLM theory.
problem Improving data loss models beyond squared error.
method Adapts GLM theory to matrix factorization for deviance losses.
result Strong consistency and robustness of the proposed decomposition.