arXiv research

Proposes squentropy loss for improved classification accuracy and model calibration.

Square loss performs comparably or better than cross-entropy in neural architectures for various tasks.

Guarantees uniform convergence for square-root Lipschitz losses.

We find a convex model for traditional nonlinear regression under L2 loss.

This work investigates square loss in overparametrized neural networks, revealing its advantages in robustness and calibration.

Gradient descent on ReLU networks with square loss implicitly favors balanced weights.

Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…

Study improves $H$-consistency bounds for regression analysis.

We consider binary classification problems with positive definite kernels and square loss, and study the convergence rates of stochastic gradient methods. We show that while the excess testing loss (squared loss) converges slowly to zero as the number of observations (and thus iterations) goes to infinity, the testing …

The paper improves Kaczmarz algorithm with momentum for linear least squares.

Study shows squared loss is robust for binary classification with noisy features.

The paper introduces a new FOR framework using Huber and ε-insensitive losses.

In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS). We propose a new randomized algorithm that has optimal generalization error bounds with respect to the square loss, closing a long-standing gap between upper and lower bounds. Moreover, we show that our al…

In this short note, we provide a sample complexity lower bound for learning linear predictors with respect to the squared loss. Our focus is on an agnostic setting, where no assumptions are made on the data distribution. This contrasts with standard results in the literature, which either make distributional assumption…

In this work we propose an adversarial learning approach to generate high resolution MRI scans from low resolution images. The architecture, based on the SRGAN model, adopts 3D convolutions to exploit volumetric information. For the discriminator, the adversarial loss uses least squares in order to stabilize the traini…

New algorithms estimate Jacobian matrices for large-scale machine learning.

We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…

The study explains delayed spikes in batch-normalized models.

This paper is concerned with the squared F(robenius)-norm regularized factorization form for noisy low-rank matrix recovery problems. Under a suitable assumption on the restricted condition number of the Hessian for the loss function, we derive an error bound to the true matrix for the non-strict critical points with r…

A novel dictionary-based approach for predicting functions.

Study finds Hilbert square of real surfaces can be maximal even when the surface has disconnected real locus.

Transformer-based models overfit financial time series data, leading to increased prediction variance.

Paper explores connections between loss functions and consistency in binary classification and regression.

Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.

The past decade has witnessed a successful application of deep learning to solving many challenging problems in machine learning and artificial intelligence. However, the loss functions of deep neural networks (especially nonlinear networks) are still far from being well understood from a theoretical aspect. In this pa…

The paper explores how different loss functions impact reinforcement learning algorithms.

Classification and regression tasks in overparameterized models show different generalization properties.

We consider regression with square loss and general classes of functions without the boundedness assumption. We introduce a notion of offset Rademacher complexity that provides a transparent way to study localization both in expectation and in high probability. For any (possibly non-convex) class, the excess loss of a …

The paper studies the loss landscape of regularized deep matrix factorization, revealing unique and sharp minimizers.

The paper improves sparse Gaussian processes by optimizing predictive loss.

Paper unifies bias and variance models for classification.

The Nyström method improves learning efficiency for convex losses.

Develops a new framework for robust regression with EGM.

This paper analyzes M-estimators under infinite-variance noise in high dimensions.

Paper proposes fitting loss functions to data using source functions from information geometry.

Improved speech enhancement using diffusion models with MSE loss.

Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (A…

The goal of temporal alignment is to establish time correspondence between two sequences, which has many applications in a variety of areas such as speech processing, bioinformatics, computer vision, and computer graphics. In this paper, we propose a novel temporal alignment method called least-squares dynamic time war…

We study tensor completion in the agnostic setting. In the classical tensor completion problem, we receive $n$ entries of an unknown rank-$r$ tensor and wish to exactly complete the remaining entries. In agnostic tensor completion, we make no assumption on the rank of the unknown tensor, but attempt to predict unknown …

This paper extends the standard chaining technique to prove excess risk upper bounds for empirical risk minimization with random design settings even if the magnitude of the noise and the estimates is unbounded. The bound applies to many loss functions besides the squared loss, and scales only with the sub-Gaussian or …

Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…

Deep nets trained with MSE loss exhibit Neural Collapse, collapsing features and classifiers to class means.

This paper presents a learning method for convolutional autoencoders (CAEs) for extracting features from images. CAEs can be obtained by utilizing convolutional neural networks to learn an approximation to the identity function in an unsupervised manner. The loss function based on the pixel loss (PL) that is the mean s…

New method optimizes fairness in predictive models for continuous sensitive attributes.

The logcosh loss function helps neural networks learn set-valued functions better.

Analytical method finds deeper optima in two-layer ReLU networks.

New algorithms avoid a dominant lower-order term in heavy-tailed loss settings.

The classical asymptotic theory for parametric $M$-estimators guarantees that, in the limit of infinite sample size, the excess risk has a chi-square type distribution, even in the misspecified case. We demonstrate how self-concordance of the loss allows to characterize the critical sample size sufficient to guarantee …