arXiv research

Paper develops methods for non-quadratic loss low-rank matrix recovery.

Study Markov chain gradient descent in Hilbert spaces for quadratic loss.

Extends quadratic loss for SVM and deep learning to improve pattern correlation.

Proposes a new Huber loss combining absolute and quadratic properties.

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points…

A new method for exponentially weighted moving models using approximations.

This paper concerns a method of selecting a subset of features for a sequential logit model. Tanaka and Nakagawa (2014) proposed a mixed integer quadratic optimization formulation for solving the problem based on a quadratic approximation of the logistic loss function. However, since there is a significant gap between …

Optimal algorithm for LQR control with improved regret bound.

This paper aims at refined error analysis for binary classification using support vector machine (SVM) with Gaussian kernel and convex loss. Our first result shows that for some loss functions such as the truncated quadratic loss and quadratic loss, SVM with Gaussian kernel can reach the almost optimal learning rate, p…

For semi-supervised techniques to be applied safely in practice we at least want methods to outperform their supervised counterparts. We study this question for classification using the well-known quadratic surrogate loss function. Using a projection of the supervised estimate onto a set of constraints imposed by the u…

Bayesian neural networks are shown to be minimax and admissible under certain conditions.

Q-SHAP efficiently calculates feature contributions in boosting trees.

Study on neural networks with quadratic activation functions, focusing on optimization and generalization.

One-pass SGD dynamics in overparameterized quadratic networks show slow escape from poor solutions.

Despite their practical success, a theoretical understanding of the loss landscape of neural networks has proven challenging due to the high-dimensional, non-convex, and highly nonlinear structure of such models. In this paper, we characterize the training landscape of the mean squared error loss for neural networks wi…

A new method automatically and dynamically sets learning rates in deep learning.

We provide new theoretical insights on why over-parametrization is effective in learning neural networks. For a $k$ hidden node shallow network with quadratic activation and $n$ training data points, we show as long as $k \ge \sqrt{2n}$, over-parametrization enables local search algorithms to find a \emph{globally} op…

New analysis improves SGD for robust and quantile regression with sub-quadratic convergence.

New findings show GD converges to a linear interpolator even with quadratic loss function under certain conditions.

Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.

RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.

LQF linearizes deep models for better interpretability.

New bound matches exact generalization error for quadratic Gaussian problem.

We study the problem of controlling linear time-invariant systems with known noisy dynamics and adversarially chosen quadratic losses. We present the first efficient online learning algorithms in this setting that guarantee $O(\sqrt{T})$ regret under mild assumptions, where $T$ is the time horizon. Our algorithms rely …

Ensemble techniques are powerful approaches that combine several weak learners to build a stronger one. As a meta-learning framework, ensemble techniques can easily be applied to many machine learning methods. Inspired by ensemble techniques, in this paper we propose an ensemble loss functions applied to a simple regre…

The paper debiases mini-batch approximations in deep learning for more accurate optimization and uncertainty quantification.

We propose in this paper a differentiable learning loss between time series, building upon the celebrated dynamic time warping (DTW) discrepancy. Unlike the Euclidean distance, DTW can compare time series of variable size and is robust to shifts or dilatations across the time dimension. To compute DTW, one typically so…

Catapult phase in neural nets shows exponential loss growth before quick decrease.

SLT reveals how grokking occurs via basin selection in training.

Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.

The paper develops methods for time-varying constrained online convex optimization.

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…

In this work we establish the first linear convergence result for the stochastic heavy ball method. The method performs SGD steps with a fixed stepsize, amended by a heavy ball momentum term. In the analysis, we focus on minimizing the expected loss and not on finite-sum minimization, which is typically a much harder p…

Traditional dictionary learning methods are based on quadratic convex loss function and thus are sensitive to outliers. In this paper, we propose a generic framework for robust dictionary learning based on concave losses. We provide results on composition of concave functions, notably regarding super-gradient computati…

A new method improves adversarial robustness and interpretability with reduced training time.

Closed-form polynomial approximations replace MLPs in transformers, enabling new interpretability methods.

New bounds show linear predictors rarely overfit with certain optimization methods.

The paper sets lower bounds for adversarial robustness in multiclass classification.

We show that the gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given $\tilde{O}(dr^{2})$ random linear measurements of a rank $r$ positive s…

Method identifies shifts leading to large model performance differences.

Homogenized SGD explains SGD dynamics in high dimensions.

The least absolute shrinkage and selection operator (lasso) and ridge regression produce usually different estimates although input, loss function and parameterization of the penalty are identical. In this paper we look for ridge and lasso models with identical solution set. It turns out, that the lasso model with shri…

Proposes sparse QSVM for better generalization and interpretability.

New algorithm reduces regret in bandit optimization for high-dimensional data.

Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.

Self-test loss functions improve data-driven modeling of weak-form operators and gradient flows.

Study non-asymptotic bounds for robust estimators under misspecified models.

Many learning algorithms are formulated in terms of finding model parameters which minimize a data-fitting loss function plus a regularizer. When the regularizer involves the l0 pseudo-norm, the resulting regularization path consists of a finite set of models. The fastest existing algorithm for computing the breakpoint…