arXiv research

Smoothness analysis of adversarial training reveals $L_\infty$ constraints cause more non-smoothness.

Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.

Theoretical analysis of cross-entropy loss functions and their robustness.

Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.

A new federated learning algorithm improves on existing methods by exploiting data smoothness.

This paper aims at refined error analysis for binary classification using support vector machine (SVM) with Gaussian kernel and convex loss. Our first result shows that for some loss functions such as the truncated quadratic loss and quadratic loss, SVM with Gaussian kernel can reach the almost optimal learning rate, p…

Paper introduces a new loss function for deep imbalanced classification.

Derandomizing PAC-Bayes bounds for smooth loss functions

Derives bounds for deterministic predictors using smooth loss functions.

New algorithm improves gradient-based ERM for smooth convex losses.

Paper relaxes SGD privacy and generalization guarantees for non-smooth convex losses.

Label smoothing improves model robustness against misspecification.

DPlis improves privacy in deep learning models by smoothing loss functions.

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…

In this paper we study the differentially private Empirical Risk Minimization (ERM) problem in different settings. For smooth (strongly) convex loss function with or without (non)-smooth regularization, we give algorithms that achieve either optimal or near optimal utility bounds with less gradient complexity compared …

Paper introduces arctan pinball loss for XGBoost quantile regression.

Guarantees uniform convergence for square-root Lipschitz losses.

Gradient descent is a simple and widely used optimization method for machine learning. For homogeneous linear classifiers applied to separable data, gradient descent has been shown to converge to the maximal margin (or equivalently, the minimal norm) solution for various smooth loss functions. The previous theory does …

Regularizers change the geometric properties of loss functions in neural networks.

Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD leads to poor generalization and easily converges to sharp minima, which prevents naive large-scale data-parallel SGD (DP-SGD) from convergin…

Traditional dictionary learning methods are based on quadratic convex loss function and thus are sensitive to outliers. In this paper, we propose a generic framework for robust dictionary learning based on concave losses. We provide results on composition of concave functions, notably regarding super-gradient computati…

Paper improves privacy and utility of SGD with bounded domain and smooth losses.

Paper improves stability analysis of SGD for various loss functions and data distributions.

SGD converges globally to logistic loss minima for two-layer nets.

New algorithm reduces prediction errors across various loss functions.

New insights into training machine learning models with momentum.

New algorithms reduce dynamic regret for convex and smooth functions in non-stationary environments.

The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical points, which only depend on the functional space, and spurious critical points, …

The Nyström method improves learning efficiency for convex losses.

Existing graph convolutional networks focus on the neighborhood aggregation scheme. When applied to semi-supervised learning, they often suffer from the overfitting problem as the networks are trained with the cross-entropy loss on a small potion of labeled data. In this paper, we propose an unsupervised manifold smoot…

In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in. This is of more importance if we consider smooth surrogates as witnessed by the fact that the smoothness is further beneficial both computationally- by at…

Piecewise polynomial interpolation-based gradient descent reduces oracle complexity for smooth loss functions.

Novel oracle-type inequality for logistic loss in DNNs achieves sharp convergence rates.

Inverse depth scaling found in LLMs due to similar layers averaging error.

The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc…

New algorithms achieve decision calibration without sample complexity dependent on feature dimension.

Paper addresses ERM in LDP, reducing sample complexity for smooth and convex losses.

Polyak step size GD reaches final radius of convergence after log iterations.

Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function such as smoothness or strong convexity constants are known, theoretical learning rate schedules can be applied. However, in practice, such …

Paper tackles robust deep learning from weakly dependent data with unbounded loss and input.

In this paper, we initiate a systematic investigation of differentially private algorithms for convex empirical risk minimization. Various instantiations of this problem have been studied before. We provide new algorithms and matching lower bounds for private ERM assuming only that each data point's contribution to the…

We introduce a novel loss function for training deep learning architectures to perform classification. It consists in minimizing the smoothness of label signals on similarity graphs built at the output of the architecture. Equivalently, it can be seen as maximizing the distances between the network function images of t…

This paper introduces SRPR for robust phase retrieval with smoothed loss functions.

Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.

Mini-batch sub-sampling in neural network training is unavoidable, due to growing data demands, memory-limited computational resources such as graphical processing units (GPUs), and the dynamics of on-line learning. In this study we specifically distinguish between static mini-batch sub-sampled loss functions, where mi…

Classification is the most important process in data analysis. However, due to the inherent non-convex and non-smooth structure of the zero-one loss function of the classification model, various convex surrogate loss functions such as hinge loss, squared hinge loss, logistic loss, and exponential loss are introduced. T…

In binary classification and regression problems, it is well understood that Lipschitz continuity and smoothness of the loss function play key roles in governing generalization error bounds for empirical risk minimization algorithms. In this paper, we show how these two properties affect generalization error bounds in …

Develops a new framework for robust regression with EGM.