arXiv research

Paper introduces MRCs that minimize worst-case 0-1 loss, providing tight performance guarantees.

MRCs minimize worst-case expected 0-1 loss and provide performance guarantees.

New solver for MKL-SVM with 0/1 loss function.

Paper introduces MKL-$L_{0/1}$-SVM for SVM with $(0, 1)$ loss.

Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (r…

This research analyzes the consistency of convex and nonconvex surrogate losses for adversarially robust classification.

ICE algorithm solves exact 0-1 loss linear classification problem efficiently.

We consider the problem of learning linear classifiers when both features and labels are binary. In addition, the features are noisy, i.e., they could be flipped with an unknown probability. In Sy-De attribute noise model, where all features could be noisy together with same probability, we show that $0$-$1$ loss ($l_{…

Support vector machine (SVM) has attracted great attentions for the last two decades due to its extensive applications, and thus numerous optimization models have been proposed. To distinguish all of them, in this paper, we introduce a new model equipped with an $L_{0/1}$ soft-margin loss (dubbed as $L_{0/1}$-SVM) whic…

Characterizes learnability of forgiving 0-1 loss functions in multiclass settings.

We present $α$-loss, $α\in [1,\infty]$, a tunable loss function for binary classification that bridges log-loss ($α=1$) and $0$-$1$ loss ($α= \infty$). We prove that $α$-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…

In classification, the de facto method for aggregating individual losses is the average loss. When the actual metric of interest is 0-1 loss, it is common to minimize the average surrogate loss for some well-behaved (e.g. convex) surrogate. Recently, several other aggregate losses such as the maximal loss and average t…

Active learning is a type of sequential design for supervised machine learning, in which the learning algorithm sequentially requests the labels of selected instances from a large pool of unlabeled data points. The objective is to produce a classifier of relatively low risk, as measured under the 0-1 loss, ideally usin…

This paper improves SAM by reformulating it as a bilevel optimization problem.

Paper proves L2 regression can learn k-juntas without distributional assumptions.

Conventional techniques for supervised classification constrain the classification rules considered and use surrogate losses for classification 0-1 loss. Favored families of classification rules are those that enjoy parametric representations suitable for surrogate loss minimization, and low complexity properties suita…

Paper unifies bias and variance models for classification.

Classification and regression tasks in overparameterized models show different generalization properties.

The paper sets lower bounds for adversarial robustness in multiclass classification.

Develops gradient boosting for multi-label classification.

Given a task of predicting $Y$ from $X$, a loss function $L$, and a set of probability distributions $Γ$ on $(X,Y)$, what is the optimal decision rule minimizing the worst-case expected loss over $Γ$? In this paper, we address this question by introducing a generalization of the principle of maximum entropy. Applying t…

New method improves transductive learning predictions with multiplicative oracle inequalities.

New criterion assesses cluster separability for validation.

We analyze the optimization landscape of α-loss in logistic models.

The study establishes SQ lower bounds for learning halfspaces and ReLUs under Gaussian marginals.

We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be optimized via stochastic gradient descent and we prove tight bounds on the so-called "calibration func…

New loss function calibrates WW-hinge loss for multiclass SVM.

This paper introduces Stochastic Gradient Langevin Boosting (SGLB) - a powerful and efficient machine learning framework that may deal with a wide range of loss functions and has provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradie…

This paper quantifies and mitigates a bias in the Hayashi-Yoshida estimator causing data loss.

We consider distributed online convex optimization problems, where the distributed system consists of various computing units connected through a time-varying communication graph. In each time step, each computing unit selects a constrained vector, experiences a loss equal to an arbitrary convex function evaluated at t…

Optimal domain adaptation model using Fisher's Linear Discriminant.

This research improves PAC-Bayesian bounds for classification tasks using convexified loss.

New PAC-Bayes bounds for unbounded loss functions.

We introduce a tunable loss function called $α$-loss, parameterized by $α\in (0,\infty]$, which interpolates between the exponential loss ($α= 1/2$), the log-loss ($α= 1$), and the 0-1 loss ($α= \infty$), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $…

We study online aggregation of the predictions of experts, and first show new second-order regret bounds in the standard setting, which are obtained via a version of the Prod algorithm (and also a version of the polynomially weighted average algorithm) with multiple learning rates. These bounds are in terms of excess l…

Boosting with tempered exponential measures improves AdaBoost's convergence rate.

We prove that, under low noise assumptions, the support vector machine with $N\ll m$ random features (RFSVM) can achieve the learning rate faster than $O(1/\sqrt{m})$ on a training set with $m$ samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from…

The majority of traditional classification ru les minimizing the expected probability of error (0-1 loss) are inappropriate if the class probability distributions are ill-defined or impossible to estimate. We argue that in such cases class domains should be used instead of class distributions or densities to construct …

Extends graph similarity theory to improve MPNNs' generalization abilities.

Proposes a hierarchical curriculum loss to improve model accuracy and interpretability.

The paper proves neural networks' consistency and optimal convergence rates for various function classes.

A new method learns proper multiclass losses and probabilities.

Denoising autoencoders (DAEs) are powerful deep learning models used for feature extraction, data generation and network pre-training. DAEs consist of an encoder and decoder which may be trained simultaneously to minimise a loss (function) between an input and the reconstruction of a corrupted version of the input. The…

This paper investigates Pareto optimal (PO, for short) insurance contracts in a behavioral finance framework, in which the insured evaluates contracts by the rank-dependent utility (RDU) theory and the insurer by the expected value premium principle. The incentive compatibility constraint is taken into account, so the …

We study consistency properties of surrogate loss functions for general multiclass learning problems, defined by a general multiclass loss matrix. We extend the notion of classification calibration, which has been studied for binary and multiclass 0-1 classification problems (and for certain other specific learning pro…

Deep neural networks achieve optimal classification rates in high dimensions.

In this paper we refine the process of computing calibration functions for a number of multiclass classification surrogate losses. Calibration functions are a powerful tool for easily converting bounds for the surrogate risk (which can be computed through well-known methods) into bounds for the true risk, the probabili…

Unified theory explains diversity in ensemble learning.