arXiv research

Classification is the most important process in data analysis. However, due to the inherent non-convex and non-smooth structure of the zero-one loss function of the classification model, various convex surrogate loss functions such as hinge loss, squared hinge loss, logistic loss, and exponential loss are introduced. T…

Paper explains learning property of logistic and softmax losses for balanced and imbalanced class data.

Paper explores connections between loss functions and consistency in binary classification and regression.

Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.

We analyze the optimization landscape of α-loss in logistic models.

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

Adversarial training makes logistic regression weight loss landscapes sharper.

Flat solutions don't guarantee generalization for logistic loss in neural networks.

SGD converges globally to logistic loss minima for two-layer nets.

Introduces Fitzpatrick losses, tighter than Fenchel-Young losses.

Unified surrogate loss framework for multi-label learning with strong consistency guarantees.

New framework optimizes classification trees with logistic loss and $\ell_1$ regularization.

GD at EoS edge minimizes logistic loss without monotonic convergence.

Prevalidated ridge regression simplifies logistic regression for high-dimensional data.

Novel oracle-type inequality for logistic loss in DNNs achieves sharp convergence rates.

Fast classification for sparse models, even with correlated features.

Large stepsize GD for logistic regression converges faster than expected.

Proposes a new loss function for robust learning.

We present $α$-loss, $α\in [1,\infty]$, a tunable loss function for binary classification that bridges log-loss ($α=1$) and $0$-$1$ loss ($α= \infty$). We prove that $α$-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…

Introduces Soft-SVM for binary classification bridging logistic and SVM.

This work proposes the Bregman-Tweedie classification model and analyzes the domain structure of the extended exponential function, an extension of the classic generalized exponential function with additional scaling parameter, and related high-level mathematical structures, such as the Bregman-Tweedie loss function an…

Improved confidence bounds for linear logistic model with applications to bandits.

The paper proves neural networks' consistency and optimal convergence rates for various function classes.

New research shows logistic regression can achieve optimal error rate for agnostic learning of halfspaces.

We consider the problem of rank loss minimization in the setting of multilabel classification, which is usually tackled by means of convex surrogate losses defined on pairs of labels. Very recently, this approach was put into question by a negative result showing that commonly used pairwise surrogate losses, such as ex…

This paper concerns a method of selecting a subset of features for a sequential logit model. Tanaka and Nakagawa (2014) proposed a mixed integer quadratic optimization formulation for solving the problem based on a quadratic approximation of the logistic loss function. However, since there is a significant gap between …

Study binary choice with asymmetric loss, offering simple solutions.

New loss functions based on f-divergences improve language model performance.

01 loss is robust to outliers and adversarial attacks, outperforming convex losses.

Proposes efficient Bayesian logistic regression for large sparse datasets.

New algorithm reduces online logistic regression regret without exponential constant.

Early stopping improves neural networks' performance on binary classification tasks.

A new learning method using hyperbolic geometry for class labels.

This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side). The heart of the…

AUC (area under ROC curve) is an important evaluation criterion, which has been popularly used in many learning tasks such as class-imbalance learning, cost-sensitive learning, learning to rank, etc. Many learning approaches try to optimize AUC, while owing to the non-convexity and discontinuousness of AUC, almost all …

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non…

Learning linear predictors with the logistic loss---both in stochastic and online settings---is a fundamental task in machine learning and statistics, with direct connections to classification and boosting. Existing "fast rates" for this setting exhibit exponential dependence on the predictor norm, and Hazan et al. (20…

Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three cali…

Decision trees and logistic regression are one of the most popular and well-known machine learning algorithms, frequently used to solve a variety of real-world problems. Stability of learning algorithms is a powerful tool to analyze their performance and sensitivity and subsequently allow researchers to draw reliable c…

We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an inf…

A successful approach to structured learning is to write the learning objective as a joint function of linear parameters and inference messages, and iterate between updates to each. This paper observes that if the inference problem is "smoothed" through the addition of entropy terms, for fixed messages, the learning ob…

We apply the network Lasso to solve binary classification and clustering problems for network-structured data. To this end, we generalize ordinary logistic regression to non-Euclidean data with an intrinsic network structure. The resulting "logistic network Lasso" amounts to solving a non-smooth convex regularized empi…

The paper revisits discriminative vs. generative classifiers, showing naive Bayes requires fewer samples.

Analysis of gradient descent on wide neural networks reveals strong generalization.

Study shows deep linear networks can converge to flatter minima at large learning rates.

Improved VB algorithm for high-dimensional logistic regression with theoretical guarantees.

Second-order methods improve differential privacy in convex optimization.

Improved sketching for logistic and $\ell_1$ regression with near-linear dimensions.