arXiv research

We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non…

A new loss function improves neural networks' out-of-distribution detection without side effects.

SGD without replacement decouples into curvature-following and flatness-regularizing steps.

Optimal algorithms for online convex optimization with random order.

CGRL improves graph neural networks' OOD generalization by blocking spurious correlations.

Out-of-distribution (OOD) detection approaches usually present special requirements (e.g., hyperparameter validation, collection of outlier data) and produce side effects (e.g., classification accuracy drop, slower energy-inefficient inferences). We argue that these issues are a consequence of the SoftMax loss anisotro…

Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the so…

In many scenarios of a language identification task, the user will specify a small set of languages which he/she can speak instead of a large set of all possible languages. We want to model such prior knowledge into the way we train our neural networks, by replacing the commonly used softmax loss function with a novel …

We consider the problem of option hedging in a market with proportional transaction costs. Since super-replication is very costly in such markets, we replace perfect hedging with an expected loss constraint. Asymptotic analysis for small transactions is used to obtain a tractable model. A general expansion theory is de…

As an effective way of metric learning, triplet loss has been widely used in many deep learning tasks, including face recognition and person-ReID, leading to many states of the arts. The main innovation of triplet loss is using feature map to replace softmax in the classification task. Inspired by this concept, we prop…

New bounds derived for machine learning algorithms using convex functions.

DLM for BNNs fails to improve over ELBO optimization.

New loss function equivalence reveals PER's uniform sampling can be improved.

In few-shot learning, typically, the loss function which is applied at test time is the one we are ultimately interested in minimising, such as the mean-squared-error loss for a regression problem. However, given that we have few samples at test time, we argue that the loss function that we are interested in minimising…

It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…

New algorithm optimizes PAC-Bayes bound without surrogate loss.

A new method SLIDE ensures fairness in AI models.

This manuscript provides optimization guarantees, generalization bounds, and statistical consistency results for AdaBoost variants which replace the exponential loss with the logistic and similar losses (specifically, twice differentiable convex losses which are Lipschitz and tend to zero on one side). The heart of the…

Paper proposes new loss functions for training energy networks.

Proposes squentropy loss for improved classification accuracy and model calibration.

Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…

Real-world large-scale datasets usually contain noisy labels and are imbalanced. Therefore, we propose derivative manipulation (DM), a novel and general example weighting approach for training robust deep models under these adverse conditions. DM has two main merits. First, loss function and example weighting are commo…

It is known that Boosting can be interpreted as a gradient descent technique to minimize an underlying loss function. Specifically, the underlying loss being minimized by the traditional AdaBoost is the exponential loss, which is proved to be very sensitive to random noise/outliers. Therefore, several Boosting algorith…

Mixability of a loss is known to characterise when constant regret bounds are achievable in games of prediction with expert advice through the use of Vovk's aggregating algorithm. We provide a new interpretation of mixability via convex analysis that highlights the role of the Kullback-Leibler divergence in its definit…

Symmetric losses improve classifier robustness from corrupted labels.

This work improves structured prediction by learning the balance between signal and random noise.

Recent advances in conditional image generation tasks, such as image-to-image translation and image inpainting, are largely accounted to the success of conditional GAN models, which are often optimized by the joint use of the GAN loss with the reconstruction loss. However, we reveal that this training recipe shared by …

Learning long-term dependencies is a key long-standing challenge of recurrent neural networks (RNNs). Hierarchical recurrent neural networks (HRNNs) have been considered a promising approach as long-term dependencies are resolved through shortcuts up and down the hierarchy. Yet, the memory requirements of Truncated Bac…

Custom loss functions improve accuracy of wildfire rate of spread forecasts.

Paper simplifies balancing weights by relaxing outcome assumptions.

We demonstrate that, in the classical non-stochastic regret minimization problem with $d$ decisions, gains and losses to be respectively maximized or minimized are fundamentally different. Indeed, by considering the additional sparsity assumption (at each stage, at most $s$ decisions incur a nonzero outcome), we derive…

Through a short sale, a person borrows a share of stock from a lender, sells the borrowed share to a third person at the current price, and purchases an identical share in the market at a future date and at a future price to replace the borrowed share of stock. This only makes sense if the short seller anticipates a do…

DE-QT detects optimal Q-learning stopping points.

New decision-theoretic characterization separates belief and decision posteriors.

New loss functions based on f-divergences improve language model performance.

Individual risk models need to capture possible correlations as failing to do so typically results in an underestimation of extreme quantiles of the aggregate loss. Such dependence modelling is particularly important for managing credit risk, for instance, where joint defaults are a major cause of concern. Often, the d…

Optimizing proper loss yields calibrated models under specific conditions.

The utility of Potential Future Exposure (PFE) for counterparty trading limits is being challenged by new market developments, notably widespread regulatory Initial Margin (using 99% 10-day exposure), and netting of trade and collateral flows. However PFE has pre-existing challenges w.r.t. portfolios/distributions, col…

Replacing MSE with f-divergence in diffusion models improves robustness under data contamination.

Paper uses AI to improve medical diagnosis accuracy.

Framework for differentiating WFSTs for structured loss functions.

A robust loss for anomaly mitigation and unsupervised contamination classification

The aim of this paper is to provide new theoretical and computational understanding on two loss regularizations employed in deep learning, known as local entropy and heat regularization. For both regularized losses we introduce variational characterizations that naturally suggest a two-step scheme for their optimizatio…

This work improves adversarial robustness by boosting model ensembles with margin maximization.

Enhanced privacy, utility, and efficiency through MUST subsampling.

A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…

Given a model $f$ that predicts a target $y$ from a vector of input features $\pmb{x} = x_1, x_2, \ldots, x_M$, we seek to measure the importance of each feature with respect to the model's ability to make a good prediction. To this end, we consider how (on average) some measure of goodness or badness of prediction (wh…