A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Flooding prevents deep networks from achieving zero training loss, improving test performance.
The paper predicts loss scaling across different datasets and compute scales.
Adversarial training makes logistic regression weight loss landscapes sharper.
The study finds a trade-off between model size, test loss, and training loss for linear predictors.
Loss-guided training accelerates node embedding methods on graphs.
Paper discovers simplicial complexes connecting trained models for improved ensembling.
Large learning rates improve neural network generalization, study shows.
AWP improves robustness by flattening weight loss landscape.
SIFT reduces training time by selecting samples with approximate losses.
This work improves diffusion models by estimating the optimal loss value for better training diagnostics.
We consider the problem of training probabilistic conditional random fields (CRFs) in the context of a task where performance is measured using a specific loss function. While maximum likelihood is the most common approach to training CRFs, it ignores the inherent structure of the task's loss function. We describe alte…
Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce mi…
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
Stable GFlowNets prevent loss spikes and mode collapse in training.
Recent work on discriminative segmental models has shown that they can achieve competitive speech recognition performance, using features based on deep neural frame classifiers. However, segmental models can be more challenging to train than standard frame-based approaches. While some segmental models have been success…
Typically, loss functions, regularization mechanisms and other important aspects of training parametric models are chosen heuristically from a limited set of options. In this paper, we take the first step towards automating this process, with the view of producing models which train faster and more robustly. Concretely…
Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called los…
Analyzes adversarial training's impact on loss landscape, proposing PAS to improve model performance.
Fewer degrees of freedom can train deep networks, showing a sharp phase transition.
In this paper, we propose a new loss function called generalized end-to-end (GE2E) loss, which makes the training of speaker verification models more efficient than our previous tuple-based end-to-end (TE2E) loss function. Unlike TE2E, the GE2E loss function updates the network in a way that emphasizes examples that ar…
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…
We present the Tamed Cross Entropy (TCE) loss function, a robust derivative of the standard Cross Entropy (CE) loss used in deep learning for classification tasks. However, unlike other robust losses, the TCE loss is designed to exhibit the same training properties than the CE loss in noiseless scenarios. Therefore, th…
In this manuscript we propose two objective terms for neural image compression: a compression objective and a cycle loss. These terms are applied on the encoder output of an autoencoder and are used in combination with reconstruction losses. The compression objective encourages sparsity and low entropy in the activatio…
Recently, we proposed short-time Fourier transform (STFT)-based loss functions for training a neural speech waveform model. In this paper, we generalize the above framework and propose a training scheme for such models based on spectral amplitude and phase losses obtained by either STFT or continuous wavelet transform …
Neural networks and linear systems linked, revealing training loss and kernel limitations.
Novel loss functions improve decision tree learning from noisy data.
A new concordance loss improves model performance and reliability in survival prediction.
Paper explores challenges in training PINNs and loss landscape effects.
Improves deep learning models by blending gradients from training loss and auxiliary objective.
Square loss performs comparably or better than cross-entropy in neural architectures for various tasks.
Proposes squentropy loss for improved classification accuracy and model calibration.
Adversarial training is a training scheme designed to counter adversarial attacks by augmenting the training dataset with adversarial examples. Surprisingly, several studies have observed that loss gradients from adversarially trained DNNs are visually more interpretable than those from standard DNNs. Although this phe…
Improved GAN training stability through tunable classification losses.
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into …
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to ind…
EGFs use ergodicity to simplify generative flows for easier training and imitation learning.
The paper explains spikes in training loss as catapults, improving feature learning and generalization.
We introduce a novel loss function for training deep learning architectures to perform classification. It consists in minimizing the smoothness of label signals on similarity graphs built at the output of the architecture. Equivalently, it can be seen as maximizing the distances between the network function images of t…
New loss function connects learning rate and momentum.
A new loss function improves neural networks' out-of-distribution detection without side effects.
Calibrating classifiers reduces grouping loss using sufficiency criteria.
A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…
Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…
Proposes SOVR loss to improve adversarial robustness by increasing logit margins.
FastFID efficiently trains generative models with FID loss.
As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions…