arXiv research

The paper predicts loss scaling across different datasets and compute scales.

Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called los…

Optimizing full likelihoods adapts loss scales and shapes for robust modeling.

We discover scaling laws for kernel regression loss under various learning rate schedules.

Model shows loss curve with two distinct exponents due to sparse activations.

New algorithm tackles multi-armed bandit with arbitrary delays and general bounded losses.

Improved loss scaling for stochastic momentum algorithms in high dimensions.

New algorithm handles bandit problems under translations and scales.

Inverse depth scaling found in LLMs due to similar layers averaging error.

Generalized algorithm for translation and scale-invariant prediction.

This paper introduces new loss functions for balanced multi-class classification.

Investigates ways to train larger models with fewer resources, finding that test loss depends only on the actual number of trainable parameters.

Topology-enhanced loss improves 3D object reconstruction from 2D images.

New DAM method improves AUC scores in medical image classification.

LLMs learn peaked distributions slowly due to power-law losses.

There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of the loss landscape that incorporates many of them. High dimensionality plays a key role in our model. Our core idea is to model the loss la…

The paper explores how different loss functions impact reinforcement learning algorithms.

Typically, operational risk losses are reported above a threshold. Fitting data reported above a constant threshold is a well known and studied problem. However, in practice, the losses are scaled for business and other factors before the fitting and thus the threshold is varying across the scaled data sample. A report…

Improved online algorithm for convex losses with near-optimal swap regret.

Tensor decomposition is a well-known tool for multiway data analysis. This work proposes using stochastic gradients for efficient generalized canonical polyadic (GCP) tensor decomposition of large-scale tensors. GCP tensor decomposition is a recently proposed version of tensor decomposition that allows for a variety of…

This research shows loss weighting remains effective in last layer retraining despite model overparameterization.

Study reveals how initialization scale affects training accuracy in linear networks.

Three training regimes found for scale-invariant neural networks on the sphere.

The paper studies entropy calibration in language models and finds that miscalibration improves slowly with scale.

New algorithms estimate Jacobian matrices for large-scale machine learning.

The study examines correlations of logarithms of integers at different scalings.

Study compares different scoring rules for machine-learned weather forecasts, finding scale-awareness improves forecast realism.

In this paper, we propose a Dual Focal Loss (DFL) function, as a replacement for the standard cross entropy (CE) function to achieve a better treatment of the unbalanced classes in a dataset. Our DFL method is an improvement on the recently reported Focal Loss (FL) cross-entropy function, which proposes a scaling metho…

New algorithm bounds MAB regret for unknown scale and magnitude of losses.

Neural networks' performance scales with data size, explained by data manifold dimensionality.

ASAM improves deep neural network generalization by adapting sharpness to scale.

New phases identified in neural scaling laws with compute limits.

Focal loss improves deep neural networks' accuracy and calibration.

ScoreStop uses gradient tests to stop gradient boosting early.

Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.

Chinchilla Approach 2 biases neural scaling law estimates, leading to unnecessary compute costs.

Boosted decision trees typically yield good accuracy, precision, and ROC area. However, because the outputs from boosting are not well calibrated posterior probabilities, boosting yields poor squared error and cross-entropy. We empirically demonstrate why AdaBoost predicts distorted probabilities and examine three cali…

Global minima found for multidimensional scaling with penalties.

Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly with the number of output classes, which can restrict the size of problems we are able to tackle with current hardware. Second, it remains unc…

Paper introduces scalable clustering for large datasets with outliers.

Theory explains neural network scaling with dataset and model size.

Two new algorithms improve performance in adversarial bandits with unbounded losses.

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth h…

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

Changing initialization scale affects deep model generalization, leading to memorization or improved performance.

The gain-loss asymmetry, observed in the inverse statistics of stock indices is present for logarithmic return levels that are over $2\%$, and it is the result of the non-Pearson type auto-correlations in the index. These non-Pearson type correlations can be viewed also as functionally dependent daily volatilities, ext…

Paper tackles non-convex constrained DRO with a stochastic algorithm for large-scale applications.

Paper proposes a novel method to improve matrix completion with median loss for large datasets.