Adaptive loss scaling speeds up and improves deep learning training.
problem Numerical underflow in mixed precision training.
method Adaptive loss scaling that automatically computes layer-wise loss scale values during training.
result Adaptive loss scaling leads to shorter convergence time and improved accuracy.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
ASAM improves deep neural network generalization by adapting sharpness to scale.
problem Fixed-radius sharpness measure is sensitive to parameter scaling, weakening its connection to generalization.
method Introduces adaptive sharpness, a scale-invariant measure, and proposes ASAM for deep learning.
result ASAM significantly improves model generalization performance across various datasets.
Improved loss scaling for stochastic momentum algorithms in high dimensions.
problem Improving loss scaling for stochastic momentum algorithms in high dimensions.
method Dimension-adapted Nesterov acceleration (DANA) scales momentum hyperparameters based on model size and data complexity.
result DANA improves loss scaling exponents across various data and target complexities.
New algorithm bounds MAB regret for unknown scale and magnitude of losses.
problem Adversarial Multi Armed Bandits with unknown scale and magnitude of losses.
method Design a bandit Follow The Regularized Leader (FTRL) algorithm with adaptive learning rate.
result First MAB bounds that adapt to L 2 L_2 L 2 and L 1 L_1 L 1 norms of losses. New method trains deep networks robustly without adaptive methods.
problem Training deep networks with robustness and efficiency.
method Scale invariant architecture + SGD + weight decay + gradient clipping.
result SGD can achieve similar performance to adaptive methods like Adam.
New algorithm tackles multi-armed bandit with arbitrary delays and general bounded losses.
problem Scale-free adversarial multi-armed bandit with arbitrary feedback delays.
method SFD-INF combines convex combination trick and doubling/skipping technique.
result Achieves adaptive regret bounds for non-negative and general scale-free losses.
New bin-wise scaling methods improve prediction uncertainty calibration for machine learning.
problem Improving prediction uncertainty calibration for machine learning regression.
method Adaptations of Binwise Variance Scaling (BVS) with alternative loss functions and feature-based binning.
result Improved adaptivity and consistency in prediction uncertainty calibration.
New algorithms avoid a dominant lower-order term in heavy-tailed loss settings.
problem Prediction with heavy-tailed losses without prior knowledge.
method Adaptive algorithms that avoid the maximum of losses as a lower-order term in regret.
result Improved regret bounds of O ( θ T log ( K ) ) \mathcal{O}(\sqrt{θT\log(K)}) O ( θ T log ( K ) ) and O ( θ log ( K T ) / Δ min ) \mathcal{O}(θ\log(KT)/Δ_{\min}) O ( θ log ( K T ) / Δ m i n ) . Adaptive rerouting reshapes impacts of maritime chokepoint disruptions
problem How disruptions to shipping traffic at chokepoints affect global economy
method Empirically calibrated full-scale agent-based model of global commercial shipping fleet
result Rerouting changes arrival losses under chokepoint closures
We consider the weighted belief-propagation (WBP) decoder recently proposed by Nachmani et al. where different weights are introduced for each Tanner graph edge and optimized using machine learning techniques. Our focus is on simple-scaling models that use the same weights across certain edges to reduce the storage and…
The paper explores how different loss functions impact reinforcement learning algorithms.
problem Improving reinforcement learning algorithms by optimizing loss functions.
method Comprehensive survey on loss functions in reinforcement learning, proving the benefits of specific loss functions.
result Binary cross-entropy loss leads to first-order bounds and is more efficient than squared loss.
Improved online algorithm for convex losses with near-optimal swap regret.
problem Minimizing swap regret for convex losses.
method Randomized online algorithm using multi-scale binning.
result Near-optimal O ~ ( T ) \widetilde O(\sqrt T) O ( T ) expected swap regret. Two new algorithms improve performance in adversarial bandits with unbounded losses.
problem Adversarial Multi-Armed Bandits with unbounded losses.
method Developed UMAB-NN and UMAB-G for non-negative and general unbounded losses respectively.
result UMAB-NN achieves the first adaptive and scale-free regret bound for non-negative unbounded losses.
Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed t…
SIGTRON improves classification accuracy for imbalanced datasets.
problem Improving classification accuracy for imbalanced datasets.
method SIGTRON is a new sigmoid function with a convex loss function for imbalanced classification.
result SIGTRON models outperform existing methods in balanced and imbalanced datasets.
Nowadays, users open multiple accounts on social media platforms and e-commerce sites, expressing their personal preferences on different domains. However, users' behaviors change across domains, depending on the content that users interact with, such as movies, music, clothing and retail products. In this paper, we pr…
Adaptive sampling for risk-averse learning on hard examples.
problem Training models to perform well on difficult examples in high-stakes applications.
method Adaptive sampling algorithm for stochastically optimizing CVaR, using distributionally robust formulation and regret minimization.
result Empirically demonstrates effectiveness on large-scale convex and non-convex learning tasks.
AdamL optimizes deep learning models by incorporating loss function information.
problem Adaptive optimizers can suffer from poor generalization due to nonuniform gradient scaling.
method AdamL is a novel adaptive optimizer that considers loss function information for better generalization.
result AdamL achieves faster convergence or lower objective function values compared to other optimizers.
Adaptive linear bandit algorithm with best-of-three-worlds regret bounds.
problem Adaptive to adversarial and stochastic environments with varying sub-optimality gaps and corruption.
method Combines SCRiBLe algorithm with scaled-up sampling and optimistic online learning.
result Achieves best-of-three-worlds regret bounds of O ( T log T ) O(\sqrt{T \log T}) O ( T log T ) for adversarial and O ( log T Δ min + C log T Δ min ) O(\frac{\log T}{Δ_{\min}} + \sqrt{\frac{C \log T}{Δ_{\min}}}) O ( Δ m i n l o g T + Δ m i n C l o g T ) for stochastic environments. Proposes a Dual Focal Loss method to improve semantic segmentation on unbalanced datasets.
problem Improving performance on unbalanced datasets in semantic segmentation.
method Adaptive Class Weight based Dual Focal Loss (DFL) function.
result DFL improves convergence and reduces class imbalance effect.
A new method speeds up deep neural network training.
problem Nonconvex optimization in deep neural networks.
method Scaled conjugate gradient method for nonconvex optimization.
result The method converges faster and achieves lower scores in practical applications.
A new framework for time series forecasting that adapts to varying patterns.
problem Forecasting multivariate time series with predictive heterogeneity.
method Validation-driven clustering framework that applies specialization based on out-of-sample predictive performance.
result Improves robustness to heavy-tailed errors and local anomalies.
Coordinate descent methods employ random partial updates of decision variables in order to solve huge-scale convex optimization problems. In this work, we introduce new adaptive rules for the random selection of their updates. By adaptive, we mean that our selection rules are based on the dual residual or the primal-du…
AMORE uses neural operators to efficiently predict multiple thermochemical states in stiff chemical kinetics.
problem Efficiently integrating stiff chemical kinetics systems to reduce computational cost.
method Developed AMORE, a framework of adaptive multi-output operator network with two adaptive loss functions.
result Demonstrated improved accuracy and efficiency in predicting thermochemical states from initial conditions.
Chinchilla Approach 2 biases neural scaling law estimates, leading to unnecessary compute costs.
problem Systematic biases in Chinchilla Approach 2's parabolic fits of neural scaling laws.
method Analyzes three sources of error: IsoFLOP sampling grid width, uncentered sampling, and loss surface asymmetry.
result Chinchilla Approach 3 largely eliminates these biases, offering a more convenient or scalable alternative.
Algorithm optimizes ε-SVR with MAPE loss and sample-dependent constraints.
problem Optimizing ε-SVR with MAPE loss and sample-dependent constraints.
method Sequential Minimal Optimization (SMO) for ε-SVR with MAPE loss and sample-dependent box constraints.
result Algorithm achieves lowest median runtime on every tested configuration.
We present a new method for high-dimensional linear regression when a scale parameter of the additive errors is unknown. The proposed estimator is based on a penalized Huber M M M -estimator, for which theoretical results on estimation error have recently been proposed in high-dimensional statistics literature. However, t…
New algorithms adapt to friendly environments in online learning.
problem Oracle-efficient algorithms struggle with friendly environments.
method Follow-the-perturbed-leader algorithms with approximability condition.
result Best-of-both-worlds bound in oracle-efficient setting.
Proposes ATM method to improve domain adaptation.
problem Mitigating distribution divergence between source and target domains.
method Adversarial Tight Match (ATM) method using Maximum Density Divergence (MDD).
result New state-of-the-art performance on domain adaptation benchmarks.
AB dynamically scales gradients to mitigate asynchronous training delays.
problem Gradient delay in asynchronous training reduces model performance.
method Adaptive Braking (AB) dynamically scales gradients based on alignment.
result AB enables training with up to 32 update steps of delay without accuracy loss.
Tensor network architecture for classification and regression using wavelet transformations.
problem Efficiently performing classification and regression tasks on complex data.
method Tensor network layers based on MERA and MPS, with adaptive fine-graining.
result Adaptive fine-graining improves model performance without loss in accuracy.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
Fisher loss improves deep domain adaptation by learning discriminative within-class compact and between-class separable representations.
problem Improving deep domain adaptation performance by learning discriminative representations.
method Proposes a Fisher loss to learn discriminative representations that are within-class compact and between-class separable.
result Noticeable improvements in deep domain adaptation performance, e.g., 6.67% absolute improvement in mean accuracy on the Office-Home dataset.
RG-TTA adapts neural forecasters to streaming time series shifts by modulating adaptation intensity.
problem Adapting neural forecasters to distribution shifts in streaming time series data.
method RG-TTA uses a meta-controller that continuously modulates adaptation intensity based on distributional similarity.
result RG-TTA achieves the lowest MSE in 156 of 224 seed-averaged experiments, reducing MSE by 5.7% vs TTA.
Optimal multiscale learning of linear operators
problem Statistical and computational limits of learning bounded linear operators between Sobolev spaces
method Reformulate as an infinite-dimensional matrix regression problem with heterogeneous multiscale structure
result Establish minimax rates and construct a finite-resolution blockwise least-squares estimator attaining these rates
Wavelet-based online learning adapts to noisy Besov spaces with high probability.
problem Minimizing integrated squared error in Besov spaces with noisy observations.
method Adaptive wavelet-based online learning algorithm that dynamically adjusts to gradient noise.
result Achieves minimax-optimal integrated squared error with high probability.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
We derive an online learning algorithm with improved regret guarantees for `easy' loss sequences. We consider two types of `easiness': (a) stochastic loss sequences and (b) adversarial loss sequences with small effective range of the losses. While a number of algorithms have been proposed for exploiting small effective…
SoftAdapt dynamically adjusts loss weights for multi-part functions.
problem Slow convergence and poor weight selection for multi-part loss functions.
method SoftAdapt dynamically changes weights based on live performance statistics.
result Improved convergence and better weight selection for multi-part loss functions.
While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and…
PIE-PINN estimates elastic properties from noisy, low-res displacement data.
problem Estimating heterogeneous elastic properties from low-resolution, noisy data.
method Probabilistic Physics-Informed Neural Network (PIE-PINN) framework combining B-spline and hierarchical scale model.
result Robust estimation of Young's modulus and Poisson's ratio from noisy, low-resolution displacement data.
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
First order optimization algorithms play a major role in large scale machine learning. A new class of methods, called adaptive algorithms, were recently introduced to adjust iteratively the learning rate for each coordinate. Despite great practical success in deep learning, their behavior and performance on more genera…
Meta-SAGE improves deep RL scalability for CO tasks by adapting pre-trained models to larger-scale problems.
problem Improving scalability of deep reinforcement learning models for combinatorial optimization tasks.
method Meta-SAGE combines a scale meta-learner and scheduled adaptation with guided exploration to adjust model parameters for larger-scale problems.
result Meta-SAGE outperforms previous methods and significantly improves scalability in CO tasks.
New framework adds trend information to Adam-type optimizers for faster convergence.
problem Efficiently optimizing complex cost surfaces in deep learning.
method Integrates trend information into Adam-type optimizers for adaptive step size and gradients.
result Framework outperforms conventional Adam and AMSGrad methods on various datasets.
Reinforce-Ada improves RL for language models by adaptively sampling difficult prompts.
problem Signal loss in RL for large language models due to undersampling.
method Adaptive sampling based on a non-linear RL objective, optimizing prompt difficulty weighting.
result Reinforce-Ada significantly outperforms uniform sampling, recovering lost signals and accelerating convergence.
Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to important successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new ad…