New meta-learning method improves domain generalization by balancing parameters closer to domain centroids.
problem Improving domain generalization by reducing overfitting to specific domains.
method Arithmetic meta-learning with arithmetic-weighted gradients to balance parameters closer to domain centroids.
result Experimental validation of improved domain generalization performance.
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
problem Balancing multiple loss terms in neural model training is challenging and computationally demanding.
method Generalized Adam algorithm that computes separate derivatives and balances gradients across layers dynamically.
result Training with MTAdam leads to faster recovery from suboptimal initial loss weighting and matches conventional training outcomes.
Proposes a method to balance tasks in multitask learning with a single gradient step update.
problem Balancing tasks in multitask learning to avoid imbalance.
method Gradient-based meta-learning to balance tasks at the gradient level, training shared and task-specific layers separately.
result Achieves state-of-the-art performance on various multitask computer vision problems.
Researchers derive a new equation for valuing American options.
problem Valuation and hedging of American options on dividend-paying assets.
method Derive a stochastic balance equation for the value function and its gradient.
result The derived equation uniquely solves the valuation problem.
Estimates causal effects using neural networks for balancing covariates.
problem Estimating causal effects from observational data.
method Neural Balancing Weights (NBW) using α-divergence for density ratio estimation. result Generalized approach for balancing multidimensional data.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
Paper proposes ARB-Loss to improve classification precision in imbalanced datasets.
problem Improving classification precision on minor classes in imbalanced datasets.
method Introduces Attraction-Repulsion-Balanced Loss (ARB-Loss) to balance gradients across different classes.
result ARB-Loss achieves state-of-the-art performance with one-stage training.
GENIE balances domain-invariant feature learning and gradient alignment for improved DG performance.
problem Domain Generalization (DG) overfitting to domain-specific features
method GENIE (Generalization-ENhancing Iterative Equalizer) optimizer
result Prevents a small subset of parameters from dominating optimization, promoting domain-invariant feature learning
The paper studies matrix normalization and graph balancing using a new functional and gradient descent.
problem Matrix normalization and graph balancing.
method A new functional called the non-normal energy, and gradient descent.
result Gradient descent of the non-normal energy converges to balanced graphs and preserves spectra and realness of weights.
Existence of balanced embedding proved for complex manifold into infinite-dimensional space.
problem Balanced embedding of non-compact complex manifolds into infinite-dimensional projective space.
method Gradient flow in a Hilbert space, long-time existence established by perturbation, convergence depends on a priori bounds.
result Existence of balanced embedding proved in a model case.
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
problem Critic-side gradient ill-conditioning in PPO for MTRL.
method Critic Balancing modules to improve gradient conditioning and balance task updates.
result TOPPO achieves stronger mean and tail-task performance than SAC-family and ARS-family baselines.
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
Gradient descent solves asymmetric low-rank matrix sensing without balancing.
problem Recovering asymmetric low-rank matrices from linear measurements.
method Gradient descent with spectral initialization, avoiding balancing term.
result Gradient descent converges linearly without balancing, factors stay balanced.
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step …
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
Proposes unbiased estimators for training mixture of experts models.
problem Efficiently training large-scale mixture of experts models on modern hardware.
method Two unbiased estimators based on principled stochastic assignment procedures.
result Both estimators are more effective and robust than biased alternatives.
Abstract: Necessary and sufficient conditions for gradient flows of relative entropy in Lindblad equations.
problem Conditions for gradient flows in finite-dimensional Lindblad equations.
method Analyzes conditions for a finite-dimensional Lindblad equation to have a gradient flow structure for the von Neumann relative entropy.
result A finite-dimensional Lindblad equation admits a gradient flow structure for the von Neumann relative entropy if and only if the BKM-detailed balance condition holds.
A number of optimization approaches have been proposed for optimizing nonconvex objectives (e.g. deep learning models), such as batch gradient descent, stochastic gradient descent and stochastic variance reduced gradient descent. Theory shows these optimization methods can converge by using an unbiased gradient estimat…
Recent sparse MRI reconstruction models have used Deep Neural Networks (DNNs) to reconstruct relatively high-quality images from highly undersampled k-space data, enabling much faster MRI scanning. However, these techniques sometimes struggle to reconstruct sharp images that preserve fine detail while maintaining a nat…
The paper explores how symmetries and noise in SGD influence parameter dynamics.
problem Understanding the dynamics of parameter updates in SGD with symmetries.
method Proved the existence of noise equilibria and showed their role in balancing gradient noise.
result Gradient noise creates a systematic motion of parameters to a unique fixed point, called noise equilibria.
A new amortized variance-reduced gradient (AVRG) algorithm was developed in \cite{ying2017convergence}, which has constant storage requirement in comparison to SAGA and balanced gradient computations in comparison to SVRG. One key advantage of the AVRG strategy is its amenability to decentralized implementations. In th…
TKIL improves class-balanced performance in incremental learning.
problem Catastrophic forgetting in sequential learning tasks.
method Introduces Tangent Kernel for Incremental Learning (TKIL) based on Neural Tangent Kernel (NTK).
result TKIL achieves better overall accuracy and variance across classes.
Regularization leads to balancedness in deep linear networks.
problem Balancedness in deep linear networks.
method Geometric invariant theory and Riemannian geometry of fibers.
result Balancing flows converge to the balanced manifold at a uniform exponential rate.
We solve tensor balancing, rescaling an Nth order nonnegative tensor by multiplying N tensors of order N - 1 so that every fiber sums to one. This generalizes a fundamental process of matrix balancing used to compare matrices in a wide range of applications from biology to economics. We present an efficient balancing a…
Proposes an automatic cyclical scheduling for gradient-based discrete sampling.
problem Gradient-based sampling in high-dimensional models can get stuck in local modes.
method Cyclical step size and balancing schedules with automatic hyperparameter tuning.
result Proves non-asymptotic convergence and inference guarantees for general discrete distributions.
New insights into how large learning rates affect transformer training dynamics.
problem Understanding how large learning rates impact the training of transformer models.
method Analyzing a simplified linear transformer model with a two-factor product map.
result Large learning rates can lead to various training outcomes including cycles, chaos, or divergence.
Coded computation techniques provide robustness against straggling servers in distributed computing, with the following limitations: First, they increase decoding complexity. Second, they ignore computations carried out by straggling servers; and they are typically designed to recover the full gradient, and thus, canno…
We study the quantization of coupled Kähler-Einstein (CKE) metrics, namely we approximate CKE metrics by means of the canonical Bergman metrics, so called the ``balanced metrics''. We prove the existence and weak convergence of balanced metrics for the negative first Chern class, while for the positive first Chern clas…
Gradient descent in deep networks tends to find flat minima, which are nearly balanced.
problem Understanding the effect of gradient descent on the structure of minima in deep neural networks.
method Characterized flat minima in linear neural networks trained with a quadratic loss.
result Flat minima correspond to nearly balanced networks where the gain from input to intermediate representations is nearly constant.
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
We develop an efficient sampling method by simulating Langevin dynamics with an artificial force rather than a natural force by using the gradient of the potential energy. The standard technique for sampling following the predetermined distribution such as the Gibbs-Boltzmann one is performed under the detailed balance…
Investigates upsampling vs. upweighting for balanced training on skewed datasets.
problem Balancing training on heavily imbalanced datasets with scarce data.
method Theoretical and empirical analysis of upsampling and upweighting strategies.
result Upsampling and upweighting diverge under stochastic gradient descent, with upsampling leading to faster convergence but higher overfitting risk.
Paper explores balancing market dynamics and interpretable forecasting models for energy prices.
problem Tackles the challenge of accurately predicting mFRR price and understanding market dynamics.
method Compares XGBoost and EBM for forecasting mFRR activation price in the balancing market.
result EBM provides comparable forecasting accuracy to XGBoost but with higher interpretability.
New approach improves model generalization through distributionally robust learning.
problem Improving model generalization in machine learning.
method Stochastic gradient descent applied to the outer minimization problem, with gradient estimation through multi-level Monte Carlo randomization.
result Our approach yields significant benefits over previous work in numerical experiments.
Large learning rates lead to various implicit biases in nonconvex optimization.
problem Understanding the conditions under which large learning rates yield edge of stability, balancing, and catapult phenomena.
method Developed a global convergence theory for nonconvex functions without globally Lipschitz continuous gradient, focusing on functions with good regularity.
result These implicit biases are more likely to occur in functions with good regularity, and large learning rates favor flatter regions.
Introduces TDRC to balance TD's ease and soundness.
problem TD learning's instability and divergence issues.
method Gradient Temporal-Difference Learning with Regularized Corrections (TDRC).
result TDRC performs as well as TD when TD works, but is sound in divergent cases.
New method trains GFlowNets from partial episodes to improve convergence and stability.
problem Improving convergence and stability of GFlowNets training.
method Introducing SubTB(λ) for GFlowNet training from partial action subsequences. result SubTB(λ) accelerates GFlowNet convergence and enables training in longer action sequences. New ODE solvers improve training efficiency and accuracy.
problem Training Neural ODEs requires efficient and accurate gradient calculation.
method Presented algebraically reversible ODE solvers that are time and memory efficient, calculate exact gradients, and are numerically stable.
result Reversible solvers strictly improve upon previous architectures in efficiency and accuracy.
This paper surveys gradient-based multi-objective deep learning methods.
problem Balancing multiple conflicting objectives in deep learning models.
method Gradient-based techniques adapted from Multi-Objective Optimization.
result Comprehensive survey of gradient-based multi-objective deep learning algorithms.
Improved GSPGS estimators reduce bias in noisy function measurements.
problem Reduced bias in noisy function measurements.
method Generalized Simultaneous Perturbation-based Gradient Search (GSPGS) with various estimators.
result Estimators requiring more function measurements have lower bias.
This work improves structured prediction by learning the balance between signal and random noise.
problem Structured prediction with random perturbations.
method Learning the variance of randomized structured predictors to balance signal and noise.
result Learning the balance improves structured prediction effectiveness.
SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.
New method balances multivariate model fitting for mixed likelihoods.
problem Multivariate models often fit only a subset of observed variables.
method Lipschitz standardization for data preprocessing.
result Lipschitz standardization leads to more accurate multivariate models.
Variational inference improves training of generative flow networks.
problem Training generative flow networks efficiently and accurately.
method Define variational objectives in terms of KL divergences and optimize convex combinations.
result Variational inference methods can reduce the variance of gradients in training generative flow networks.
Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration. The adaptively adjus…
SkMM selects data for finetuning by balancing bias and variance.
problem Balancing bias and variance in high-dimensional finetuning.
method Gradient sketching for bias reduction and moment matching for variance reduction.
result Gradient sketching selects samples efficiently and accurately.
Gradient descent solves asymmetric low-rank matrix factorization efficiently.
problem Optimizing asymmetric low-rank matrix factorization with non-convex and non-smoothness issues.
method Randomly initialized gradient descent with new symmetrization and perturbation techniques.
result Gradient descent converges to a global minimum of the asymmetric low-rank factorization problem.
Paper proposes new gradient codes for robust distributed machine learning.
problem Robustness against slow machines (stragglers) in distributed machine learning.
method Constructs gradient codes based on probabilistic and Kronecker product methods.
result Gradient codes exist for a wide range of system parameters with comparable error performance.