Enhances MRI image quality with Conditional WGAN and adaptive balancing.
problem Struggles to reconstruct sharp images with fine detail.
method Conditional Wasserstein Generative Adversarial Network (WGAN) with Adaptive Gradient Balancing.
result Produces sharper images than other techniques.
Researchers derive a new equation for valuing American options.
problem Valuation and hedging of American options on dividend-paying assets.
method Derive a stochastic balance equation for the value function and its gradient.
result The derived equation uniquely solves the valuation problem.
AdaGrad-Norm achieves linear convergence for certain functions.
problem Proving linear convergence for specific types of functions.
method Introducing RUIG, a measure of gradient balance; developing a two-stage framework.
result AdaGrad-Norm achieves linear convergence for certain functions.
Proposes MEDM to balance entropy minimization and diversity maximization for better domain adaptation.
problem Trivial solutions in entropy minimization for unsupervised domain adaptation.
method Introduces diversity maximization to balance with entropy minimization, controlled by deep embedded validation.
result MEDM outperforms state-of-the-art methods on four domain adaptation datasets.
Proposes blockwise adaptive stepsize for faster training and better generalization in deep learning.
problem Widespread use of coordinate-wise adaptive methods like RMSprop and Adam leads to worse generalization than SGD.
method Splits network parameters into blocks and uses a blockwise adaptive stepsize, balancing adaptivity and generalization.
result Blockwise adaptive gradient descent converges faster and has lower generalization error than coordinate-wise adaptive methods.
New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.
problem Current deep learning models lack transparency and high convergence rates.
method Reconsider neural networks as single-species population dynamics with balanced growth and harvesting rates.
result SGD with balanced growth and harvesting rates outperforms adaptive methods in all three requirements.
Cross-regularization adapts model complexity during training.
problem Manual tuning of model complexity for overfitting prevention.
method Directly adapts regularization parameters through validation gradients during training.
result Organic emergence of architecture-specific regularization during training.
The paper studies matrix normalization and graph balancing using a new functional and gradient descent.
problem Matrix normalization and graph balancing.
method A new functional called the non-normal energy, and gradient descent.
result Gradient descent of the non-normal energy converges to balanced graphs and preserves spectra and realness of weights.
Proposes ADC for cross-domain recommendation balancing user preferences.
problem Users' preferences change across different domains (e.g., social media, e-commerce).
method Designs a neural architecture and cross-domain loss function to adaptively balance user preferences.
result ADC model effectively balances the impact of domains with different complexities.
NeAda solves nonconvex minimax optimization by balancing primal and dual variables adaptively.
problem Nonconvex minimax optimization challenges with parameter-agnostic adaptive algorithms.
method Nested Adaptive (NeAda) framework with inner and outer loops for primal and dual variables.
result Achieves near-optimal convergence rates for nonconvex-strongly-concave problems.
Proposes an automatic cyclical scheduling for gradient-based discrete sampling.
problem Gradient-based sampling in high-dimensional models can get stuck in local modes.
method Cyclical step size and balancing schedules with automatic hyperparameter tuning.
result Proves non-asymptotic convergence and inference guarantees for general discrete distributions.
Stochastic gradient descent updates parameters with summation gradient computed from a random data batch. This summation will lead to unbalanced training process if the data we obtained is unbalanced. To address this issue, this paper takes the error variance and error mean both into consideration. The adaptively adjus…
TKIL improves class-balanced performance in incremental learning.
problem Catastrophic forgetting in sequential learning tasks.
method Introduces Tangent Kernel for Incremental Learning (TKIL) based on Neural Tangent Kernel (NTK).
result TKIL achieves better overall accuracy and variance across classes.
New scalable methods for unbalanced optimal transport improve efficiency and applicability.
problem Scalable algorithms for unbalanced optimal transport remain underexplored.
method Analysis of semi-dual formulation and adaptive gradient methods.
result SGD methods achieve a convergence rate of O(n/εT) for large-scale applications.
New meta-learning method improves domain generalization by balancing parameters closer to domain centroids.
problem Improving domain generalization by reducing overfitting to specific domains.
method Arithmetic meta-learning with arithmetic-weighted gradients to balance parameters closer to domain centroids.
result Experimental validation of improved domain generalization performance.
This paper surveys gradient-based multi-objective deep learning methods.
problem Balancing multiple conflicting objectives in deep learning models.
method Gradient-based techniques adapted from Multi-Objective Optimization.
result Comprehensive survey of gradient-based multi-objective deep learning algorithms.
MTAdam optimizes multiple loss terms in neural models, balancing gradients dynamically.
problem Balancing multiple loss terms in neural model training is challenging and computationally demanding.
method Generalized Adam algorithm that computes separate derivatives and balances gradients across layers dynamically.
result Training with MTAdam leads to faster recovery from suboptimal initial loss weighting and matches conventional training outcomes.
Proposes a method to balance tasks in multitask learning with a single gradient step update.
problem Balancing tasks in multitask learning to avoid imbalance.
method Gradient-based meta-learning to balance tasks at the gradient level, training shared and task-specific layers separately.
result Achieves state-of-the-art performance on various multitask computer vision problems.
Transfer learning has achieved promising results by leveraging knowledge from the source domain to annotate the target domain which has few or none labels. Existing methods often seek to minimize the distribution divergence between domains, such as the marginal distribution, the conditional distribution or both. Howeve…
Estimates causal effects using neural networks for balancing covariates.
problem Estimating causal effects from observational data.
method Neural Balancing Weights (NBW) using α-divergence for density ratio estimation. result Generalized approach for balancing multidimensional data.
ADAPT method controls balance between sparse generator and discriminator in GANs.
problem High computational complexity in GANs training.
method Balanced Dynamic Sparse Training (ADAPT) method.
result ADAPT method achieves a good trade-off between performance and computational cost.
In this paper, we propose a novel technique to implement stochastic gradient methods, which are beneficial for learning from large datasets, through accelerated stochastic dynamics. A stochastic gradient method is based on mini-batch learning for reducing the computational cost when the amount of data is large. The sto…
Paper proposes ARB-Loss to improve classification precision in imbalanced datasets.
problem Improving classification precision on minor classes in imbalanced datasets.
method Introduces Attraction-Repulsion-Balanced Loss (ARB-Loss) to balance gradients across different classes.
result ARB-Loss achieves state-of-the-art performance with one-stage training.
BI-MAML learns multiple tasks without forgetting old ones.
problem Catastrophic forgetting in meta learning.
method Incremental model adaptation with balanced learning strategy.
result BI-MAML outperforms state-of-the-art models in accuracy and efficiency.
New approach tackles class imbalance in long-tailed datasets using domain adaptation techniques.
problem Class imbalance in long-tailed datasets leading to poor model performance.
method Proposes a meta-learning approach to estimate differences between class-conditioned distributions.
result Validated approach on six benchmark datasets and three loss functions.
RLMH improves adaptive MCMC by optimizing contrastive divergence reward.
problem Tuning MCMC samplers is challenging and time-consuming.
method Formulated Metropolis-Hastings as a Markov decision process and used RL to adaptively tune it.
result A novel reward function based on contrastive divergence outperforms existing ones.
Improved DeepONets for PDE solution operators with adaptive re-weighting and new architecture.
problem Training DeepONets for PDE solution operators without paired data.
method Adaptive re-weighting of training examples and novel network architecture.
result Consistently improved predictive accuracy by a factor of 10-50x.
GENIE balances domain-invariant feature learning and gradient alignment for improved DG performance.
problem Domain Generalization (DG) overfitting to domain-specific features
method GENIE (Generalization-ENhancing Iterative Equalizer) optimizer
result Prevents a small subset of parameters from dominating optimization, promoting domain-invariant feature learning
Several useful variance-reduced stochastic gradient algorithms, such as SVRG, SAGA, Finito, and SAG, have been proposed to minimize empirical risks with linear convergence properties to the exact minimizer. The existing convergence results assume uniform data sampling with replacement. However, it has been observed in …
Existence of balanced embedding proved for complex manifold into infinite-dimensional space.
problem Balanced embedding of non-compact complex manifolds into infinite-dimensional projective space.
method Gradient flow in a Hilbert space, long-time existence established by perturbation, convergence depends on a priori bounds.
result Existence of balanced embedding proved in a model case.
TOPPO improves PPO for MTRL by balancing critic gradients, outperforming SAC.
problem Critic-side gradient ill-conditioning in PPO for MTRL.
method Critic Balancing modules to improve gradient conditioning and balance task updates.
result TOPPO achieves stronger mean and tail-task performance than SAC-family and ARS-family baselines.
Noise balance theory explains SGD's behavior in neural networks.
problem Understanding SGD's navigation in neural network loss landscapes.
method Analyzes minibatch noise and loss function symmetries.
result Derives the stationary distribution of SGD for deep networks.
Transformer learns to infer partial MDPs for efficient in-context adaptation and exploration.
problem Efficiently adapt and explore in-context without gradient-based updates.
method Uses a transformer to learn inference from training tasks, considering hypothesis space of partial models.
result Adaptation speed and exploration-exploitation balance approach those of an exact posterior sampling oracle.
Gradient descent solves asymmetric low-rank matrix sensing without balancing.
problem Recovering asymmetric low-rank matrices from linear measurements.
method Gradient descent with spectral initialization, avoiding balancing term.
result Gradient descent converges linearly without balancing, factors stay balanced.
Adaptive algorithm improves nonlinear data assimilation for non-Gaussian systems.
problem Challenges of non-Gaussian statistics in data assimilation.
method Triangular measure transport with P-spline basis functions and an information criterion.
result Automatic selection of parsimonious parametrization for efficient adaptation.
SIFG uses noisy particles to efficiently sample from complex distributions.
problem Efficient sampling from complex distributions using particle-based methods.
method SIFG introduces a semi-implicit functional gradient flow with Gaussian noise to improve sampling efficiency and accuracy.
result SIFG achieves strong theoretical convergence guarantees and efficient sampling.
A new random forest algorithm improves tree construction for optimal performance.
problem Improving the performance of random forests, especially in complex and smooth scenarios.
method Adaptive split-balancing method using permutation-based splitting criterion.
result Achieves minimax optimality under various Lipschitz and Hölder classes.
ALPE improves mid-price forecasting in HFT with real-time data.
problem Real-time mid-price forecasting in high-frequency trading.
method Adaptive Learning Policy Engine (ALPE) using RL and adaptive epsilon decay.
result ALPE outperforms other models in mid-price forecasting.
Paper solves pendulum swing-up problem using RL.
problem Solving the classic pendulum swing-up problem.
method Deep Deterministic Policy Gradient algorithm applied to continuous action domain.
result Optimal pendulum achieved with increasing average return and decreasing loss.
Adaptive weights improve physics-informed neural networks and deep operator networks.
problem Training physics-informed neural networks and deep operator networks can be challenging, leading to unsatisfactory accuracy and efficiency.
method Proposes a pointwise adaptive weighting method that balances the residual decay rate across different training points.
result Our proposed approach of balanced residual decay rates offers advantages including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step …
Stochastic gradient descent (SGD) is commonly used for optimization in large-scale machine learning problems. Langford et al. (2009) introduce a sparse online learning method to induce sparsity via truncated gradient. With high-dimensional sparse data, however, the method suffers from slow convergence and high variance…
New method predicts graph structure changes over time.
problem Existing graph prediction methods assume static vertices, limiting their applicability.
method Combines time series prediction with adapted FBA for growing graphs.
result Efficacy demonstrated on synthetic and real datasets.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
Dynamic information balancing reduces catastrophic forgetting in modular neural networks.
problem Catastrophic forgetting in neural networks when learning multiple tasks.
method Dynamic Information Balancing (DIB) using reinforcement learning to adaptively route inputs based on module information load.
result DIB combined with EWC regularization outperforms models with similar capacity and EWC regularization.
Improves domain adaptation by clustering target representations.
problem Learning invariant and discriminative representations for unlabeled target domains.
method Simultaneously learns tightly clustered target representations and assigns each cluster to a unique class from the source.
result Achieves state-of-the-art performance in balanced, imbalanced, and partial domain adaptation.
Distributed Lion optimizes large model training by reducing communication costs.
problem Training large AI models efficiently with reduced communication costs.
method Adapted Lion optimizer for distributed training, using binary or lower-precision vectors for communication.
result Distributed Lion achieves comparable performance to standard optimizers but with significantly reduced communication bandwidth.
We address the problem of {\it adaptivity} in the framework of reproducing kernel Hilbert space (RKHS) regression. More precisely, we analyze estimators arising from a linear regularization scheme $g_\lam$. In practical applications, an important task is to choose the regularization parameter $\lam$ appropriately, i.e.…