Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
CWGD measures gradient diversity weighted by curvature, improving SGD convergence.
problem Gradient noise in high-curvature directions is underestimated by standard methods.
method CWGD weights gradient diversity by the inverse square root of the Hessian.
result CWGD-Cosine reduces optimization error by up to 20% compared to standard cosine annealing.
WSqD extends learning rate schedules for large model training without fixed horizons.
problem Fixed learning rate schedules limit training horizon extension.
method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.
result WSqD achieves minimax-optimal convergence rate and horizon-independence.
New method RECAST improves uncertainty calibration in neural networks.
problem Improving uncertainty estimation in neural networks for better predictions.
method Proposed RECAST method combining cosine annealing, warm restarts, and Stochastic Gradient Langevin Dynamics.
result RECAST offers the best calibrated measure of uncertainty compared to recent methods.
Robust Transformer-Based One-Step Stock Index Forecasting via Shifted Data Augmentation
problem Robust stock index forecasting
method Modified Transformer architecture with Shifted Data Augmentation
result Best performance on benchmark datasets
The study introduces anytime learning schedules for large language models without fixed horizons.
problem Training large language models without knowing the total training horizon.
method Theoretical analysis and weight averaging to create anytime learning schedules.
result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.
Study convergence of simulated annealing in continuous and discrete settings.
problem Analyzing convergence rate of simulated annealing methods.
method Apply Eyring-Kramers law to prove polynomial decay of tail probabilities.
result Explicit rate of convergence for continuous and discrete simulated annealing.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
Paper analyzes Annealed Langevin Dynamics for multimodal sampling stability.
problem Ensuring stability of Annealed Langevin Dynamics across dimensions.
method Uniform-in-dimension analysis of ALD for Gaussian-mixture targets.
result ALD achieves prescribed accuracy in KL divergence with spectral conditions.
Paper analyzes Langevin dynamics for multimodal Gaussian mixtures, controlling errors across dimensions.
problem Challenges in obtaining stable diffusion-based samplers in high- and infinite-dimensional settings.
method Study of preconditioned Annealed Langevin Dynamics (ALD) for Gaussian mixtures, focusing on Euler-Maruyama (EM) and exponential-integrator schemes.
result Proves dimension-uniform KL bounds for the exponential-integrator scheme, allowing arbitrarily small divergence with dimension.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final an…
The paper presents a multi-power law for predicting loss curves across different learning rate schedules.
problem Understanding and optimizing the relationship between model performance and hyperparameters, especially learning rates.
method Proposes a multi-power law that combines power laws based on the sum of learning rates and additional laws for loss reduction due to decay.
result The multi-power law accurately predicts loss curves for unseen learning rate schedules and finds a schedule that outperforms cosine learning rate.
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strate…
The COS method for European options pricing is improved with a new bound for the number of terms.
problem Determining the optimal number of terms in the COS method for accurate European option pricing.
method Using Fourier-cosine expansion, the study finds an explicit bound for the number of terms N in the cosine series approximation.
result The COS method achieves exponential convergence when the log-return density is smooth, but not when it has heavy tails.
Demon improves neural network training with a decaying momentum approach.
problem Improving neural network training efficiency and robustness.
method Proposes a decaying momentum ( extsc{Demon}) rule for neural network optimization.
result Demon achieves the highest number of Top-1 and Top-3 finishes across various settings and architectures.
Unified method for calculating financial option prices from characteristic functions.
problem Calculating financial option prices from characteristic functions in high dimensions.
method Damped Fourier-cosine expansion (COS) method.
result The method converges exponentially if the characteristic function decays exponentially.
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trai…
Novel RL-based NPG improves multi-objective NAS efficiency and performance.
problem Discovering optimal neural architectures with multiple conflicting objectives.
method Non-stationary policy gradient with adaptive reward functions and shared model.
result Framework efficiently approximates full Pareto front and achieves superior performance.
Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hil…
The paper proves convergence to global optima for a class of distributed algorithms for nonconvex optimization in network-based multi-agent settings. Agents are permitted to communicate over a time-varying undirected graph. Each agent is assumed to possess a local objective function (assumed to be smooth, but possibly …
Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.
problem Optimizing learning rates under functional scaling laws for model training.
method Deriving optimal learning-rate schedules based on exponents s and β. result Sharp phase transition between easy and hard tasks, with different decay behaviors.
Modified cosine distance improves similarity performance in data with variance and correlation.
problem Limitations of traditional cosine similarity in random variable spaces with variance and correlation.
method Proposed a variance-adjusted cosine distance metric to overcome limitations of traditional cosine similarity.
result Modified cosine distance shows 100% test accuracy in KNN model on the Wisconsin Breast Cancer Dataset.
The role of L2 regularization, in the specific case of deep neural networks rather than more traditional machine learning models, is still not fully elucidated. We hypothesize that this complex interplay is due to the combination of overparameterization and high dimensional phenomena that take place during training …
New method SF-AdamW trains large models without decay phases or memory overhead.
problem Inadequate fixed compute budgets for large-scale training.
method Schedule-Free (SF) method revisited and refined.
result SF-AdamW effectively navigates loss landscape without decay phases or memory overhead.
Improved MoE performance through perturbing cosine router.
problem Representation collapse and parameter redundancy in MoE models.
method Least square estimation of cosine router in MoE, followed by noise addition to improve convergence rates.
result Perturbed cosine router leads to polynomial convergence rates for MoE models.
Reverse annealing boosts quantum matrix factorization performance.
problem Improving quantum matrix factorization performance.
method Combining forward and reverse annealing for nonnegative/binary matrix factorization.
result Combination of forward and reverse annealing significantly improves performance.
We find optimal learning rate schedules for a random feature model.
problem Choosing optimal learning rates for deep learning models.
method We analyze a powerlaw random feature model trained with SGD, considering optimal schedules as numerical and analytical problems.
result We discover two regimes: easy and hard phases, with different optimal learning rate schedules.
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…
Researchers establish bounds and continuity of decomposed Möbius energies using cosine formula.
problem Estimating the bounds and continuity of decomposed Möbius energies.
method Using the cosine formula to evaluate upper and lower bounds and modulus of continuity of decomposed energies.
result Affirmative answer to the question of estimating decomposed energies using the cosine formula.
Cosine schedule is optimal for discrete diffusion models.
problem Choosing the best discretization schedule for diffusion models.
method Optimized using Fisher-Rao geometry.
result Cosine schedule is Fisher-Rao optimal.
Derives hyperbolic laws of cosines and sines with fermionic corrections.
problem Deriving hyperbolic laws of cosines and sines with new mathematical corrections.
method Using Minkowski supergeometry, the laws of cosines and sines are derived in the super hyperbolic plane.
result Identical formulae to classical cases with fermionic corrections for cosines and sines.
We empirically evaluate a stochastic annealing strategy for Bayesian posterior optimization with variational inference. Variational inference is a deterministic approach to approximate posterior inference in Bayesian models in which a typically non-convex objective function is locally optimized over the parameters of t…
GradPower speeds up language model training with minimal code changes.
problem Slower training of large language models.
method Elementwise sign-power transformation applied to gradients.
result Consistently lower terminal loss across various models and datasets.
Cosine similarity can force points to grow in magnitude, causing convergence issues.
problem Cosine similarity loss can lead to convergence issues in deep learning.
method Analyzing under-explored settings and proposing cut-initialization.
result Cosine similarity optimization forces points to grow in magnitude, leading to convergence issues.
GrokAlign aligns Jacobians to accelerate grokking in deep networks.
problem Accelerating the training dynamics of deep networks to avoid delayed generalisation and robustness.
method Aligning the Jacobians of a deep network with the training data to ensure grokking under a low-rank assumption.
result GrokAlign regularizes Jacobians to induce grokking sooner than conventional methods.
New method uses reinforcement learning to improve Simulated Annealing.
problem Optimization problems with unknown cost functions.
method Replaces Metropolis engine with Macau Algorithm.
result Effective heuristic for unknown cost functions.
We study the rigidity of polyhedral surfaces using variational principle. The action functionals are derived from the cosine laws. The main focus of this paper is on the cosine law for a non-triangular region bounded by three possibly disjoint geodesics. Several of these cosine laws were first discovered and used by Fe…
New method for European option pricing faster and more robust.
problem Pricing European options efficiently and accurately.
method Fourier cosine series expansions for models with known characteristic functions.
result More robust and faster than the original COS method.
aMCL uses annealing to improve hypothesis diversity in ambiguous tasks.
problem Limitations of Winner-takes-all in predicting plausible hypotheses.
method Combines simulated annealing with Multiple Choice Learning (MCL).
result Enhanced exploration of hypothesis space during training.
This paper presents studies on a deterministic annealing algorithm based on quantum annealing for variational Bayes (QAVB) inference, which can be seen as an extension of the simulated annealing for variational Bayes (SAVB) inference. QAVB is as easy as SAVB to implement. Experiments revealed QAVB finds a better local …
Quantum annealers aim at solving non-convex optimization problems by exploiting cooperative tunneling effects to escape local minima. The underlying idea consists in designing a classical energy function whose ground states are the sought optimal solutions of the original optimization problem and add a controllable qua…
Mathematical analysis shows annealing prevents mode collapse in Gaussian mixtures.
problem Mode collapse in variational inference for multimodal distributions.
method Analyzed annealing strategies for Gaussian mixtures, derived formulas, and tested on neural networks.
result Appropriately chosen annealing schemes can robustly prevent mode collapse.
Proposes a method to improve SLMC for multimodal distributions.
problem Difficulty of applying SLMC to multimodal distributions.
method Parallel adaptive annealing with VAE-SLMC.
result Can proficiently obtain accurate samples from multimodal distributions.
We introduce a novel framework for adversarial training where the target distribution is annealed between the uniform distribution and the data distribution. We posited a conjecture that learning under continuous annealing in the nonparametric regime is stable irrespective of the divergence measures in the objective fu…
Simulated annealing improves candidate optimization for multi-objective Bayesian optimization.
problem Efficient candidate optimization for multi-objective acquisition functions in Bayesian optimization.
method Simulated annealing-based approach for batch acquisition function optimization.
result Simulated annealing outperforms SLSQP in most multi-objective optimization problems, achieving higher hypervolume values and better convergence characteristics.
Paper studies convergence of Mean-Field GDA dynamics for MNE of continuous games.
problem Finding mixed Nash equilibria in continuous games.
method Two-scale Mean-Field Gradient Descent Ascent dynamics.
result Two-scale Mean-Field GDA converges exponentially to MNE without convexity assumptions.