Learning-rate schedules for large models match optimization theory closely, leading to better training.
problem Improving training of large models with optimal learning rates.
method Used a bound from non-smooth convex optimization theory to match learning-rate schedules with practical benefits.
result Extending the learning-rate schedule with optimal learning-rate and transferring it across schedules improves model training.
PAGE optimizes nonconvex problems with optimal convergence rates.
problem Nonconvex optimization problems.
method PAGE algorithm for achieving optimal convergence rates.
result PAGE achieves optimal convergence rates for nonconvex optimization.
Paper improves learning rates for SGD and NAG.
problem Generalization performance of stochastic optimization algorithms.
method Establishes new learning rates for SGD and NAG.
result Improved guarantees in some settings or comparable rates under weaker assumptions.
The paper analyzes how learning rate affects SGD and provides insights into optimal rates.
problem Understanding the impact of learning rate on stochastic gradient descent.
method Developed a learning-rate-dependent stochastic differential equation (lr-dependent SDE) to analyze SGD.
result Established a linear rate of convergence for SGD and found the optimal linear rate by analyzing the spectrum of the Witten-Laplacian.
Theory extends optimal learning rates without realizability assumption.
problem Agnostic binary classification without realizability assumption.
method Identifies tetrachotomy of optimal rates and combinatorial structures.
result Optimal universal rates for binary classification in agnostic setting.
Paper reconciles minimax rates and optimal recovery rates for noisy observations.
problem Estimating a function from noisy observations.
method Develops NLA minimax rates for Besov classes in Lq-norms. result NLA minimax rates continuously depend on noise level and match optimal recovery rates as noise decreases.
Optimal classification rules control error rates in multiclass mixture models.
problem Classifying observations in multiclass mixture models while controlling error rates.
method Finding optimal classification rules by searching an optimal region in the observation space, using Maximum A Posteriori (MAP) rule and heuristic computation.
result The FDR-like optimal rule can be significantly less conservative than thresholded MAP rules.
Study proposes optimal risk-aware interest rates for crypto lending protocols.
problem Determining optimal interest rates for decentralized lending protocols to maximize profit and minimize risk.
method Agent-based model, Riccati-type ODEs for linear behaviors, Monte-Carlo estimator and deep learning for nonlinear behaviors.
result Calibrated model shows superior risk-adjusted performance compared to industry-standard interest rate models.
Study optimizes dividend payout strategies under fluctuating interest rates.
problem Maximizing dividends under stochastic interest rates with negative values.
method Analytical HJB approach and backward SDEs for analysis.
result Explicit optimal strategies found for both time-dependent and strategy-independent stopping times.
Study examines how data augmentation impacts optimization in linear regression.
problem Understanding how data augmentation schedules affect optimization in linear regression.
method Analyzed the effect of augmentation on optimization in linear regression with MSE loss, using classical convex optimization and recent work on implicit bias.
result Proved that under certain joint schedules for learning rate and augmentation scheme, augmented gradient descent converges and characterized the resulting minimum.
Cyclical learning rate improves neural machine translation performance.
problem Optimizing learning rate for neural machine translation.
method Applied cyclical learning rate to transformer-based neural networks.
result Cyclical learning rate significantly impacts neural machine translation performance.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
A new method for optimizing non-decomposable metrics with constraints.
problem Optimizing complex machine learning objectives with thresholded constraints.
method Formulate rate-constrained optimization using the Implicit Function theorem and solve with gradient-based methods.
result Demonstrated effectiveness over existing methods on benchmark datasets.
This work analyzes the convergence rate of unrolling for optimizing quadratic objectives.
problem The challenge of accurately computing Jacobians through optimization.
method Non-asymptotic convergence-rate analysis of unrolled differentiation for gradient descent and Chebyshev method.
result There is a trade-off between fast asymptotic convergence and immediate but slower convergence due to the learning rate.
New insights into optimizing Local SGD's outer optimizer for faster convergence.
problem Understanding the impact of outer optimizer and its hyperparameters in Local SGD.
method Analyzing convergence guarantees with new outer learning rates and momentum.
result Tuning the outer learning rate can improve convergence and handle inner learning rate ill-tuning.
Optimizes portfolio growth rate for a behavioral investor considering terminal relative growth rate.
problem Optimizing a behavioral investor's portfolio growth rate under relative growth criterion.
method Martingale method, concavification, and quantile optimization techniques.
result Derives closed-form optimal growth rate and finds significant impact of benchmark growth rate.
Unified framework for ESG-inclusive portfolio optimization and pricing.
problem Incorporating ESG ratings into dynamic asset pricing theory.
method Introducing ESG-valued return as a linear transformation of financial and ESG scores, preserving traditional risk aversion with an ESG affinity parameter.
result Developed a more complex portfolio optimization problem in a space governed by reward, risk, and ESG score.
AutoGD automatically adjusts learning rates for gradient descent.
problem Optimizing learning rates for gradient descent methods.
method AutoGD automatically adjusts learning rates based on iteration.
result AutoGD can recover the optimal rate of GD for a broad class of functions.
The paper studies convergence rates of Tsallis entropic regularization in optimal transport.
problem Optimal transport with regularization.
method Γ-convergence and quantization/shadow arguments.
result Derives convergence rate of Tsallis entropic regularization.
Method calibrates local volatility and stochastic short rate models for equity-rate dynamics.
problem Joint calibration of local volatility and stochastic short rate models.
method Iterative approach using semimartingale optimal transport.
result Demonstrated performance on market data using European SPX options and cap interest rate options.
VAV method optimizes learning rate for faster, stable SGD convergence.
problem Optimizing learning rate for efficient and stable machine learning models.
method Energy-based self-adaptive learning rate with auxiliary variable r. result VAV method achieves faster convergence and superior stability with larger learning rates.
Optimizes identifying the best arm with fixed samples.
problem Finding the arm with the highest mean in a fixed number of samples.
method Characterizes minimax optimal rates and introduces algorithms to achieve them.
result Characterizes and introduces algorithms for optimal best arm identification.
Study on convergence rates for optimal transport with regularization.
problem Convergence analysis of divergence-regularized optimal transport.
method Novel methodology using quantization and martingale couplings.
result Sharp rates for various divergences and transport costs.
SALR improves deep learning generalization by dynamically adjusting learning rates.
problem Improving generalization in deep learning models.
method Sharpness-aware learning rate scheduling based on local loss function sharpness.
result SALR drives solutions to flatter regions, improving generalization and convergence.
We establish minimax optimal rates of convergence for estimation in a high dimensional additive model assuming that it is approximately sparse. Our results reveal an interesting phase transition behavior universal to this class of high dimensional problems. In the {\it sparse regime} when the components are sufficientl…
D-Adaptation automatically sets optimal learning rates without manual tuning.
problem Optimizing learning rates for efficient convergence in machine learning.
method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.
result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
Study confirms optimal minimax rate for nonlocal interaction kernel estimation.
problem Estimating nonlocal interaction kernels in interacting particle systems.
method Introduced tamed least squares estimator (tLSE) achieving optimal convergence rate.
result Optimal minimax rate of convergence confirmed for β≥1/4. Paper establishes universal lower bounds and optimal rates for clustering sub-exponential mixture models.
problem Achieving optimal error rates in clustering sub-exponential mixture models.
method Establishes universal lower bounds and demonstrates iterative algorithms' optimality in sub-exponential mixture models.
result Iterative algorithms achieve the universal lower bound in sub-exponential mixture models.
Develops a parameter-free SGD algorithm with optimal convergence rate.
problem Optimizing parameters in stochastic convex optimization.
method A novel parameter-free algorithm for SGD with high-probability guarantees and adaptive properties.
result Achieves optimal convergence rate with only a double-logarithmic factor increase compared to known-parameter settings.
In the present paper, we investigate the optimal capital injection behaviour of an insurance company if the interest rate is allowed to become negative. The surplus process of the considered insurance entity is assumed to follow a Brownian motion with drift. The changes in the interest rate are described via a Markov-s…
Analyzes optimal learning rate schedules in high-dimensional non-convex optimization problems.
problem Optimizing high-dimensional non-convex loss landscapes.
method Langevin optimization with learning rate decaying as \(η(t) = t^{-β}\).
result To speed up optimization without getting stuck in saddles, a decay rate \(β < 1\) is optimal, contrary to convex setups where \(β = 1\).
Study optimizes learning rates for conditional mean embedding estimates.
problem Consistency of kernel ridge regression for conditional mean embedding.
method Adaptive statistical learning rate derived for misspecified setting.
result Upper bound matches optimal O(logn/n) rates without assuming finite dimensionality. Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
Optimizes prediction error method for time-varying models.
problem Achieving optimal prediction error rates for time-varying models.
method Nonlinear least squares method for time-varying parametric models.
result First rate-optimal non-asymptotic analysis for time-varying models.
Modern online platforms rely on effective rating systems to learn about items. We consider the optimal design of rating systems that collect binary feedback after transactions. We make three contributions. First, we formalize the performance of a rating system as the speed with which it recovers the true underlying ran…
Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.
problem Optimizing learning rates under functional scaling laws for model training.
method Deriving optimal learning-rate schedules based on exponents s and β. result Sharp phase transition between easy and hard tasks, with different decay behaviors.
New algorithm AG-OG optimizes separable convex-concave problems efficiently.
problem Efficiently solving separable convex-concave minimax optimization problems.
method Leverages Nesterov acceleration and optimistic gradient on component and coupling parts of the problem.
result Achieves optimal convergence rate for various settings including bilinearly coupled problems.
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
Stochastic algorithm achieves sublinear convergence for bi-objective optimization.
problem Optimizing two conflicting functions using gradient or subgradient descent.
method Stochastic alternating algorithm with varying steps for each objective.
result Achieves sublinear convergence rate of O(1/T) under strong convexity.
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
GAIL with neural networks converges to global optima and has a known rate.
problem Uncertainty about GAIL with neural networks achieving global optimality.
method Gradient-based alternating updates algorithm.
result Established sublinear convergence to globally optimal solution.
Study problem-dependent rates in statistical learning theory, achieving optimal generalization error bounds.
problem Generalization error in statistical learning theory.
method Uniform localized convergence framework.
result Optimal generalization error bounds for various learning problems.
We consider the optimal dividend problem under a habit formation constraint that prevents the dividend rate to fall below a certain proportion of its historical maximum, the so-called drawdown constraint. This is an extension of the optimal Duesenberry's ratcheting consumption problem, studied by Dybvig (1995) [Review …
We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have opti…
Optimal rates for vector-valued regression on various norms.
problem Optimal rates for vector-valued ridge regression on continuous norms.
method Combining standard capacity assumptions with tensor product constructions of vector-valued interpolation spaces.
result Optimal rates for vector-valued ridge regression, independent of output space dimension.
Paper investigates optimal transport map estimation in infinite-dimensional spaces.
problem Estimating optimal transport maps in infinite-dimensional spaces is challenging.
method Characterizes γ-smoothness for optimal transport maps and develops a polynomial-rate estimator. result Shows polynomial-order minimax risk for optimal transport map estimation.