The paper optimizes k-NN for distributed learning with minimax optimal performance.
problem Minimizing error rates in classification, regression, and density estimation.
method Optimal aggregation of fixed-k nearest neighbors from multiple subsets of data.
result Achieves minimax optimal error rates up to a logarithmic factor.
Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed eit…
Adaptive OMD reduces variance in learning optimal strategies for imperfect information games.
problem High variance in learning optimal strategies for imperfect information games.
method Fixed sampling approach with locally applied Online Mirror Descent (OMD) algorithm.
result Convergence rate of ildeO(T−1/2) with high probability. Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
WSqD extends learning rate schedules for large model training without fixed horizons.
problem Fixed learning rate schedules limit training horizon extension.
method WSqD replaces constant stable phase with a shifted inverse-square-root base, retaining linear cooldown.
result WSqD achieves minimax-optimal convergence rate and horizon-independence.
New method accelerates optimization in fixed time, improving convergence rates.
problem Optimization in large-scale data-driven problems.
method Gradient-based optimization framework with fixed-time stable dynamical systems.
result Achieves convergence to the optimizer in a fixed number of iterations, independent of initialization.
A simplified model for fixed income portfolio optimisation.
problem Modeling interest rates and credit risk in fixed income portfolios.
method Proposes a two-factor model for the time evolution of the efficient frontier.
result The efficient frontier is mainly controlled by linear constraints, with standard deviation less important.
The study introduces anytime learning schedules for large language models without fixed horizons.
problem Training large language models without knowing the total training horizon.
method Theoretical analysis and weight averaging to create anytime learning schedules.
result Theoretical and empirical evidence shows that weight averaging with simple step sizes can achieve comparable final loss to well-tuned cosine schedules.
Cyclical learning rates improve DRL performance without manual tuning.
problem Manual hyperparameter tuning in DRL is time-consuming and error-prone.
method Proposes cyclical learning rates for DRL problems.
result Cyclical learning achieves similar or better results than fixed learning rates.
We consider the problem of learning a non-negative linear classifier with a 1-norm of at most k, and a fixed threshold, under the hinge-loss. This problem generalizes the problem of learning a k-monotone disjunction. We prove that we can learn efficiently in this setting, at a rate which is linear in both k and…
Optimizer memory affects learning rate sensitivity in shuffle order, impacting fine-tuning noise.
problem Optimizer memory affects the learning rate sensitivity in shuffle order, leading to fine-tuning noise.
method Isolated the mechanism of fixed-clock optimizer memory affecting the learning rate sensitivity in shuffle order, deriving a fit-free way to size the noise.
result Fixed-clock optimizers like AdamW produce a larger first-order noise channel compared to memoryless optimizers, affecting fine-tuning comparisons.
Optimizes identifying the best arm with fixed samples.
problem Finding the arm with the highest mean in a fixed number of samples.
method Characterizes minimax optimal rates and introduces algorithms to achieve them.
result Characterizes and introduces algorithms for optimal best arm identification.
Gradient descent forces neural network eigenvalues to a specific threshold.
problem Understanding why gradient descent drives eigenvalues to a specific threshold.
method Introduced edge coupling, a functional on consecutive iterate pairs, to explain the trajectory towards the eigenvalue threshold.
result Gradient descent forces the Hessian eigenvalue to the threshold 2/η from arbitrary initialization. Study efficient derivative computation for nondifferentiable maps in machine learning.
problem Efficiently compute derivatives of fixed-point of nondifferentiable contractions.
method Iterative Differentiation (ITD), Approximate Implicit Differentiation (AID), and New Stochastic Implicit Differentiation (NSID).
result Established convergence rates for ITD, AID, and NSID, matching or improving smooth setting rates.
A new method for optimizing non-decomposable metrics with constraints.
problem Optimizing complex machine learning objectives with thresholded constraints.
method Formulate rate-constrained optimization using the Implicit Function theorem and solve with gradient-based methods.
result Demonstrated effectiveness over existing methods on benchmark datasets.
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
We study two time-scale linear stochastic approximation algorithms, which can be used to model well-known reinforcement learning algorithms such as GTD, GTD2, and TDC. We present finite-time performance bounds for the case where the learning rate is fixed. The key idea in obtaining these bounds is to use a Lyapunov fun…
Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.
problem Optimizing learning rates under functional scaling laws for model training.
method Deriving optimal learning-rate schedules based on exponents s and β. result Sharp phase transition between easy and hard tasks, with different decay behaviors.
New findings on complexity limits in fixed budget bandit identification.
problem Determining the best possible error rate for fixed budget bandit identification.
method Analyzing the best non-adaptive sampling procedures and showing the existence of complexities.
result No fixed complexity for certain bandit identification tasks.
Unified framework for isotropic SG noise in posterior sampling.
problem Bayesian posterior sampling with practical and robust methods.
method Designing a novel, isotropic SG noise approach with fixed learning rate.
result Competitive and practical method compared to state-of-the-art.
FedSplit improves federated learning by ensuring correct convergence to optimal solutions.
problem Federated learning's fixed points do not always correspond to optimal solutions in simple convex settings.
method FedSplit uses operator splitting procedures to solve distributed convex minimization problems with additive structure.
result FedSplit ensures that the fixed points correspond to optima of the original optimization problem.
Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured pro…
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
Stochastic delay differential equations (SDDE's) have been used for financial modeling. In this article, we study a SDDE obtained by the equation of a CIR process, with an additional fixed delay term in drift; in particular, we prove that there exists a unique strong solution (positive and integrable) which we call fix…
We show that if F is a convex class of functions that is L-subgaussian, the error rate of learning problems generated by independent noise is equivalent to a fixed point determined by `local' covering estimates of the class, rather than by the gaussian averages. To that end, we establish new sharp upper and lower e…
DFIV uses deep features for IV regression, achieving optimal rates.
problem Optimal IV regression with deep features for complex target functions.
method Two-stage approach: deep feature learning followed by IV regression.
result DFIV achieves minimax optimal learning rate under certain conditions.
Exact risk and learning rate curves derived for adaptive SGD on high-dimensional problems.
problem Analyzing risk and learning rate dynamics in high-dimensional optimization problems.
method Developed a framework to give exact expressions for risk and learning rate curves using ODEs.
result Exact expressions for risk and learning rate curves, with detailed analysis of two adaptive learning rates.
Gradient descent-based optimization methods underpin the parameter training of neural networks, and hence comprise a significant component in the impressive test results found in a number of applications. Introducing stochasticity is key to their success in practical problems, and there is some understanding of the rol…
Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank…
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
Empirical study on SGD hyperparameters and adversarial robustness.
problem Effect of SGD hyperparameters on adversarial robustness and generalization.
method Empirical observation of learning rate, batch size, and momentum effects on adversarial robustness and generalization.
result Constant learning rate to batch size ratio leads to good generalization and almost constant adversarial robustness.
Study on discrepancy principle for learning algorithms in nonparametric regression.
problem Determining optimal iteration number in nonparametric regression with unknown optimal iteration.
method Investigates discrepancy principle and modified principles for kernelized spectral filters, using deviation inequalities and change-of-norm arguments.
result Classical discrepancy principle is adaptive for slow rates, while modified principles are adaptive for faster rates.
Warm-up improves training by adapting learning rate based on curvature.
problem Improving training efficiency in deep learning models.
method Introducing a curvature condition to explain warm-up, and showing empirically that it leads to faster convergence.
result Adapting learning rate based on curvature condition naturally induces warm-up-like schedule, leading to faster convergence.
Step sizes in neural network training are largely determined using predetermined rules such as fixed learning rates and learning rate schedules. These require user input or expensive global optimization strategies to determine their functional form and associated hyperparameters. Line searches are capable of adaptively…
Efficient global optimization is the problem of minimizing an unknown function f, using as few evaluations f(x) as possible. It can be considered as a continuum-armed bandit problem, with noiseless data and simple regret. Expected improvement is perhaps the most popular method for solving this problem; the algorithm pe…
Study optimal portfolio strategies with time-varying discount rates.
problem Optimizing portfolio decisions with a non-constant discount rate.
method Introduced subgame perfect strategies to handle time inconsistency, using fixed point iteration to find the utility-weighted discount rate.
result Subgame perfect strategies are equivalent to optimal strategies under certain utility function assumptions.
This paper studies universal rates of ERM for binary classification under agnostic learning.
problem The challenge of achieving universal rates of ERM for binary classification under agnostic learning.
method The paper explores the agnostic universal rates of ERM for binary classification, revealing three possible rates: e−n, o(n−1/2), or arbitrarily slow. result The paper provides a complete characterization of which concept classes fall into each of the three categories of agnostic universal rates.
Develops accelerated fixed-point methods with delayed oracles for scientific computing.
problem Approximating fixed points of nonexpansive operators.
method Combines Nesterov's acceleration and KM iteration with delayed inexact oracles.
result Establishes improved convergence rates for fixed-point approximation.
New research shows fixed-budget best-arm identification cannot match static oracle performance.
problem Fixed-budget best-arm identification's performance limitations.
method Analysis of various adaptive and static algorithms for best-arm identification.
result For any algorithm, there exists at least one instance where the error decay rate is at most \((1 + \frac{\log(K)}{8})^{-1}\) times that of the static oracle.
Paper defines when early exercise of American options is optimal under negative rates.
problem Determining optimal exercise times for American options with negative interest rates.
method Developed a new integral equation to price options and find exercise boundaries under negative rates, using modified fixed point method.
result Successfully developed and validated a new algorithm for pricing American options under negative rates.
SGD works well with large learning rates at the edge of stability.
problem Stochasticity at the edge of stability in deep learning.
method Sharp convergence guarantees for SGD with multiclass cross-entropy loss.
result SGD self-stabilizes, ensuring convergence with large learning rates.
Study uses ML to analyze how interest rates affect fund returns, finding gradient boosting is effective.
problem Understanding how interest rate changes impact fund returns.
method Combines Machine Learning and causal inference, using Double Machine Learning framework.
result Gradient boosting is useful for predicting fund returns, showing a significant negative effect of interest rate increases.
New method for learning indirectly through control variables.
problem Learning relationships when direct manipulation of variables is impossible.
method Study of indirect active learning under nonparametric models with fixed budget.
result Minimax rates for estimating relationships between variables.
Paper studies MCCR models with scale parameters tending to zero, revealing optimal learning rate and comparing robustness.
problem Analyzing MCCR models with scale parameters approaching zero.
method Investigates MCCR models with scale parameters tending to zero, revealing optimal learning rate and comparing robustness.
result Optimal learning rate of MCCR models is O(n−1) in the asymptotic sense. In this paper, asymptotic results in a long-term growth rate portfolio optimization model under both fixed and proportional transaction costs are obtained. More precisely, the convergence of the model when the fixed costs tend to zero is investigated. A suitable limit model with purely proportional costs is introduced …
Study expands multiclass classification models with new rates and partial concept classes.
problem Multiclass classification with a bounded number of labels under various conditions.
method Extends traditional PAC model to distribution-dependent and data-dependent learning rates, characterizes optimal rates for universal and partial concept classes.
result Characterizes three types of learning rates (exponential, linear, arbitrarily slow) for fixed distributions and complexity measures for partial concept classes.
Enhances swaption modeling with rough stochastic volatility.
problem Modeling swaption volatility in post-LIBOR markets.
method Introduces rough stochastic volatility into FMM and rigorously justifies the freezing approximation.
result Establishes a new framework connecting FMM to rough Bergomi for forward swap rates.