Study shows gap between uniform convergence and test error in random feature models.
problem Understanding the gap between uniform convergence and test error in random feature models.
method Analytical expressions for uniform convergence over norm balls, interpolators, and minimum norm interpolator risk derived and proved.
result Uniform convergence over interpolators still gives a non-trivial bound of test error even when classical uniform convergence is vacuous.
This research analyzes the error convergence rate of GAN models.
problem Understanding the error convergence rate of GAN models.
method Applying Talagrand inequality and Borel-Cantelli lemma to establish a tight convergence rate.
result Established a tight convergence rate for the error of GAN models.
We consider binary classification problems with positive definite kernels and square loss, and study the convergence rates of stochastic gradient methods. We show that while the excess testing loss (squared loss) converges slowly to zero as the number of observations (and thus iterations) goes to infinity, the testing …
Paper proves convergence of Kalman filter on Stiefel manifolds with measurement errors.
problem Filtering constant particle with measurement errors on Stiefel manifolds.
method Extended Kalman filter applied to Stiefel manifold-valued observations.
result Convergence of the extended Kalman filter proved for constant system process.
Study improves least squares estimation for heavy-tailed errors.
problem Improving least squares estimation under heteroscedastic and heavy-tailed errors.
method Analyzes the rate of convergence of least squares estimator under bounded conditional variance and finitely many moments of errors.
result Upper bounds on rates of convergence of LSE for heavy-tailed errors are found.
Paper develops an online learning algorithm for functional data models.
problem Recovering slope functions or predictors in functional data models.
method Online regularized learning algorithm in reproducing kernel Hilbert spaces with polynomially decaying step-size.
result Established fast convergence rates for estimation error without capacity assumption.
Gradient descent with biased rounding errors converges faster under certain conditions.
problem Stagnation or negative impact of rounding errors in neural network training with low precision.
method Analysis of gradient descent with stochastic fixed-point rounding errors under the Polyak-Lojasiewicz inequality.
result Biased rounding errors can improve convergence rates, especially when the Polyak-Lojasiewicz inequality holds.
New bounds show multicalibration error is close to prediction error.
problem Addressing fairness in machine learning systems.
method Sample complexity bounds for uniform convergence of multicalibration error.
result Uniform convergence guarantees for multicalibration error, independent of prediction error.
Paper analyzes convergence of two time-scale stochastic approximation using martingale approach.
problem Analyzing convergence of two time-scale stochastic approximation algorithms.
method Uses martingale approach to establish convergence conditions and rates.
result Establishes different rates of convergence for fast and slow subsystems.
Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does conver…
Study shows exponential convergence in classification errors using random features and SGD.
problem Scalability issues in kernel methods for large datasets.
method Binary classification problem with random features and stochastic gradient descent.
result Exponential convergence rate of expected classification error achieved.
The paper explores why a specific type of predictor works well in noisy data.
problem Understanding why a specific type of predictor (minimum-norm interpolator) works well in noisy data.
method The paper uses uniform convergence and zero-error predictors in a norm ball to explain the success of the minimum-norm interpolator.
result The minimum-norm interpolator is consistent, and this can be explained by uniform convergence of zero-error predictors in a norm ball.
We consider stochastic gradient descent and its averaging variant for binary classification problems in a reproducing kernel Hilbert space. In the traditional analysis using a consistency property of loss functions, it is known that the expected classification error converges more slowly than the expected risk even whe…
Study on error rates for approximating rough volatility models.
problem Simulation of rough volatility models with fractional Brownian motion.
method Analysis of weak error rates for numerical schemes, focusing on fBm and cubic test functions.
result Convergence rates for approximations are ( 3 H + 1 2 ) ∧ 1 (3H+ \frac{1}{2}) \wedge 1 ( 3 H + 2 1 ) ∧ 1 for exact left-point discretization and H + 1 2 H+\frac{1}{2} H + 2 1 for hybrid schemes. Clip21 improves convergence of gradient-clipped methods in DP settings.
problem Gradient clipping introduces bias in distributed training, causing convergence issues.
method Clip21 designs an error feedback mechanism to mitigate bias in gradient-clipped methods.
result Clip21 converges at the same rate as distributed gradient descent, improving from previous $\mathcal{O}\left(\frac{1}{\sqrt{K}}
ight)$ to $\mathcal{O}\left(\frac{1}{K}
ight)$ .
New algorithm reduces feature count and accelerates error convergence.
problem Exponential error convergence in data classification with optimized random features.
method Optimized random features accelerated by quantum machine learning.
result Achieves exponential error convergence under low-noise condition.
Study on how sampling works for complex data functions.
problem Analyzing convergence of sampling algorithms for RKHS functions.
method Minimalistic assumptions on kernel and data, error estimates in RKHS norm, uniform convergence on compact domains.
result New convergence rates for Lipschitz and Hölder continuous kernels.
New analysis improves convergence guarantees for diffusion-based samplers in Wasserstein distance.
problem Improving convergence guarantees for diffusion-based generative models.
method Simple framework to analyze discretization, initialization, and score estimation errors.
result First Wasserstein convergence bound for the Heun sampler and improved results for Euler sampler.
This paper analyzes error feedback in compressed federated learning for non-convex optimization problems.
problem Reducing communication cost in federated learning with biased gradient compression.
method Proposes Fed-EF, a compressed federated learning scheme with error feedback, and analyzes its convergence rate and performance under partial client participation.
result Fed-EF can match the convergence rate of full-precision FL under data heterogeneity with a linear speedup and no extra slow-down factor due to stale error compensation.
Estimates and convergence of neural network approximations without structural assumptions.
problem Estimating and understanding the generalization error of neural networks.
method Introducing a new approach to estimate and analyze the convergence of neural network approximations.
result Estimates of the error without structural assumptions and convergence under mild regularity assumptions.
Proposes SGELU for neural networks to improve performance and convergence.
problem Improving neural network performance and convergence.
method Integrates Gaussian Error Linear Unit (GELU) with symmetrical characteristics to create SGELU.
result SGELU outperforms GELU and LiSHT in MNIST classification and auto-encoder tasks.
Optimal trading strategy with unobservable pricing errors for co-integrated assets.
problem Dynamic portfolio optimization of convergence trading with unobservable pricing errors.
method Modeling of convergence trading strategy with unobservable Markov-modulated pricing errors, extending Liu and Timmermann (2013) model.
result Characterization of optimal portfolio strategies in full and partial information settings.
GANs learn distributions well from samples, with rates depending on intrinsic dimension.
problem Learning distributions from samples using GANs.
method Oracle inequality, Hölder functions approximation, neural network approximation, integral probability metrics.
result Convergence rates of GANs depend on intrinsic dimension, not ambient dimension.
The paper analyzes kNN density estimation's convergence rates under different conditions.
problem Analyzing convergence rates of kNN density estimation under bounded and unbounded support conditions.
method Examined two cases: bounded support with known and unknown support sets, and unbounded support with smooth density function.
result kNN density estimation is minimax optimal under certain conditions and better than kernel density estimation in some cases.
Improved spectral convergence bounds for diffusion maps on tori.
problem Weak theoretical error bounds for diffusion maps.
method Spatial Hardy space estimates, PDE spectral stability, Sinkhorn weights.
result Matched pointwise error bounds for spectral data and operator convergence.
The paper provides convergence guarantees for multicalibration gradient boosting.
problem Understanding the convergence properties of multicalibration gradient boosting.
method Computational guarantees for multicalibration gradient boosting algorithms, including adaptive variants.
result The magnitude of successive prediction updates decays at O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) , leading to convergence in empirical multicalibration error. Improved BO algorithms reduce prediction error under Gaussian noise.
problem Reducing prediction error in Bayesian optimization with Gaussian noise.
method Established new prediction error bounds for Gaussian process under frequentist setting.
result Proved improved convergence rates of cumulative regret for GP-UCB and GP-TS.
Accelerates convergence in global non-convex optimization with reversible diffusion.
problem Global non-convex optimization challenges.
method Utilizes reversible diffusion processes with adaptive diffusion coefficients.
result Accelerated convergence with reduced discretization error.
Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate) and geometrically decreasing the step size after a number of iterations. It has been observed that the stagewise SGD has much faster conve…
LEAD algorithm speeds up decentralized optimization with compression.
problem Slow convergence and stability issues in decentralized optimization with compression.
method Proposes the first linearly convergent decentralized algorithm with compression.
result First consensus error bound for coupled dynamics of primal and dual updates.
Renormalized pruning improves neural network accuracy.
problem Over-parameterized neural networks waste many parameters.
method Propose renormalizing sparse neural networks.
result Renormalized pruning converges to zero error.
Recently, {\it stochastic momentum} methods have been widely adopted in training deep neural networks. However, their convergence analysis is still underexplored at the moment, in particular for non-convex optimization. This paper fills the gap between practice and theory by developing a basic convergence analysis of t…
Paper proposes an algorithm to reduce hypothesis space for faster convergence in high-dimensional settings.
problem Over-conservativeness of existing regularization approaches in high-dimensional settings.
method Empirical hypothesis space reduction to achieve faster convergence without dependence on the size of the hypothesis space.
result Achieves faster convergence of generalization error O ( log n / n ) O(\sqrt{\log n/n}) O ( log n / n ) independent of the dimension d d d . Improved TD learning reduces variance and bias errors.
problem Inefficient optimization variance in TD learning.
method Proposed a mathematically solid analysis of VRTD, showing linear convergence rate and reduced variance and bias errors.
result VRTD converges to a fixed-point solution with reduced variance and bias errors compared to vanilla TD.
Study shows rate of convergence for particle approximation of PDEs in Wasserstein space.
problem Analyzing convergence rates for particle approximations of PDEs in Wasserstein space.
method Backward stochastic differential equations techniques.
result Proved a rate of convergence of order 1/N for pathwise error and 1/sqrt(N) for L2-error on the derivative.
DFM models are analyzed for generating distributions with provable convergence.
problem Training DFM models to generate distributions that match true data.
method Theoretical analysis decomposes error into approximation and estimation errors.
result DFM models converge to true data distribution as training set size increases.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
EControl improves fast distributed optimization with compression and error control.
problem Stable convergence issues in distributed training with compression.
method Proposes EControl to regulate error compensation and prove fast convergence.
result Proves fast convergence for EControl in various convex settings without additional assumptions.
COS method convergence conditions expanded for heavy-tailed distributions.
problem Ensuring convergence of the COS method for various densities.
method Analyzing truncation error and providing conditions for convergence.
result Conditions for COS method convergence extended to include heavy-tailed distributions.
Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control pro…
We study convergence rates of variational posterior distributions for nonparametric and high-dimensional inference. We formulate general conditions on prior, likelihood, and variational class that characterize the convergence rates. Under similar "prior mass and testing" conditions considered in the literature, the rat…
MiMuon optimizer improves generalization for large models by reducing generalization error.
problem Improving generalization of Muon optimizer for large models.
method Enhanced Muon optimizer using orthogonalization of gradient, proving lower generalization error.
result MiMuon optimizer has a lower generalization error of O ( 1 N ) O\big(\frac{1}{N}\big) O ( N 1 ) compared to Muon's O ( 1 N κ T ) O\big(\frac{1}{Nκ^{T}}\big) O ( N κ T 1 ) . New gradient coding schemes reduce decoding error in both random and adversarial straggler settings.
problem Creating efficient approximate gradient coding schemes for distributed optimization.
method Introduced novel approximate gradient codes based on expander graphs, achieving optimal decoding coefficients.
result Achieved nearly optimal error in random setting and nearly half the error in adversarial setting compared to existing codes.
Paper addresses group synchronization with incomplete measurements and proves linear convergence of GPM.
problem Orthogonal group synchronization with incomplete measurements and additive noise.
method Generalized power method (GPM) with local error bound analysis.
result Linear convergence of GPM to a global maximizer under general additive noise model.
Entropy-regularized NPG converges linearly with linear function approximation.
problem Analyzing convergence of entropy-regularized NPG with function approximation.
method Established finite-time convergence analyses with entropy regularization and linear function approximation.
result Entropy-regularized NPG achieves linear convergence up to a function approximation error.
The overall performance or expected excess risk of an iterative machine learning algorithm can be decomposed into training error and generalization error. While the former is controlled by its convergence analysis, the latter can be tightly handled by algorithmic stability. The machine learning community has a rich his…
Sharp bounds on uniform generalization errors in binary linear classification.
problem Understanding the uniform generalization errors in binary linear classification.
method Isoperimetric arguments, Poincaré and log-Sobolev inequalities for joint distributions.
result Sharp concentration bounds on uniform generalization errors, almost sure convergence in broad settings.
Improved bounds for proximal gradient algorithms with computational errors.
problem Analyzing convergence of proximal gradient algorithms with inaccuracies.
method Deriving new tighter deterministic and probabilistic bounds for convex composite problems.
result Probabilistic bounds are more robust and accurate for algorithm verification and performance guarantees.