The paper provides convergence guarantees for multicalibration gradient boosting.
problem Understanding the convergence properties of multicalibration gradient boosting.
method Computational guarantees for multicalibration gradient boosting algorithms, including adaptive variants.
result The magnitude of successive prediction updates decays at O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) , leading to convergence in empirical multicalibration error. Empirical Bayes rates via variational approximations and prior decomposition.
problem Nonparametric and high-dimensional inference convergence rates.
method Variational perspective and prior decomposition.
result Empirical Bayes posterior rates derived from variational Bayes.
This paper studies convergence properties of multivariate distributions constructed by endowing empirical margins with a copula. This setting includes Latin Hypercube Sampling with dependence, also known as the Iman--Conover method. The primary question addressed here is the convergence of the component sum, which is r…
Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.
problem Convergence rate of empirical minimizer for unbounded functions with fast growth.
method Analyzes L 1 L^1 L 1 -distance convergence rate of the empiric minimizer for coercive functions sampled with noise. result Convergence rate is bounded above by a n n − 1 / q a_n n^{-1/q} a n n − 1/ q , where q q q is the dimension and a n = o ( n ε ) a_n = o(n^\varepsilon) a n = o ( n ε ) for every ε > 0 \varepsilon > 0 ε > 0 . We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
New metrics avoid high-dimensional analysis challenges, proving convergence without 'curse of dimensionality'.
problem High-dimensional analysis challenges in empirical measure convergence.
method Proposed a new class of probability metrics free of the curse of dimensionality.
result Convergence of empirical measures is free of the curse of dimensionality.
This work establishes uniform convergence of subdifferentials in stochastic optimization.
problem Understanding how empirical stationary points approximate population ones in nonsmooth, nonconvex stochastic optimization.
method Reduction principle for weakly convex stochastic objectives, focusing on subgradient convergence.
result Sharp uniform convergence rates for subdifferential mappings in stochastic convex-composite optimization.
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n − 1 / 4 n^{-1/4} n − 1/4 L2 convergence rate. Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …
We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …
The MEM method uses data-driven priors for linear inverse problems, proving convergence and estimating differences.
problem Linear inverse problems with approximate priors.
method Maximum Entropy on the Mean (MEM) method with data-driven priors.
result Empirical mean convergence and estimates for prior differences based on epigraphical distance.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
This paper studies the landscape of empirical risk of deep neural networks by theoretically analyzing its convergence behavior to the population risk as well as its stationary points and properties. For an l l l -layer linear neural network, we prove its empirical risk uniformly converges to its population risk at the rat…
Study sharp convergence rates of empirical UOT for spatio-temporal point processes.
problem Statistical analysis of UOT for spatio-temporal point processes.
method Empirical plug-in estimators for Kantorovich-Rubinstein distance between intensity measures.
result Sharp convergence rates of empirical UOT in terms of intrinsic dimensions of measures.
Study provides convergence rates for risk measure estimation.
problem Estimating risk measures from limited data.
method Plug-in estimation using empirical measures.
result Non-asymptotic convergence rates for risk measure estimation.
The paper analyzes the mean field Langevin dynamics and its convergence rate.
problem The convergence property of the mean field Langevin dynamics in the context of neural networks.
method The analysis uses a proximal Gibbs distribution and techniques from convex optimization.
result A concise convergence rate analysis of the mean field Langevin dynamics in both continuous and discrete time settings.
Adam-type optimizers show one-sided convergence in GAN training, not reaching critical points.
problem Theoretical understanding of Adam-type optimizers in non-convex non-concave min-max optimization.
method Empirical and theoretical analysis of Adam-type algorithms' convergence in GAN training.
result Adam-type algorithms converge to one-sided first order stationary points under the one-sided MVI condition.
Kernel learning FBSDE filter improves nonlinear filtering efficiency.
problem Nonlinear filtering problem in high-dimensional systems.
method Iterative and adaptive meshfree approach using forward backward SDE and KDE.
result Rigorous convergence analysis provided, supporting empirical results.
This work studies the smooth 1-Wasserstein distance and its limit distribution in high dimensions.
problem Addressing the curse of dimensionality in empirical approximation.
method Conducts a statistical study including limit distribution, bootstrap consistency, and concentration inequalities.
result Derives a nondegenerate limit distribution for empirical SWD, contrasting with classic W 1 W_1 W 1 . Recently, deep neural networks (DNNs) have shown advantages in accelerating optimization algorithms. One approach is to unfold finite number of iterations of conventional optimization algorithms and to learn parameters in the algorithms. However, these are forward methods and are indeed neither iterative nor convergent…
Selecting appropriate regularization coefficients is critical to performance with respect to regularized empirical risk minimization problems. Existing theoretical approaches attempt to determine the coefficients in order for regularized empirical objectives to be upper-bounds of true objectives, uniformly over a hypot…
New shuffling methods improve convergence without Lipschitz smoothness.
problem Lack of convergence guarantees for shuffling methods under non-Lipschitz conditions.
method Revisit shuffling methods, prove convergence under general bounded variance condition.
result Matched current best-known convergence rates without Lipschitz smoothness.
The paper studies Hodge Laplacians from point clouds, proving spectral convergence and harmonic form consistency.
problem Analyzing Riemannian submanifolds from point cloud data.
method Constructing deformed Hodge Laplacians and empirical operators from point clouds, proving convergence properties.
result Empirical spectral cluster contains the k k k -th Betti number and converges to harmonic k k k -forms. Paper analyzes convergence of DDPM for general distributions.
problem Theoretical understanding of DDPM's convergence properties remains limited.
method Introduced a relaxed smoothness condition and proved near-optimal convergence rates.
result Established a convergence rate of \( \widetilde{O}\left(\frac{d\min\{d,L^2\}}{T^2}
ight) \) in Kullback-Leibler divergence.
This paper is concerned with improving the empirical convergence speed of block-coordinate descent algorithms for approximate nonnegative tensor factorization (NTF). We propose an extrapolation strategy in-between block updates, referred to as heuristic extrapolation with restarts (HER). HER significantly accelerates t…
The paper provides convergence guarantees for VAEs using SGD and Adam.
problem Understanding theoretical convergence guarantees for VAEs.
method Derives non-asymptotic convergence rates for VAEs trained with SGD and Adam.
result Convergence rate of \(\mathcal{O}(\log n / \sqrt{n})\) with explicit hyperparameter dependencies.
New methods improve convergence in non-convex non-smooth learning problems.
problem Sparse learning from high-dimensional data with non-convex, non-smooth regularizers.
method Stochastic proximal gradient methods with arbitrary sampling.
result Independent sampling improves performance over uniform sampling.
New method uses robust estimators for Newton's method in empirical risk minimization.
problem Improving robustness in empirical risk minimization.
method Robust Newton's method with gradient and Hessian replaced by robust estimators.
result Faster convergence rates in high-dimensional settings.
We revisit the problem of inferring the overall ranking among entities in the framework of Bradley-Terry-Luce (BTL) model, based on available empirical data on pairwise preferences. By a simple transformation, we can cast the problem as that of solving a noisy linear system, for which a ready algorithm is available in …
The paper improves the empirical bootstrap method for non-normal estimators.
problem Theoretical properties of empirical bootstrap for non-asymptotically normal estimators.
method Establishing limiting distribution, deriving consistency conditions, proposing alternative methods.
result The empirical bootstrap method can be asymptotically consistent under stability conditions.
Gradient descent converges linearly in finite-width networks with positive NTK and compatible conditions.
problem Local convergence of gradient descent in finite-width networks.
method Positive Neural Tangent Kernel (NTK), local Polyak-Łojasiewicz inequality, fixed-step containment in Locally Quasi-Convex Region (LQCR).
result Linear convergence achieved under specific conditions.
New algorithms achieve uniform stability for empirical risk minimization.
problem Designing uniformly stable optimization algorithms for empirical risk minimization.
method Black-box conversion of smooth optimization algorithms and development of Mirror Descent for smooth optimization.
result Optimal algorithms with uniform stability and convergence rates for smooth optimization.
This work improves SGMs' convergence guarantees for semiconvex distributions with discontinuous gradients.
problem Establishing convergence guarantees for SGMs under weak regularity conditions.
method Developed non-asymptotic Wasserstein-2 convergence analysis for SGMs targeting semiconvex distributions with discontinuous gradients.
result Achieved optimal dependence of O ( d ) O(\sqrt{d}) O ( d ) on data dimension d d d and convergence rate of order one. Develops uniform convergence guarantees for a broad class of risk functionals in supervised learning.
problem Bounding generalization gaps for various risk functionals beyond the expectation.
method Establishes uniform convergence for Hölder risk functionals, providing guarantees for empirical risk minimization.
result First uniform convergence results for estimating the CDF of loss distributions, applicable to various risk functionals.
Gradient descent with noise converges to a unique optimum in nonconvex matrix factorization.
problem Gradient descent with noise converges to a unique optimum in nonconvex matrix factorization.
method A perturbed form of gradient descent with arbitrary initialization.
result Gradient descent with noise converges to a unique optimum.
Paper shows robust estimators converge to true risk minimizers at optimal rates.
problem Understanding asymptotic properties of robust risk minimizers.
method Investigates robust analogues of empirical risk minimization, focusing on median of means estimator.
result Robust minimizers converge to true minimizers at optimal rates and have similar asymptotic variance.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
The aim of this paper is to present a further contribution to the analysis of absolute convergence (and), associated with the neoclassical theory, and conditional, associated with endogenous growth theory, of the sectoral productivity at regional level. Presenting some empirical evidence of absolute convergence of prod…
New method uses SURE to denoise signals, outperforming NPMLE.
problem Learning to optimally denoise signals corrupted by Gaussian noise.
method Hyvärinen's score matching (SM) is shown equivalent to SURE minimization.
result SURE achieves nearly parametric rates of convergence in empirical Bayes settings.
New insights into convergence of optimization methods for DAG structure learning.
problem Unclear convergence properties of optimization methods for structure learning.
method Examined the convergence of augmented Lagrangian method (ALM) and quadratic penalty method (QPM) for structure learning.
result Standard convergence result of ALM does not hold in various cases, and QPM is prone to ill-conditioning.
Algorithm minimizes risk for multiclass classification of stochastic diffusion paths.
problem Multiclass classification of stochastic diffusion paths with distinct drift functions.
method Empirical risk minimization using L2 risk.
result Achieves fast rates of convergence under margin assumption.
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large …
In recent years, unfolding iterative algorithms as neural networks has become an empirical success in solving sparse recovery problems. However, its theoretical understanding is still immature, which prevents us from fully utilizing the power of neural networks. In this work, we study unfolded ISTA (Iterative Shrinkage…
Global convergence of SGD proven for two-layer neural nets with regularization.
problem Proving global convergence of SGD for two-layer neural nets.
method Regularized empirical risk, SGD iterates, Villani functions.
result Global convergence of SGD for a special class of initializations.
New convergence rates for shuffling gradient methods without strong convexity.
problem Theoretical gap between shuffling gradient methods' empirical success and established convergence rates.
method Proved last-iterate convergence rates for shuffling gradient methods using function value gap.
result First last-iterate convergence rates for shuffling gradient methods without strong convexity.
TSAW improves MCMC integral estimation with faster convergence.
problem Estimating integrals using MCMC with standard random walks is slow.
method Introduces TSAW to penalize overuse in finite-state adaptive sampling.
result TSAW-based estimators converge faster, achieving O ( log t / t ) O(\sqrt{\log t}/t) O ( log t / t ) error. learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
New L2 regularization improves softmax MAB performance.
problem Improving softmax MAB performance with vanishing regularization.
method L2 regularization with vanishing parameter analyzed and proven convergent.
result Vanishing L2 regularization makes softmax MAB more numerically advantageous.