New NPG variants ensure parameter convergence in multi-agent learning.
problem Non-convergence of parameters in NPG for multi-agent learning.
method Proposed variants of NPG for multi-agent learning scenarios.
result Global last-iterate parameter convergence guarantees in various multi-agent learning settings.
Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
Paper proves EM algorithm convergence for mixtures of discrete and continuous parameters.
problem Nontrivial convergence analysis for EM algorithms with mixed-integer parameters.
method Introduces conditions for EM convergence in mixed-integer optimization.
result Proves convergence of EM-based sparse Bayesian learning algorithm.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Develops a parameter-free SGD algorithm with optimal convergence rate.
problem Optimizing parameters in stochastic convex optimization.
method A novel parameter-free algorithm for SGD with high-probability guarantees and adaptive properties.
result Achieves optimal convergence rate with only a double-logarithmic factor increase compared to known-parameter settings.
Study analyzes convergence of parameter estimation in contaminated mixture of experts.
problem Challenges in learning from prompts in large-scale models.
method Convergence analysis, distinguishability condition, partial differential equations.
result Comprehensive convergence rates and minimax lower bounds for parameter estimation.
Unsupervised pretraining and dropout have been well studied, especially with respect to regularization and output consistency. However, our understanding about the explicit convergence rates of the parameter estimates, and their dependence on the learning (like denoising and dropout rate) and structural (like depth and…
Singularities of a statistical model are the elements of the model's parameter space which make the corresponding Fisher information matrix degenerate. These are the points for which estimation techniques such as the maximum likelihood estimator and standard Bayesian procedures do not admit the root-n parametric rate…
We study the convergence of the Expectation-Maximization (EM) algorithm for mixtures of linear regressions with an arbitrary number k of components. We show that as long as signal-to-noise ratio (SNR) is Ω~(k), well-initialized EM converges to the true regression parameters. Previous results for k≥3 hav…
New convergence guarantees for learning with unknown nuisance parameters.
problem Learning problems with unknown nuisance parameters.
method Stochastic gradient optimization with Neyman orthogonality and approximately orthogonalized updates.
result Stochastic gradient algorithms can converge under conditions of nuisance parameters.
Paper extends a method to estimate Hurst parameter for rough stochastic volatility models.
problem Estimating Hurst parameter of rough stochastic volatility models from discrete observations.
method Extends a scale-invariant estimator to a general nonlinear function.
result Consistent estimation of Hurst parameter for a wide class of rough stochastic volatility models.
We provide a simple convergence proof for Adam and Adagrad.
problem Smooth objective functions with bounded gradients.
method Simple proof covering Adam and Adagrad.
result Explicit upper-bound on the squared norm of the objective gradient.
MPF method improves parameter estimation in probabilistic models.
problem Difficulty in fitting probabilistic models due to intractable partition function.
method Minimum Probability Flow (MPF) method for parameter estimation.
result MPF outperforms existing techniques in convergence time and accuracy.
This study improves convergence of two-timescale SA under Markovian noise in reinforcement learning.
problem Stability and convergence of two-timescale stochastic approximations under Markovian noise.
method Introduced a new control strategy for the fast timescale parameter.
result Established almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
The study uses the Merton model to estimate PD and finds a phase transition affecting convergence speed.
problem Estimating the probability of default (PD) using limited historical data.
method Adopted the Merton model and analyzed phase transitions in default correlation.
result PD estimation converges slowly when temporal correlation decays by power law less than one.
In this paper, we study a popular method for inference of the Bradley-Terry model parameters, namely the MM algorithm, for maximum likelihood estimation and maximum a posteriori probability estimation. This class of models includes the Bradley-Terry model of paired comparisons, the Rao-Kupper model of paired comparison…
ANIL adapts only a subset of parameters, reducing computational cost.
problem Efficiently adapt model parameters in meta-learning.
method Adapts only a small subset of parameters in the inner loop of ANIL.
result Theoretical convergence and computational complexity analysis for ANIL.
New framework for neural networks converging to low loss without overparameterization.
problem Training deep neural networks without overparameterization assumptions.
method Construction of random sparse lifts and analysis using algebraic topology and random graph theory.
result Provable convergence to low loss for large sparse neural networks.
We derive new formulas for the price of the European call and put options in the Black-Scholes model, under the form of uniformly convergent series generalizing previously known approximations. We also provide precise boundaries for the convergence speed and apply the results to the calculation of hedge parameters (Gre…
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
Newton's method converges faster than gradient descent in overparameterized neural networks.
problem Training neural networks efficiently in the overparameterized limit.
method Developed a convergence analysis for the regularized Newton method in this context.
result The NN training dynamics converge to the solution of a deterministic limit equation involving a Newton neural tangent kernel (NNTK).
Analyzes convergence rates for Gaussian-gated MoE model.
problem Theoretical understanding of Gaussian-gated MoE model is incomplete.
method Maximum likelihood estimation with novel Voronoi loss functions.
result MLE has distinct behaviors under different settings of Gaussian gating function parameters.
We revisit the problem of estimating the parameters of a partially observed diffusion process, consisting of a hidden state process and an observed process, with a continuous time parameter. The estimation is to be done online, i.e. the parameter estimate should be updated recursively based on the observation filtratio…
Chebyshev steps improve convergence in deep-unfolded gradient descent.
problem Improving convergence speed in iterative algorithms.
method Introducing Chebyshev steps to bound convergence rate of gradient descent.
result Chebyshev steps lead to asymptotically optimal convergence rate.
Deep neural networks with specific parameter sets can approximate smooth functions efficiently.
problem Approximating smooth functions with deep neural networks.
method Deep neural networks with ReLU activation and specific parameter sets {0,±21,±1,2} are used to approximate Cβ-smooth functions. result The constructed networks can approximate Cβ-smooth functions with parameters {0,±21,±1,2} efficiently, achieving the same convergence rate as sparse networks with parameters in [−1,1]. TiAda adapts adaptive gradient methods for nonconvex minimax optimization.
problem Nonconvex minimax optimization challenges in achieving convergence.
method TiAda is a time-scale adaptive GDA algorithm for nonconvex minimax optimization.
result TiAda achieves near-optimal complexities in deterministic and stochastic settings.
New method improves Robbins-Monro algorithm convergence with prior information.
problem Improving convergence speed of Robbins-Monro algorithm.
method Integrates prior information into Robbins-Monro iteration without regression model.
result Prior-information Robbins-Monro sequence converges faster than standard.
The alternating direction method of multipliers (ADMM) has been successfully applied to solve structured convex optimization problems due to its superior practical performance. The convergence properties of the 2-block ADMM have been studied extensively in the literature. Specifically, it has been proven that the 2-blo…
Recently, deep neural networks (DNNs) have shown advantages in accelerating optimization algorithms. One approach is to unfold finite number of iterations of conventional optimization algorithms and to learn parameters in the algorithms. However, these are forward methods and are indeed neither iterative nor convergent…
This work formalizes and extends parameter sharing in multi-agent reinforcement learning.
problem Parameter sharing limits multi-agent learning to a single policy, preventing different tasks or action spaces.
method Introduces agent indication and extends parameter sharing to heterogeneous observation and action spaces.
result Proves convergence to optimal policies for parameter sharing in heterogeneous environments.
New method improves convergence of SGD for heavy-tailed noise.
problem Heavy-tailed noise in machine learning applications.
method Normalized SGD (NSGD) to handle heavy-tailed noise.
result Improved sample complexity and high-probability convergence.
MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
Study shows consistency of shallow GCNNs on sampled point clouds under manifold assumption.
problem Consistency of shallow GCNNs on sampled point clouds under manifold assumption.
method Functional analysis perspective, weakly compact product of unit balls, Sobolev regularity, frequency cutoff.
result Proves Γ-convergence of regularized empirical risk minimization functionals and convergence of their global minimizers. New method uses sparse deep neural networks for high-dimensional regression with improved parameter estimation.
problem Improving parameter estimation in high-dimensional sparse regression models.
method Proposes nonparametric estimation of partial derivatives in sparse deep neural networks.
result Established convergence rate of nonparametric estimation of partial derivatives as O(n−1/4). Iterative thresholding algorithms are well-suited for high-dimensional problems in sparse recovery and compressive sensing. The performance of this class of algorithms depends heavily on the tuning of certain threshold parameters. In particular, both the final reconstruction error and the convergence rate of the algori…
The multinomial logistic regression (MLR) model is widely used in statistics and machine learning. Stochastic gradient descent (SGD) is the most common approach for determining the parameters of a MLR model in big data scenarios. However, SGD has slow sub-linear rates of convergence. A way to improve these rates of con…
New method for analyzing learning dynamics in singular models.
problem Challenges in analyzing learning of singular models with no one-to-one parameter space.
method Relative reparameterization technique to extract regular sub-models.
result Demonstrated differences in convergence behavior due to algorithmic and intrinsic aspects.
Muon optimizer outperforms GD in neural networks.
problem Optimizing matrix-structured parameters in neural networks.
method Muon optimizer specifically designed for matrix parameters, analyzing convergence rate and low-rank Hessian structure.
result Muon can outperform Gradient Descent due to its ability to leverage the low-rank structure of Hessian matrices.
Study reveals convergence properties of SGD with random learning rate.
problem Analyzing convergence of SGD with random learning rate in non-convex optimization.
method Introduced Poisson SGD with random learning rate and used stationary distribution analysis.
result Poisson SGD converges to a stationary distribution and finds global minima in non-convex optimization.
New method improves likelihood-free parameter estimation in complex models.
problem Estimating parameters in simulation-based models with unknown likelihood.
method Nested multi-time-scale stochastic approximation (NMTS) method.
result Eliminates bias and accelerates convergence in likelihood-free inference.
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss…
The adaptive moment estimation algorithm Adam (Kingma and Ba) is a popular optimizer in the training of deep neural networks. However, Reddi et al. have recently shown that the convergence proof of Adam is problematic and proposed a variant of Adam called AMSGrad as a fix. In this paper, we show that the convergence pr…
NSGD-M optimizes machine learning models without hyperparameter tuning, even under relaxed smoothness.
problem Training machine learning models with optimal complexity under relaxed smoothness assumptions.
method Normalized Stochastic Gradient Descent with Momentum (NSGD-M) without stepsize tuning.
result NSGD-M achieves nearly optimal complexity without prior knowledge of problem parameters.
The regularization and output consistency behavior of dropout and layer-wise pretraining for learning deep networks have been fairly well studied. However, our understanding of how the asymptotic convergence of backpropagation in deep architectures is related to the structural properties of the network and other design…
We study mechanisms to characterize how the asymptotic convergence of backpropagation in deep architectures, in general, is related to the network structure, and how it may be influenced by other design choices including activation type, denoising and dropout rate. We seek to analyze whether network architecture and in…
Study shows neural network parameters converge to ridgelet spectrum.
problem Characterization of local minima in over-parametrized neural networks.
method Developed ridgelet transform to analyze neural network parameters.
result Distribution of parameters converges to ridgelet spectrum.
New Riemannian radial distributions help estimate parameters on symmetric spaces.
problem Challenges in manifold data analysis due to lack of parametric distributions.
method Introduced Riemannian radial distributions on symmetric spaces, utilized symmetry, and developed M-estimators.
result MLE achieves root-n convergence rate up to logarithmic terms, demonstrating optimality.