Study max- and min-stability under first-order stochastic dominance, finding new functional characterizations.
problem Understanding max- and min-stability in stochastic dominance.
method Representation theorem for functionals satisfying max-stability, combining max- and min-stability to define Lambda-quantiles.
result New characterizations of functionals, including Lambda-quantiles, in finance and political science.
The paper guarantees global stability for stochastic subgradient methods in nonsmooth nonconvex optimization.
problem Minimizing nonsmooth nonconvex functions with convergence guarantees.
method Developed a framework for stochastic subgradient methods with global stability guarantees.
result Iterates are uniformly bounded and asymptotically stabilize around the stable set of the differential inclusion.
Paper analyzes stability and generalization of SCO algorithms.
problem Understanding how SCO algorithms perform on unseen data.
method Algorithmic stability analysis in statistical learning theory.
result Derives dimension-independent excess risk bounds for SCGD and SCSC.
The paper analyzes stability and generalization of decentralized SGD.
problem Stability and generalization of decentralized stochastic gradient descent.
method Novel formulation of decentralized stochastic gradient descent combined with non/convex optimization theory.
result First stability and generalization guarantees for decentralized stochastic gradient descent.
Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.
problem Improving convergence and stability of stochastic optimization methods.
method Developed and analyzed the convergence and stability of the stochastic proximal point algorithm with momentum (SPPAM).
result SPPAM converges faster and is more stable than standard stochastic proximal point algorithm (SPPA) and stochastic gradient descent with momentum (SGDM).
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.
SGD works well with large learning rates at the edge of stability.
problem Stochasticity at the edge of stability in deep learning.
method Sharp convergence guarantees for SGD with multiclass cross-entropy loss.
result SGD self-stabilizes, ensuring convergence with large learning rates.
Paper proves stability of complex equations under various conditions.
problem Stability of backward stochastic differential equations with jumps.
method General framework for convergent sequences of data and solutions.
result Convergent sequence of solutions for associated data.
Momentum affects optimization differently at small vs large batch sizes near instability.
problem Understanding how momentum impacts optimization near the edge of stability.
method Demonstrated through batch-size dependent behavior of SGD with momentum.
result Momentum operates in two distinct regimes: amplifying stochastic fluctuations at small batch sizes and stabilizing at large batch sizes.
RELTA-SGLD stabilizes nonconvex SGLD updates with a lighter taming scheme.
problem Stabilizing superlinear stochastic-gradient updates in nonconvex optimization.
method Threshold-based taming with relative-growth principle for stability.
result Polynomial moment stability and first-order stationary accuracy in nonconvex SGLD.
Paper proposes a new method to stabilize noisy gradient algorithms.
problem Stochastic-gradient Langevin algorithms can introduce bias when taming denominators depend on stochastic-gradient realizations.
method Proposes a structure-preserving framework for designing tamed denominators that avoid unnecessary taming and maintain the stabilizing effect of taming.
result The method avoids stationary bias and explains the stationary error split into bias and remaining error.
Bayesian algorithm stabilizes unknown continuous-time systems from unstable data.
problem Learning and stabilizing unknown continuous-time systems with uncertain dynamics.
method Bayesian learning algorithm that learns from unstable data to stabilize the system in finite time.
result The algorithm stabilizes unknown continuous-time stochastic linear systems effectively after a short time period.
This paper analyzes stability and generalization of Markov chain stochastic gradient methods.
problem Analyzing stability and generalization of Markov chain stochastic gradient methods.
method Algorithmic stability in statistical learning theory.
result Established optimal generalization bounds for both smooth and non-smooth cases.
Enhanced stability improves privacy in machine learning.
problem Improving privacy in machine learning training while maintaining accuracy.
method Study of stability in private empirical risk minimization, focusing on strongly-convex loss functions and uniform stability.
result An algorithm with uniform stability of β implies a bound of O(√β) on the scale of noise required for differential privacy.
Develops a new theory for neural systems stability and width effects.
problem Stability and finite-width effects in deep neural systems.
method Gauge-covariant stochastic effective field theory using classical commuting fields.
result Predicts the edge of chaos and low-frequency spectral deformation.
Paper relaxes stability and generalization assumptions for SGD.
problem Stability and generalization for SGD under restrictive assumptions.
method Introduces on-average model stability and develops novel bounds.
result First-ever-known fast bounds in low-noise setting using stability approach.
Paper improves generalization bounds for noisy stochastic algorithms.
problem Improving generalization bounds for noisy stochastic algorithms.
method Introduces Exponential Family Langevin Dynamics (EFLD) and establishes data-dependent expected stability based generalization bounds.
result Sharp generalization bounds with O(1/n) sample dependence and gradient discrepancy.
New method reveals insights about stochastic optimization methods using modified equations.
problem Understanding the qualitative behavior of stochastic optimization algorithms.
method Developed a class of stochastic differential equations to approximate the dynamics of stochastic optimization methods.
result Mean-square stability of the modified equation provides qualitative insights about stochastic coordinate descent.
The paper analyzes generalization bounds for NC-SC/NC-C stochastic minimax optimization.
problem Generalization analysis of nonconvex-(strongly)-concave stochastic minimax optimization.
method Established algorithm-agnostic and algorithm-dependent generalization bounds via uniform convergence and stability arguments.
result Sample complexities and generalization bounds for NC-SC and NC-C settings.
New bounds improve generalization in learning scenarios.
problem Limitations of existing information-theoretic bounds in SCO problems.
method Sample-conditioned hypothesis stability and neighboring-hypothesis matrix.
result Sharper generalization guarantees in various learning scenarios.
Paper explores stability, regularization, and gradient flows for stochastic inverse problems.
problem Recovering random probability distributions from measurements.
method Direct inversion, variational formulation with regularization, and optimization via gradient flows.
result The choice of metric impacts stability and properties of the optimizer.
New analysis improves understanding of bilevel optimization stability and generalization.
problem Understanding how well bilevel optimization algorithms generalize.
method Algorithmic stability arguments and generalization bounds for three bilevel minimax solvers.
result Precise trade-off between algorithmic stability, generalization gaps, and practical settings.
Studied how heavy-tailed behavior affects SGD's generalization in quadratic optimization.
problem Link between heavy-tailed behavior and generalization in SGD.
method Used heavy-tailed stochastic differential equation and proved stability bounds.
result Stability of SGD depends on the loss function's tail behavior.
Paper explores using bootstrap methods to improve SGD's stability and robustness.
problem Improving the stability and robustness of SGD.
method Investigates empirical bootstrap approaches for SGD from algorithmic stability and statistical robustness perspectives.
result Demonstrates construction of purely distribution-free confidence intervals using bootstrap SGD.
New framework improves worst-case generalization bounds for stochastic optimization.
problem Challenges in providing generalization guarantees for stochastic optimization algorithms.
method Introduces random set stability and empirically relevant complexity measures to avoid intractable mutual information terms.
result Bounded worst-case generalization error in terms of random set stability and empirically relevant complexity measures.
The paper analyzes stability of random matrix products with Markovian noise.
problem Analyzing stability of random matrix products with Markovian noise.
method Using a super-Lyapunov drift condition and controlled growth of matrix-valued functions, the paper provides an exponential stability result for the p-th moment of random matrix product.
result Finite-time p-th moment bounds for linear stochastic approximation and TD learning algorithms.
Stochastic momentum methods have been widely adopted in training deep neural networks. However, their theoretical analysis of convergence of the training objective and the generalization error for prediction is still under-explored. This paper aims to bridge the gap between practice and theory by analyzing the stochast…
SIM-Shapley improves SV approximation efficiency and stability.
problem High computational costs of Shapley value methods in high-dimensional settings.
method Stochastic Iterative Momentum for Shapley Value Approximation (SIM-Shapley).
result Reduced computation time by up to 85% while maintaining feature attribution quality.
Noise in RNNs promotes flatter minima and more stable dynamics.
problem Understanding and optimizing the training of RNNs with noise.
method Formalizing RNNs as stochastic differential equations and analyzing the effect of noise in the hidden states.
result Noise injection in RNNs leads to flatter minima, more stable dynamics, and improved robustness.
In this paper we measured the stability of stochastic gradient method (SGM) for learning an approximated Fourier primal support vector machine. The stability of an algorithm is considered by measuring the generalization error in terms of the absolute difference between the test and the training error. Our problem is to…
Studied how SGD's stability regularization affects generalization in neural networks.
problem Understanding why SGD often generalizes better than GD in neural networks.
method Analyzed stability of SGD and GD through Frobenius norm and trace of Hessian, and compared their generalization properties.
result Stable minima of SGD generalize well, while GD's stability-induced regularization is too weak.
Unified stability bounds for noisy SGD across convex and non-convex losses.
problem Deriving generalization bounds for noisy stochastic gradient descent.
method Unified approach using Lyapunov functions and applied probability.
result Time-uniform stability bounds for SGD on various loss functions.
New stability bounds for SGD on nonsmooth convex losses.
problem Understanding stability of SGD on nonsmooth convex losses.
method Sharp upper and lower bounds for SGD and full-batch GD on nonsmooth convex losses.
result SGD can be less stable but still useful for generalization bounds.
Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.
problem Understanding the impact of stochasticity and large stepsizes on gradient descent and SGD solutions.
method Investigation of GD and SGD over diagonal linear networks with macroscopic stepsizes, proving convergence and characterizing solutions.
result Large stepsizes consistently benefit SGD for sparse regression problems, but can hinder GD recovery of sparse solutions, especially in the edge of stability regime.
The need for parameter estimation with massive datasets has reinvigorated interest in stochastic optimization and iterative estimation procedures. Stochastic approximations are at the forefront of this recent development as they yield procedures that are simple, general, and fast. However, standard stochastic approxima…
We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimizat…
Optimism stabilizes Thompson Sampling for adaptive inference in multi-armed bandits.
problem Subtle inferential properties of Thompson Sampling under adaptive data collection.
method Introduced optimism as a key mechanism to restore stability and validity of inference.
result Suitably implemented optimism stabilizes Thompson Sampling and enables asymptotically valid Wald inference.
This work analyzes the stability of graph filters under large perturbations.
problem Stability of graph filters under large edge rewires.
method Proves a bound on stability using frequency response and community structure.
result Graph filter stability depends on perturbation to community structure.
New method stabilizes saddle-point optimization with unbounded gradients.
problem Stochastic saddle-point optimization faces instability due to large gradients.
method Proposes a regularization technique to stabilize iterates.
result Yields meaningful performance guarantees even with unbounded gradients.
ULES embeds dynamic networks with stability guarantees.
problem Stability of time-varying node embeddings in evolving networks.
method Unfolded Laplacian Spectral Embedding (ULSE) using normalized Laplacian operators.
result ULES satisfies cross-sectional and longitudinal stability under dynamic stochastic block model.
Study uses VIX for zero-coupon Treasury rates, proving long-term stability and returns.
problem Modeling zero-coupon Treasury rates with VIX for volatility.
method Multivariate autoregressive stochastic volatility model, proving stability and Law of Large Numbers.
result VIX accurately models zero-coupon Treasury rates and returns.
Unified treatment of RC in stochastic and deterministic settings.
problem Understanding and generalizing reservoir computing in both deterministic and stochastic contexts.
method Investigation of state-space systems, analysis of fading memory and solution stability, introduction of stochastic echo states.
result Generality of fading memory and solution stability in state-space systems, even without the echo state property.
New algorithms help machines forget old data efficiently.
problem Machine learning models can retain old data, hindering new learning.
method Developed TV-stable algorithms based on noisy SGD for convex and non-convex functions.
result Achieved efficient unlearning with upper and lower bounds on risk.
In this paper we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking,…
We introduce a notion of algorithmic stability of learning algorithms---that we term \emph{argument stability}---that captures stability of the hypothesis output by the learning algorithm in the normed space of functions from which hypotheses are selected. The main result of the paper bounds the generalization error of…
This paper extends stability analysis to non-convergent neural network training.
problem Generalization of neural networks whose training does not converge to fixed points.
method Introduces statistical algorithmic stability (SAS) to study non-convergent algorithms and their generalization.
result Stability of non-convergent training dynamics correlates with generalization performance.
Paper solves PDEs for optimal investment strategies in volatile markets.
problem Finding optimal investment strategies in volatile markets.
method Numerical methods using time-changed Bessel bridges.
result Solves PDEs for relative arbitrage opportunities in volatility-stabilized markets.
Unified framework for solving fixed-point equations in deterministic and stochastic settings.
problem Solving fixed-point equations for seminorm-contractive operators in both deterministic and stochastic contexts.
method Fixed-point theorem and stochastic approximation analysis.
result Unified finite-sample bounds for various reinforcement learning algorithms.