High-probability bound for distributed stochastic approximation tracking error.
problem Analyzing the convergence of distributed stochastic approximation schemes.
method Analysis using ODE approach to stochastic approximation.
result High probability bound for tracking error between iterates and limiting differential equation.
Paper extends stochastic dominance for compound binomial distributions.
problem Stochastic dominance for infinite-mean random variables.
method Investigates properties and inclusion relationships of distribution classes, extends results to compound binomial distributions.
result Establishes necessary and sufficient conditions for first-order stochastic dominance preservation.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
problem Understanding and comparing risks in heavy-tailed distributions.
method Introducing a new class of heavy-tailed distributions and proving stochastic dominance relations.
result Weighted averages of random variables in this class are stochastically larger than individual variables.
A new algorithm reduces communication rounds for distributed convex optimization.
problem Efficiently solving convex optimization problems in distributed systems.
method Proposes a stochastic Newton algorithm for homogeneous distributed stochastic convex optimization.
result Reduces the number and frequency of communication rounds compared to existing methods.
Modified Jones-Faddy skew t-distribution captures asymmetry in stock returns.
problem Negative skew and positive mean in stock returns due to broken symmetry of stochastic volatility.
method Modified Jones-Faddy skew t-distribution applied to split gains and losses, using stochastic differential equations for stock returns and volatility.
result The modified distribution effectively captures the asymmetry in daily S&P500 returns, including its tails.
We extend probabilistic programming to handle conditioning on marginal distributions.
problem Conditioning probabilistic programs on marginal distributions of observable variables.
method We define and implement stochastic conditioning, allowing inference in probabilistic programs conditioned on marginal distributions.
result We demonstrate the effectiveness of stochastic conditioning in various real-life scenarios.
We show that stochastic interpolation flow maps are Lipschitz with a sharp constant.
problem High dimensional sampling and transport problems.
method Investigating stochastic interpolation flow for generating data samples.
result Stochastic interpolation flow maps are Lipschitz with a sharp constant matching optimal transport maps.
Paper establishes sufficient condition for comparing linear combinations of infinite-mean risks.
problem Comparing linear combinations of infinite-mean risks under stochastic dominance.
method Introduced a new class of distributions and used majorization order to compare weights.
result Linear combinations of random variables are stochastically larger when their weight vectors are smaller in majorization order.
Unexpectedly, weighted Pareto variables are stochastically dominant.
problem Understanding stochastic dominance in Pareto distributions.
method Analyzing weighted averages of Pareto random variables with infinite mean.
result The weighted average of Pareto variables is stochastically dominant.
New sampling method uses stochastic interpolants and FBSDEs.
problem Sampling from high-dimensional distributions with unnormalized densities.
method Stochastic interpolants and FBSDEs to define and solve diffusion process.
result Effective sampling from challenging distributions.
Moate Simulation improves accuracy and speed of financial derivative pricing.
problem Efficiently pricing financial derivatives with high accuracy.
method Discrete time simulation of probability distributions using Moate Simulation.
result Moate Simulation provides highly accurate distributions for financial derivatives pricing.
Quantized Stochastic Primal-Dual Methods for Distributed Optimization
problem Distributed optimization with stochastic gradients and finite-bit communication
method q-PDGD, a quantized stochastic primal-dual method
result Linear contraction to an explicit neighborhood under RSI, O(1/k) convergence under PL inequality
MF-GLaM models improve stochastic simulator emulation with multifidelity data.
problem Challenging to emulate stochastic simulators' full conditional probability distribution.
method Proposes MF-GLaMs to efficiently emulate HF stochastic simulators using LF data.
result MF-GLaMs achieve improved accuracy or comparable performance at reduced cost.
For option pricing models and heavy-tailed distributions, this study proposes a continuous-time stochastic volatility model based on an arithmetic Brownian motion: a one-parameter extension of the normal stochastic alpha-beta-rho (SABR) model. Using two generalized Bougerol's identities in the literature, the study sho…
New framework for ranking distributions using variable fractional parameters.
problem Ordering distributions with varying steepness and local non-concavities.
method Introducing a function γ:Ro[0,1] to replace the fixed parameter in fractional SD. result Enables ranking of a broader range of distributions and incorporates dynamic greediness.
New method for sampling from complex distributions using stochastic localization.
problem Sampling from unnormalized target densities in multi-modal distributions.
method Stochastic Localization via Iterative Posterior Sampling (SLIPS) framework.
result Approximate samples from target distribution and denoiser learned iteratively.
Study on Kyle's model with stochastic liquidity impacts asset volatility.
problem Impact of stochastic volatility of noise trading on asset volatility.
method Construct equilibrium for continuous-time Kyle's model with stochastic liquidity.
result In equilibrium, Kyle's Lambda and its inverse are submartingales.
The paper studies stochastic gradient descent with infinite variance gradients.
problem Theoretical properties of SGD with infinite variance gradients.
method Establish asymptotic behavior of SGD with infinite variance gradients.
result Asymptotic distribution of SGD is characterized as a stationary distribution of an Ornstein-Uhlenbeck process driven by a stable Lévy process.
CSI method learns conditional distributions by estimating flow equations.
problem Learning conditional distributions in generative models.
method Estimates probability flow equations to transport reference to target distribution.
result Derives explicit expressions for conditional drift and score functions.
This work tackles resource allocation in asynchronous and stochastic systems.
problem Distributed resource allocation in asynchronous and stochastic settings.
method Approximate stochastic primal-dual approach with asynchronous updates.
result The Asynchronous stochastic Primal-Dual (Asyn-PD) algorithm converges to the saddle point solution at a rate of O(1/t). Bayesian neural networks can be partially stochastic without losing predictive power.
problem The necessity of fully stochastic parameters in Bayesian neural networks.
method Theoretical and empirical investigation of partially stochastic networks compared to fully stochastic ones.
result Expressive predictive distributions require only small amounts of stochasticity, and partially stochastic networks can match or outperform fully stochastic networks.
AB-SAGA optimizes distributed optimization over directed graphs using variance reduction and stochastic weights.
problem Optimizing distributed stochastic optimization over directed graphs with stochastic weights.
method AB-SAGA combines variance reduction and network-level gradient tracking, using both row and column stochastic weights.
result AB-SAGA converges linearly to the global optimal with a constant step-size and achieves a linear speed-up over centralized methods.
New algorithms improve distributed optimization under mild variance conditions.
problem Improving distributed optimization for large-scale machine learning problems.
method Revisited Federated Averaging and SCAFFOLD algorithms under a general variance condition.
result Established convergence results for smooth nonconvex objective functions under mild variance conditions.
LASG improves communication efficiency in distributed learning.
problem Efficiently communicating gradients in distributed machine learning.
method Develops a new stochastic gradient descent approach, LASG, that predicts and selects significant communication rounds.
result Achieves communication savings by an order of magnitude in federated learning.
Recent advances in stochastic gradient techniques have made it possible to estimate posterior distributions from large datasets via Markov Chain Monte Carlo (MCMC). However, when the target posterior is multimodal, mixing performance is often poor. This results in inadequate exploration of the posterior distribution. A…
Paper addresses fault-tolerance in distributed machine learning with stochastic gradient descent.
problem Fault-tolerance in distributed stochastic gradient descent (D-SGD) for machine learning.
method Proposes norm-based comparative gradient elimination (CGE) to robustify D-SGD against Byzantine faulty agents.
result CGE guarantees fault-tolerance against a bounded fraction of Byzantine agents under standard stochastic assumptions.
Paper develops online statistical inference methods for stochastic optimization using Kiefer-Wolfowitz algorithms.
problem Online statistical inference of model parameters in stochastic optimization problems.
method Kiefer-Wolfowitz algorithm with random search directions, asymptotic distribution analysis.
result Developed valid confidence intervals for online statistical inference.
This paper constructs Brownian motion on complex flag manifolds and finds joint distribution of stochastic areas.
problem Modeling stochastic areas on complex partial flag manifolds.
method Constructs Brownian motion on complex partial flag manifolds and uses it to find joint distribution of stochastic areas.
result Limit law of stochastic areas is a multivariate Cauchy distribution.
Stochastic algo learns from evolving data, achieving optimal performance.
problem Performative prediction and multiplayer extensions.
method Stochastic approximation with decision-dependent distributions.
result Asymptotic normality and optimality of the algorithm's performance.
In this letter, we introduce a distributed Nesterov method, termed as ABN, that does not require doubly-stochastic weight matrices. Instead, the implementation is based on a simultaneous application of both row- and column-stochastic weights that makes this method applicable to arbitrary (strongly-connected…
Despite having various attractive qualities such as high prediction accuracy and the ability to quantify uncertainty and avoid over-fitting, Bayesian Matrix Factorization has not been widely adopted because of the prohibitive cost of inference. In this paper, we propose a scalable distributed Bayesian matrix factorizat…
Proposes new rule for ranking investment prospects over long horizons.
problem Ranking investment prospects over long horizons considering bounded risk aversion.
method Introduces asymptotic fractional-order stochastic dominance with bounded relative risk aversion.
result Establishes equivalent conditions for the new rule under lognormal returns without mean non-negativity constraint.
This paper develops a communication-efficient algorithm to solve the stochastic optimization problem defined over a distributed network, aiming at reducing the burdensome communication in applications such as distributed machine learning.Different from the existing works based on quantization and sparsification, we int…
New method optimizes SDE models using continuous-time gradient descent.
problem Optimizing over the stationary distribution of SDE models.
method Continuous-time stochastic gradient descent for SDE models.
result Asymptotic convergence to the direction of steepest descent.
This work introduces a new model for complex stochastic processes.
problem Difficulties in representing non-stationary distributions with conventional models.
method Recurrent Autoregressive Flows using normalizing flows with recurrent neural connections.
result Demonstrates the effectiveness of the proposed model through experiments.
We propose a stochastic process driven by the memory effect with novel distributions which include both exponential and leptokurtic heavy-tailed distributions. A class of the distributions is analytically derived from the continuum limit of the discrete binary process with the renormalized auto-correlation. The moment …
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
Study on Wasserstein distance for numerical approximations of stochastic differential equations.
problem Estimating the Wasserstein distance between stochastic differential equation distributions and their numerical approximations.
method Unified framework for analyzing different integrators and a novel splitting method for underdamped Langevin dynamics.
result A novel splitting method for underdamped Langevin dynamics with optimal complexity.
Proposes EDM algorithm to accelerate model training in distributed networks.
problem Hindered effectiveness of distributed stochastic optimization algorithms due to data heterogeneity and network sparsity.
method Introduces Exact-Diffusion with Momentum (EDM) algorithm, incorporating momentum techniques to mitigate bias and enhance convergence rate.
result EDM algorithm converges sub-linearly to the optimal solution, radius independent of data heterogeneity, for non-convex objective functions.
We present a framework for describing the evolution of stochastic observables having a non-stationary distribution of values. The framework is applied to empirical volume-prices from assets traded at the New York stock exchange. Using Kullback-Leibler divergence we evaluate the best model out from four biparametric mod…
Stochastic normalizing flows use SDEs for efficient training and sampling.
problem Efficient maximum likelihood estimation and variational inference.
method Continuous normalizing flows extended with stochastic differential equations (SDEs) and rough path theory.
result Stochastic normalizing flows enable efficient training and sampling from complex distributions.
Framework learns stochastic dynamics from endpoint and intermediate distributions using soft energy constraints.
problem Learning stochastic dynamics from endpoint and intermediate distributional observations.
method Formulates generation as a McKean-Vlasov control problem with soft energy constraints, solving it through FBSDE.
result Model learns coherent stochastic trajectories matching prescribed marginal laws.
Proposes a fair machine learning framework robust to distribution shifts without causal graph knowledge.
problem Fairness issues in machine learning models under distribution shifts.
method Stochastic distributionally robust optimization with Exponential Renyi Mutual Information (ERMI) fairness measure.
result First stochastic framework for fair learning robust to distribution shifts without causal graph knowledge.
How to model distribution of sequential data, including but not limited to speech and human motions, is an important ongoing research problem. It has been demonstrated that model capacity can be significantly enhanced by introducing stochastic latent variables in the hidden states of recurrent neural networks. Simultan…
We generalize stochastic subgradient descent methods to situations in which we do not receive independent samples from the distribution over which we optimize, but instead receive samples that are coupled over time. We show that as long as the source of randomness is suitably ergodic---it converges quickly enough to a …
Motivated by machine learning applications in networks of sensors, internet-of-things (IoT) devices, and autonomous agents, we propose techniques for distributed stochastic convex learning from high-rate data streams. The setup involves a network of nodes---each one of which has a stream of data arriving at a constant …
New framework for understanding adversarial and stochastic learning.
problem Understanding the continuum from adversarial to stochastic settings in online learning.
method Distributionally constrained adversaries framework.
result Characterization of learnable distribution classes for various function classes.
Many machine learning algorithms minimize a regularized risk, and stochastic optimization is widely used for this task. When working with massive data, it is desirable to perform stochastic optimization in parallel. Unfortunately, many existing stochastic optimization algorithms cannot be parallelized efficiently. In t…