The paper studies stochastic optimization on matrices and its limits as dimensions grow.
problem Optimizing functions on large symmetric matrices using stochastic gradient descent.
method Deterministic limits of random curves on matrices, using graphons and stochastic differential equations.
result The limit is a gradient flow on graphons, extending classical McKean-Vlasov limits.
A novel approach models rating transitions using Lie groups and Deep Learning.
problem Modeling rating transitions with geometric properties and stochastic processes.
method Introducing Itô-SDEs on Lie groups, using TimeGAN for calibration, and examining rating matrix properties.
result The geometric approach using Lie groups and Deep Learning generates a good fit for rating transitions.
This article provides the mathematical foundation for stochastically continuous affine processes on the cone of positive semidefinite symmetric matrices. This analysis has been motivated by a large and growing use of matrix-valued affine processes in finance, including multi-asset option pricing with stochastic volatil…
Paper proposes Sinkformers for Transformers with doubly stochastic attention.
problem Improving Transformer models' accuracy in vision and natural language processing.
method Using Sinkhorn's algorithm to make attention matrices doubly stochastic instead of SoftMax normalization.
result Sinkformers enhance model accuracy in vision and natural language processing tasks.
Origin-destination (OD) matrices are often used in urban planning, where a city is partitioned into regions and an element (i, j) in an OD matrix records the cost (e.g., travel time, fuel consumption, or travel speed) from region i to region j. In this paper, we partition a day into multiple intervals, e.g., 96 15-min …
Graph alignment problem solved with convex relaxations for correlated matrices.
problem Recovering hidden vertex permutations from correlated Gaussian matrices.
method Convex relaxations of the quadratic assignment problem over doubly stochastic matrices.
result The solution of the convex relaxation concentrates around the ground-truth permutation matrix for certain correlation parameters.
Develops a novel stochastic algorithm for diagonal estimation of large matrices.
problem Efficient diagonal estimation for large or implicit matrices.
method Adaptive parameter selection in a stochastic algorithm.
result Lower bound on random query vectors needed for estimation.
A method to complete incomplete correlation matrices using maximum entropy.
problem Incomplete correlation matrices in financial applications.
method Maximizing entropy of the distribution described by the matrix, constructing a chordal graph.
result A proper correlation matrix can be constructed for large models involving multiple currencies.
A new method for stochastic optimal control improves accuracy over existing techniques.
problem Improving the accuracy of stochastic optimal control for noisy systems.
method Stochastic Optimal Control Matching (SOCM) using Iterative Diffusion Optimization (IDO) with path-wise reparameterization trick.
result SOCM achieves lower error than existing techniques for three out of four control problems, sometimes by an order of magnitude.
Researchers study heavy-tail properties of SGD using stochastic recurrence equations.
problem Analyzing heavy-tail properties of Stochastic Gradient Descent (SGD).
method Modeling SGD iterations as multivariate affine stochastic recursions and applying the theory of irreducible-proximal (i-p) matrices.
result Extended results of Gürbüzbalaban et al. (2020) by using the theory of i-p matrices.
We consider three different approaches to define natural Riemannian metrics on polytopes of stochastic matrices. First, we define a natural class of stochastic maps between these polytopes and give a metric characterization of Chentsov type in terms of invariance with respect to these maps. Second, we consider the Fish…
New estimator learns symmetric dynamics from few observations.
problem Learning parameters of stochastic linear dynamics from limited data.
method Method of moments estimator using T=O(logN) observations. result Achieves small maximum element-wise error on symmetric matrices.
π-GNN learns soft permutations for graph representations, improving graph classification and regression.
problem Limitations of MPNNs in graph neural networks.
method Proposes π-GNN, which learns a soft permutation matrix for each graph, projecting graphs into a common vector space.
result π-GNN achieves performance competitive with state-of-the-art models on graph classification and regression tasks.
Spectral portfolio theory links neural networks to wealth dynamics via SGD weight matrices.
problem Understanding wealth dynamics from neural network training.
method Direct identification of weight matrices as portfolio allocation matrices, linking SGD forces to portfolio dynamics.
result Spectral properties of SGD weight matrices transition between additive and multiplicative regimes, influencing wealth dynamics.
A new method learns noise characteristics for better state estimation in real-time systems.
problem Challenges in accurately estimating states due to uncertainty in process and measurement models.
method Proposes a learning-based approach with different loss functions to identify noise characteristics.
result Demonstrates improved performance in real-time vehicle state estimation.
Stochastic gradient descent optimizes Nyström samples for kernel matrix approximation.
problem Optimizing Nyström samples for kernel matrix approximation.
method Stochastic gradient descent applied to multisets of landmark points (Nyström samples) using a surrogate criterion (radial SKD).
result Local minimization of the radial SKD yields improved Nyström approximation accuracy.
Overview of high-dimensional dynamical systems and their applications to machine learning.
problem Characterizing behavior of high-dimensional dynamical systems driven by random matrices.
method Cavity method arguments, path integrals, dynamical mean field theory (DMFT), and random matrix resolvents.
result Connections between random matrix resolvents and DMFT response, and non-monotonic loss curves in training.
In this paper, we establish the stochastic ordering of the Gini indexes for multivariate elliptical risks which generalized the corresponding results for multivariate normal risks. It is shown that several conditions on dispersion matrices and the components of dispersion matrices of multivariate normal risks for the m…
SGD's training dynamics align with Hessian and gradient spectra in high-dimensional classification tasks.
problem Understanding the spectra of Hessian and gradient matrices in high-dimensional classification tasks.
method Rigorous analysis of SGD dynamics and spectra of Hessian and gradient matrices.
result SGD trajectory and emergent outlier eigenspaces align with a common low-dimensional subspace in multi-class high-dimensional mixtures and neural networks.
Several important families of computational and statistical results in machine learning and randomized algorithms rely on uniform bounds on quadratic forms of random vectors or matrices. Such results include the Johnson-Lindenstrauss (J-L) Lemma, the Restricted Isometry Property (RIP), randomized sketching algorithms, …
ShuffleNet is a state-of-the-art light weight convolutional neural network architecture. Its basic operations include group, channel-wise convolution and channel shuffling. However, channel shuffling is manually designed empirically. Mathematically, shuffling is a multiplication by a permutation matrix. In this paper, …
Spectral regularization simplifies sequence models by focusing on grammatical simplicity.
problem Sequence modeling challenges in learning tasks.
method Introduces spectral regularization based on Hankel matrices and trace norm, addressing bi-infinite matrices with an unbiased estimator.
result Demonstrates spectral regularization's potential benefits on Tomita grammars.
Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces.…
New methods estimate mixed memberships in multi-layer networks.
problem Complex community structure in multi-layer networks.
method Spectral methods using eigen-decomposition of aggregate matrices.
result Theoretical guarantees and empirical validation for mixed membership estimation.
This work tackles regression on non-Euclidean spaces, specifically positive-definite matrices with the Bures-Wasserstein metric.
problem Regression on non-Euclidean spaces, specifically positive-definite matrices with the Bures-Wasserstein metric.
method Developed a sufficient condition for the existence of a minimizer of the conditional barycenter problem, characterized the optimization landscape, and developed a projection-free algorithm for approximate computation of first-order stationary points.
result The objective is free of local maxima under the sufficient condition, and the algorithm enables the use of stochastic Riemannian optimization methods for large-scale setups.
We present a novel technique for learning the mass matrices in samplers obtained from discretized dynamics that preserve some energy function. Existing adaptive samplers use Riemannian preconditioning techniques, where the mass matrices are functions of the parameters being sampled. This leads to significant complexiti…
Geometric approach for unsupervised word embedding alignment.
problem Learning alignment between word embeddings of source and target languages.
method Formulates alignment as domain adaptation on the manifold of doubly stochastic matrices, employing Riemannian conjugate gradient algorithm.
result Empirically outperforms state-of-the-art methods on bilingual lexicon induction tasks.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
Efficient algorithm for CCA on Riemannian manifolds with fast convergence.
problem Efficiently computing canonical correlation components on Riemannian manifolds.
method Reparametrization of projection matrices for stochastic optimization on Riemannian manifolds.
result Achieves $O( rac{1}{t})$ convergence rate for top k components with O(d2k) runtime complexity. A method for accurate pricing of multidimensional derivatives under uncertain volatility.
problem High-dimensional stochastic control problem in uncertain volatility model.
method Backward actor-critic stochastic policy gradient scheme combining DP, PPO, and neural networks.
result Accurate and efficient pricing of multidimensional derivatives compared to benchmarks.
Study on estimating unstable open-loop matrices from state trajectories.
problem System identification for stochastic continuous-time dynamics.
method Employing randomized control inputs to estimate unstable open-loop matrix.
result Estimation error decays with trajectory length, signal-to-noise ratio, and excitability.
Paper introduces OMD for ordered state transitions in SSMs.
problem Modeling ordered latent states in dynamic systems.
method Ordered Matrix Dirichlet (OMD) prior over ordered stochastic matrices.
result OMD models recover interpretable ordered latent structure without sacrificing predictive performance.
A non-Hermitean extension of paradigmatic Wishart random matrices is introduced to set up a theoretical framework for statistical analysis of (real, complex and real quaternion) stochastic time series representing two "remote" complex systems. The first paper in a series provides a detailed spectral theory of non-Hermi…
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
Sorting input objects is an important step in many machine learning pipelines. However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization. In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting…
We consider a short rate model, driven by a stochastic process on the cone of positive semidefinite matrices. We derive sufficient conditions ensuring that the model replicates normal, inverse or humped yield curves.
Nonnegative matrix factorization (NMF), a dimensionality reduction and factor analysis method, is a special case in which factor matrices have low-rank nonnegative constraints. Considering the stochastic learning in NMF, we specifically address the multiplicative update (MU) rule, which is the most popular, but which h…
Bayesian method for dynamic correlation matrices improves accuracy and responsiveness.
problem Challenges in estimating time-varying correlation matrices, including slow adaptation, insufficient regularization, and diffuse uncertainty.
method Low-rank factor representation with dynamic shrinkage prior and multivariate factor stochastic volatility model.
result Improved accuracy and responsiveness compared to competing methods in various challenging scenarios.
Two spectral clustering methods for multi-layer networks are analyzed and compared.
problem Community detection in multi-layer networks.
method Sum and debiased sum of squared adjacency matrices for spectral clustering.
result Debiased sum of squared adjacency matrices outperforms sum of adjacency matrices.
Neural Markov models improve time series analysis by balancing deep learning and classical models.
problem Modeling non-stationary time series with high data sparsity.
method Hybrid approach using neural networks to parameterize stochastic matrices, estimating time-inhomogeneous Markov chains.
result Reduction of Chapman-Kolmogorov discrepancy and superior likelihood in financial markets.
New algorithms learn graph structures privately, matching best results.
problem Private learning of graph structures with multiple blocks.
method Sum-of-squares relaxation and exponential mechanism for score function.
result Matches statistical utility of previous best non-private methods.
The paper provides bounds for LSA with fixed stepsizes under random estimates.
problem Analyzing the performance of LSA algorithms with fixed stepsize.
method Non-asymptotic analysis based on new results about matrix moments and high probability bounds.
result Derives high probability bounds on LSA performance under weaker conditions than previous works.
We propose an L-BFGS optimization algorithm on Riemannian manifolds using minibatched stochastic variance reduction techniques for fast convergence with constant step sizes, without resorting to linesearch methods designed to satisfy Wolfe conditions. We provide a new convergence proof for strongly convex functions wit…
The book explores stochastic areas and heat kernels on manifolds.
problem Understanding stochastic area functionals and heat kernels on manifolds.
method Study of Brownian motions and heat kernels on Lie groups and Riemannian manifolds.
result Rich interactions between stochastic calculus, geometry, and random matrices.
Many matching, tracking, sorting, and ranking problems require probabilistic reasoning about possible permutations, a set that grows factorially with dimension. Combinatorial optimization algorithms may enable efficient point estimation, but fully Bayesian inference poses a severe challenge in this high-dimensional, di…
We analyze a class of estimators based on convex relaxation for solving high-dimensional matrix decomposition problems. The observations are noisy realizations of a linear transformation X of the sum of an approximately) low rank matrix Θ⋆ with a second matrix Γ⋆ endowed with a complementary …
Estimates matrix trace optimization with statistical learning theory.
problem Optimizing trace of parameter-dependent matrices.
method Monte Carlo estimator with bounds derived from epsilon nets and generic chaining.
result Predicts small sampling amount for matrices with small off-diagonal mass.
Graphical models for covariance matrices improve structure learning.
problem Learning structure in graphical models for covariance matrices.
method Structural learning via ℓ1-penalized loss minimization. result Method outperforms alternatives in simulations and real-world applications.