New algorithms improve CCA with stochastic approximation.
problem Efficiently compute canonical correlation analysis.
method Inexact MSG and MEG algorithms for CCA.
result Achieves ε-suboptimality in poly(1/ε) iterations.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,291 papers · 148 categories
Trend · papers per month
48 results for “matrix exponentiated gradient”
Matrix Krasulina achieves fast convergence for online k-PCA.
problem Online k-PCA with low-rank data.
method Generalized Krasulina's method for matrix case, without variance reduction.
result Exponential convergence to principal subspace.
Efficiently implements MEG for low-rank matrix optimization problems.
problem Optimization over spectrahedron with low-rank matrices.
method Matrix Exponentiated Gradient (MEG) method with efficient implementations.
result Methods converge from a warm-start initialization with similar rates to full-SVD-based counterparts.
Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networkscs.LG
Gradient descent requires exponentially many iterations for deep linear neural networks.
problem Understanding convergence time for gradient descent in deep linear neural networks.
method Analysis of gradient descent dynamics on specific objective functions.
result Convergence time scales exponentially with network depth.
A new black-box optimizer using implicit natural gradient.
problem Efficient optimization for complex, computationally intensive problems.
method Stochastic update with implicit natural gradient of an exponential-family distribution.
result Theoretical convergence rate for convex functions and continuous non-differentiable functions.
QBVI uses natural gradients for efficient Bayesian learning.
problem Efficient Bayesian learning in complex models.
method Natural gradient updates in a black-box framework for exponential-family distributions.
result QBVI framework is effective for a wide range of Bayesian inference problems.
Unified approach combining gradient descent and multiplicative updates.
problem Combining gradient descent and multiplicative updates for machine learning.
method Introduces hypentropy and a family of matrix-based updates.
result Derives tight regret bounds for the new family of updates.
Incorporates matrix exponential into generative flows for improved performance.
problem Improving generative flow models for better density estimation.
method Integrates matrix exponential into generative flows, proposing new layers and modifying network architecture.
result The proposed model achieves great performance on density estimation.
sgdGMF efficiently estimates generalized matrix factorization models for single-cell RNA sequencing data.
problem Challenges in dimensionality reduction for large single-cell RNA sequencing datasets.
method Scalable adaptive stochastic gradient descent algorithm for generalized matrix factorization models.
result sgdGMF outperforms existing methods in scalability and accuracy for large datasets.
Gradient-flow helps find good minima in complex models.
problem Understanding why gradient-based algorithms work in non-convex optimization.
method Kac-Rice analysis and gradient-flow from statistical physics.
result Gradient-flow finds good global minima in the presence of many spurious local minima.
Incremental computation of block triangular matrix exponentials with application to option pricingmath.NA
Efficiently computes option pricing matrix exponentials.
problem Computing matrix exponentials of nested block triangular matrices.
method Incremental computation using scaling and squaring, reusing intermediate quantities.
result Efficiently computes option pricing matrix exponentials.
A new machine learning model uses matrix exponentials for universal approximation.
problem Developing a robust and efficient machine learning model.
method Introduces a novel architecture using matrix exponentials as the only nonlinearity.
result The model achieves universal approximation properties and outperforms other models on benchmark tasks.
Training avoids edge of stability by aligning Jacobian matrices.
problem Training neural networks on the edge of stability causes inaccuracies.
method Used an exponential Euler solver to prevent entering the edge of stability.
result Alignment of Jacobian matrices causes sharpness increase in Hessian.
Sub-gradient method recovers low-rank matrices robustly from noisy measurements.
problem Recovering low-rank matrices from noisy measurements with unknown rank.
method Sub-gradient method with small initialization, robust to over-parameterization and noise.
result Sub-gradient method converges exponentially fast to the true solution under noisy and over-parameterized conditions.
SNN architecture shows gradient descent converges to regularized solution in matrix sensing problems.
problem Understanding implicit regularization in neural networks for matrix sensing.
method Developed Spectral Neural Networks (SNN) for matrix learning problems, rigorously demonstrating implicit regularization.
result Gradient descent converges to the solution of a regularized learning problem in matrix sensing problems.
We develop a more efficient NGD method for structured parameters.
problem Computational challenges in NGD for structured parameter spaces.
method Local-parameter coordinates to simplify Fisher-matrix computations.
result New structured second-order algorithms and learning methods.
Muon outperforms GD in associative memory learning by balancing frequency components.
problem Training dynamics and scaling behavior of Muon in associative memory learning.
method Study of Muon in a linear associative memory model with softmax retrieval and hierarchical frequency spectrum over query-answer pairs.
result Muon achieves exponential speedup over GD in noiseless case and superior scaling efficiency in noisy case.
EPG unifies SPG and DPG for reinforcement learning.
problem Improving reinforcement learning algorithms for policy optimization.
method EPG integrates across actions for gradient estimation, using practical results for Gaussian policies and extending to broader classes of policies.
result EPG reduces gradient variance without deterministic policies and with minimal overhead.
New algorithm reduces complexity for SPD manifold optimization.
problem Efficiently minimize functions over SPD manifold.
method Low-complexity Riemannian subspace descent with sparse updates.
result Innovative updates avoid costly matrix operations.
Convex Analysis of Relaxation Dynamics in Chemical Reaction Networks and Generalized Gradient Flowsq-bio.MN
Bounds on chemical reaction network relaxation rates using convex analysis.
problem Understanding relaxation dynamics in chemical reaction networks.
method Convex analysis, generalized gradient flows, singular values of stoichiometric matrix.
result Bounds on Kullback-Leibler divergence to equilibrium for CRNs.
For a Morse map Novikov [11] has introduced an analog of Morse complex, defined over the ring $\ZZZ[[t]][t^{-1}]$ of integer Laurent power series. Novikov conjectured, that generically the matrix entries of the differentials in this complex are of the form , where grow at most exponenti…
Efficient approximations for AdaGrad reduce computation while maintaining performance.
problem Training deep neural networks efficiently in high dimensions.
method Ada-LR and RadaGrad use random projections to approximate full-matrix AdaGrad.
result Regret of Ada-LR is close to full-matrix AdaGrad, achieving similar performance with less computation.
Framework for transforming manifold constraints into unconstrained problems.
problem Optimization on manifolds with constraints.
method Introducing 'trivializations' and 'dynamic trivializations' for gradient-based optimization.
result Dynamic trivializations improve performance on tasks involving long-term memory in neural networks.
Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
Exponential Weighting and Random-Matrix-Theory-Based Filtering of Financial Covariance Matrices for Portfolio Optimizationcond-mat.stat-mech
We introduce a covariance matrix estimator that both takes into account the heteroskedasticity of financial returns (by using an exponentially weighted moving average) and reduces the effective dimensionality of the estimation (and hence measurement noise) via techniques borrowed from random matrix theory. We calculate…
Characterizes uncertainty in low-rank matrix completion with noisy data.
problem Uncertainty quantification in low-rank matrix completion with heterogeneous sub-exponential noise.
method Characterizes the distribution of estimated matrix entries under low-rank estimators with heterogeneous sub-exponential noise.
result Explicit formulas for the distribution of estimated matrix entries under Poisson and Binary noise.
Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?math.ST
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
A new method for efficiently computing derivatives of skew-symmetric matrix exponentials.
problem Efficient computation of derivatives for skew-symmetric matrices.
method Characterization of invertibility, construction of nearby logarithm, and efficient implementation.
result Explicit formulae for differentiation and its inverse of skew-symmetric matrix exponentials.
The paper introduces a new class of multivariate mixtures for actuarial applications.
problem Developing a new class of multivariate mixtures for actuarial calculations.
method Proposed a class of multivariate matrix-exponential affine mixtures with matrix-exponential marginals.
result Explicit calculations of actuarial quantities are possible due to the proposed class's properties.
Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-pro…
Deep networks improve generalization without explicit regularization.
problem Understanding the generalization of deep learning models.
method Analysis of deep network approximation power, optimization landscape, and uniform convergence.
result Gradient descent can find global minima in deep learning optimization.
We present a new method for online prediction and learning of tensors (-way arrays, ) from sequential measurements. We focus on the specific case of 3-D tensors and exploit a recently developed framework of structured tensor decompositions proposed in [1]. In this framework it is possible to treat 3-D tensors …
Develops a gradient flow for Muon optimizer, a method for optimization.
problem Optimization of complex systems with matrix-valued parameters.
method Gradient flow on probability measures induced by regularized Muon optimizer.
result Derives continuous-time limits and proves Hamiltonian dissipation.
Collective Matrix Completionstat.ML
This work tackles collective matrix completion with multiple and heterogeneous data sources.
problem Reconstructing data from multiple heterogeneous matrices.
method Estimation based on minimizing goodness-of-fit and nuclear norm penalization of the whole collective matrix.
result Proposed estimators achieve fast rates of convergence under two settings.
Exponential testing error reduction with stochastic gradient methods under low-noise conditions.
problem Binary classification with positive definite kernels and square loss.
method Stochastic gradient methods under low-noise conditions.
result Testing error converges exponentially fast, while testing loss converges slowly.
Gradient-free optimizers are ineffective on barren plateaus in quantum computing.
problem Effect of barren plateaus on gradient-free optimization in quantum computing.
method Numerical simulations and theoretical analysis of gradient-free optimization algorithms.
result Gradient-free optimizers are not effective in barren plateau landscapes due to exponentially suppressed cost function differences.
The matrix completion problem consists in reconstructing a matrix from a sample of entries, possibly observed with noise. A popular class of estimator, known as nuclear norm penalized estimators, are based on minimizing the sum of a data fitting term and a nuclear norm penalization. Here, we investigate the case where …
We extend natural-gradient methods to mixtures of exponential-family distributions, improving inference speed.
problem Complex, multimodal posterior distributions are difficult to approximate with simple exponential-family distributions.
method We use minimal conditional-EF representations and derive simple natural-gradient updates.
result Our natural-gradient method converges faster than black-box methods with reparameterization gradients.
Exponentially fast SMF algorithm for multi-class classification.
problem Learning interpretable features from high-dimensional data.
method Novel framework that 'lifts' SMF as a low-rank matrix estimation problem.
result Provable exponential convergence to global minimizer under mild assumptions.
A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
Random Matrix Theory explains loss surface Hessians in neural networks.
problem Understanding the loss surfaces of neural networks.
method Investigation of local spectral statistics of neural network Hessians.
result Excellent agreement with Gaussian Orthogonal Ensemble statistics.
Neural networks solve copositive programs, revealing insights into training problems.
problem Training two-layer vector-output ReLU neural networks.
method Convex analysis and copositive programming.
result Neural networks solve copositive programs, providing insights into training problems.
AdaX improves Adam by exponentially accumulating past gradients, leading to better performance in machine learning tasks.
problem Adam's fast convergence can lead to local minimums in non-convex problems.
method AdaX exponentially accumulates past gradients to adaptively tune the learning rate.
result AdaX outperforms Adam in various machine learning tasks, including computer vision and natural language processing.
Improved SVI with adjustable annealing for better optimization.
problem Improving optimization in stochastic variational inference.
method Tuneable stochastic annealing in SVI with adjustable batch size.
result Approximation to maximum entropy stochastic gradient at desired variance level.
Improved stability for large-scale Bayesian sampling.
problem Reducing instability in Langevin dynamics for large datasets.
method Introducing a modified CCAdL thermostat with a scaling and squaring method and a truncated Taylor series approximation.
result Significantly improved numerical stability and accuracy over existing methods.
Unified framework for analyzing gradient flows of measures with exponential decay of entropy.
problem Analyzing exponential decay of entropy functionals in gradient flows of measures.
method Characterization of global exponential decay behaviors using Hellinger-Kantorovich geometry, shape-mass decomposition, and Polyak-Łojasiewicz-type inequalities.
result Unified theoretical framework for gradient flows with complete analysis of exponential decay behaviors.
We approximate sticky diffusions using Markov chains for efficient simulation.
problem Approximating sticky diffusions for accurate simulation.
method CTMC approximation of sticky diffusions, efficient matrix exponentials, and Euler scheme comparison.
result Second order convergence of CTMC approximation for sticky diffusions.
Accelerated gradient method's stability deteriorates exponentially with steps.
problem Algorithmic stability of Nesterov's accelerated gradient method.
method Analysis of two notions of algorithmic stability for Nesterov's accelerated gradient method.
result Stability of Nesterov's accelerated method deteriorates exponentially with the number of gradient steps.