LightOn OPUs accelerate randomized numerical linear algebra, reducing computational costs.
problem Computational bottleneck in randomization step for large-scale linear algebra.
method Near constant-time linear random projections from LightOn OPUs.
result Significant acceleration of RandNLA algorithms with negligible precision loss.
This chapter is based on lectures on Randomized Numerical Linear Algebra from the 2016 Park City Mathematics Institute summer school on The Mathematics of Data.
RandNLA uses randomness for matrix problems in machine learning.
problem Matrix problems in machine learning.
method Randomized Numerical Linear Algebra.
result New challenges in RandNLA due to hardware trends and advances in ML.
This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.
problem Time-consuming computation of Kronecker factors in K-FAC for large layers.
method Theoretical analysis and randomized numerical linear algebra to approximate eigen-spectrum decay.
result Reduces time complexity from cubic to quadratic in layer width, improving efficiency.
Develops asymptotic analysis for RandNLA sampling estimators in least-squares problems.
problem Lack of distributional information for RandNLA estimators in statistical inference.
method Asymptotic analysis of sampling estimators for least-squares problems in two settings.
result Sampling estimators are asymptotically normally distributed under mild conditions.
Efficient kernel methods for large datasets using GPU acceleration.
problem Handling large-scale nonparametric learning problems efficiently.
method Preconditioned gradient solver, GPU acceleration, parallelization, out-of-core linear algebra, numerical precision optimization.
result Dramatic speedups on datasets with billions of points, maintaining state-of-the-art performance.
SALSA efficiently approximates leverage scores for big data, improving ARMA model fitting.
problem Efficiently approximating leverage scores for large matrices.
method Sequential approximate leverage-score algorithm (SALSA) using randomized numerical linear algebra.
result SALSA approximates leverage scores within (1+O(ε)) with high probability. CoLA automates efficient numerical linear algebra for complex matrix structures.
problem Efficiently solving large-scale linear algebra problems with complex matrix structures.
method Combining linear operator abstraction with compositional dispatch rules.
result Automatic and efficient numerical algorithms for various linear algebra operations.
These are lecture notes that are based on the lectures from a class I taught on the topic of Randomized Linear Algebra (RLA) at UC Berkeley during the Fall 2013 semester.
This paper offers a new algebraic perspective of GCCA using subspace intersection.
problem Finding common variables across multiple feature representations.
method Subspace intersection approach based on a (bi-)linear generative model.
result GCCA is equivalent to subspace intersection, with conditions for identifiable common subspace.
FCM efficiently approximates committor function with interpretable kernel model.
problem Approximating committor function in stochastic systems.
method Kernel-based approach using randomized linear algebra.
result FCM outperforms neural networks in accuracy and training speed.
Two algorithms improve fitting autoregressive models for big data.
problem Efficiently solving Toeplitz least squares problems for large time series data.
method Applied randomized numerical linear algebra (RandNLA) techniques.
result LSAR algorithm is more robust for real-world time series data.
Dimension reduction is the process of embedding high-dimensional data into a lower dimensional space to facilitate its analysis. In the Euclidean setting, one fundamental technique for dimension reduction is to apply a random linear map to the data. This dimension reduction procedure succeeds when it preserves certain …
Reconstructing signature features from randomized vector fields in differential equations.
problem Reconstructing signature features from controlled differential equations with random vector fields.
method Using controlled ordinary differential equations driven by continuous bounded variation curves, the study explores the extent to which signature features can be reconstructed from the non-linear flow of these equations.
result The number of signature features that can be reconstructed from the non-linear flow of controlled ordinary differential equations with random vector fields is exponential in the hidden dimension, under certain conditions.
New method preserves spectral clustering performance under aggressive sparsification and quantization.
problem Maintaining spectral clustering performance with sparse and quantized data.
method Random matrix theory applied to eigenspectrum changes under sparsification and quantization.
result Spectral clustering performance is preserved even with aggressive sparsification and quantization.
This chapter introduces quaternion machine learning for 3D rotations.
problem Lack of quaternion machine learning for 3D rotations.
method Augmented statistics, widely linear models, quaternion calculus, mean square estimation.
result Foundation for quaternion machine learning.
In recent years, randomized methods for numerical linear algebra have received growing interest as a general approach to large-scale problems. Typically, the essential ingredient of these methods is some form of randomized dimension reduction, which accelerates computations, but also creates random approximation error.…
Unified theory and debiasing framework for random oblique projections in high dimensions.
problem Systematic statistical bias in random oblique projections induced by sampling.
method Unified non-asymptotic theory and debiasing framework.
result Sharp bias--variance characterizations and improved approximation accuracy.
Conjugate gradient methods improve efficiency for high-dimensional GLMMs.
problem Efficiency bottleneck in computing high-dimensional GLMM precision matrices.
method Combining spectral analysis and random graph theory with conjugate gradient methods.
result CG-based methods achieve linear scaling in cost with model parameters and observations.
High order discretization schemes of SDEs by using free Lie algebra valued random variables are introduced by Kusuoka, Lyons-Victoir, Ninomiya-Victoir and Ninomiya-Ninomiya. These schemes are called KLNV methods. They involve solving the flows of vector fields associated with SDEs and it is usually done by numerical me…
Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.
problem Training neural networks to global optimality via convex optimization.
method Randomized algorithms in Clifford's Geometric Algebra for hypercomplex vector spaces.
result Convex optimization and geometric algebra improve LLMs' robustness and reliability in transfer learning.
Efficient approximations reduce computation of matrix-based Renyi's entropy.
problem High computational complexity of matrix-based Renyi's entropy.
method Taylor, Chebyshev, and Lanczos approximations to reduce complexity.
result Reduced complexity to significantly less than O(n2) with negligible accuracy loss. Researchers develop Malliavin calculus for signatures, simplifying option Greeks computation.
problem Lack of tractability and explicit representations in Malliavin calculus.
method Focus on finite linear combinations of time-extended Brownian motion signatures, derive explicit formulas for Malliavin derivative, and compute Greeks for path-dependent options.
result Closed-form expressions for classical operators of Malliavin calculus, providing algebraic formulations.
Improves numerical solution of ill-conditioned linear systems for machine learning.
problem Wastefulness and instability in solving ill-conditioned linear systems.
method autonugget combines Richardson extrapolation to determine the solution of the ill-conditioned system, improving accuracy over a single nugget.
result Improves accuracy of numerical solution of ill-conditioned linear systems.
Corrects bias in random sampling matrices for improved ML methods.
problem Inversion bias in random sampling matrices hampers ML applications.
method Corrects inversion bias for various random sampling methods.
result Establishes local convergence rates for sub-sampled Newton methods.
We investigate the computational complexity of several basic linear algebra primitives, including largest eigenvector computation and linear regression, in the computational model that allows access to the data via a matrix-vector product oracle. We show that for polynomial accuracy, Θ(d) calls to the oracle are nece…
Simulating the time-evolution of quantum mechanical systems is BQP-hard and expected to be one of the foremost applications of quantum computers. We consider classical algorithms for the approximation of Hamiltonian dynamics using subsampling methods from randomized numerical linear algebra. We derive a simulation tech…
A new method improves convergence in low-rank approximation.
problem Efficiently solving large-scale numerical linear algebra problems.
method Error-Powered Sketched Inverse Iteration (EPSI) Method.
result Convergence rate improves at least linearly with sketch size.
A new algorithm approximates logistic regression probabilities efficiently.
problem Efficiently approximating probabilities in logistic regression for large datasets.
method Randomized sampling-based algorithm with leverage scores.
result Accurate approximations to estimated probabilities with smaller sample sizes.
This paper optimizes sampling for least-squares approximation.
problem Optimizing sampling for least-squares approximation in arbitrary linear spaces.
method Introducing the Christoffel function to construct near-optimal random sampling strategies.
result The number of samples scales log-linearly in the dimension of the approximation space.
Unified error analysis for low-rank approximation improves data assimilation performance.
problem Analyzing the error in low-rank approximation methods for data assimilation.
method Unified stochastic analysis framework for Frobenius norm error bounds on centered and non-standard Gaussian matrices.
result Unified bounds provide clearer interpretations and enable better practical choices for covariance matrices.
By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous sy…
Methods from learning theory are used in the state space of linear dynamical and control systems in order to estimate the system matrices. An application to stabilization via algebraic Riccati equations is included. The approach is illustrated via a series of numerical examples.
New method solves optimal stopping problems using rough path signatures.
problem Optimal stopping problems in finance and other fields.
method Using rough path signatures and deep neural networks.
result Solves optimal stopping problems efficiently under minimal assumptions.
Multivariate splines linked to infinitely-wide neural networks with improved numerical performance.
problem Understanding the relationship between multivariate splines and neural networks.
method Showed multivariate splines can be represented as random features in infinitely-wide neural networks with a homogeneous activation function.
result The function space of multivariate splines is a Sobolev space on a Euclidean ball with explicit norm bounds on derivatives.
The statistical leverage scores of a complex matrix A∈Cn×d record the degree of alignment between col(A) and the coordinate axes in Cn. These score are used in random sampling algorithms for solving certain numerical linear algebra problems. In this paper we present a max-plus algebr…
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
NGRC shows numerical instabilities with short lags and high-degree polynomials.
problem Numerical instabilities in NGRC feature matrix.
method Combining numerical linear algebra and dynamical systems theory, we study feature matrix conditioning. We evaluate different numerical algorithms for solving the regularized least-squares problem.
result SVD-based training achieves accurate forecasts without regularization, preferable for short lags and high-degree polynomials.
Graph neural networks improve AMG convergence for sparse systems.
problem Efficiently constructing algebraic multigrid prolongation operators for sparse linear systems.
method Train a graph neural network to learn prolongation operators from matrix classes, using an unsupervised loss function.
result Improved convergence rates compared to classical AMG methods.
Fermat-Torricelli points help assess investment risks by smoothing series data.
problem Analyzing investment risks in series with large variance, nonlinear trends, or non-normal distributions.
method Construct Fermat-Torricelli points to reduce random component influence.
result Smoothing series by Fermat-Torricelli points reduces risk assessment errors.
We derive a numerical algorithm for evaluating the Riemannian logarithm on the Stiefel manifold with respect to the canonical metric. In contrast to the existing optimization-based approach, we work from a purely matrix-algebraic perspective. Moreover, we prove that the algorithm converges locally and exhibits a linear…
Paper improves performance guarantees for Rademacher projections.
problem Improving statistical guarantees for Rademacher random projections.
method Algebraic framework for proving Schur-concavity properties.
result Novel Schur-concavity property of Rademacher projections with improved performance.
Paper develops a method for estimating PFLM with minimized rates in high dimensions.
problem Estimating PFLM with minimized rates in high dimensions.
method Least square approach with mixed regularizations of function-norm and ℓ1-norm.
result Established optimal minimax rates of estimation for PFLM.
The problem of determining the volume of a tubular neighbourhood has a long and rich history. Bounds on the volume of neighbourhoods of algebraic sets have turned out to play an important role in the probabilistic analysis of condition numbers in numerical analysis. We present a self-contained derivation of bounds on t…
Projection-cost preservation is a low-rank approximation guarantee which ensures that the cost of any rank-k projection can be preserved using a smaller sketch of the original data matrix. We present a general structural result outlining four sufficient conditions to achieve projection-cost preservation. These condit…
A new method for estimating large-scale linear models with improved precision.
problem Estimating large-scale linear statistical models efficiently.
method Sequential Least-Squares Estimators with Fast Randomized Sketching (SLSE-FRS), integrating Sketch-and-Solve and Iterative-Sketching methods.
result SLSE-FRS produces high-precision estimators, outperforming state-of-the-art methods.
The paper tackles learning mixtures of two multinomial logits, showing identifiability and presenting an algorithm.
problem Learning an arbitrary mixture of two multinomial logits.
method Reduction to solving a system of univariate quartic equations, followed by an algorithm using polynomial and linear samples.
result Identifiability of the mixture models may only fail on an algebraic variety of negligible measure.
Linear algebra algorithms are used widely in a variety of domains, e.g machine learning, numerical physics and video games graphics. For all these applications, loop-level parallelism is required to achieve high performance. However, finding the optimal way to schedule the workload between threads is a non-trivial prob…