LightOn OPUs accelerate randomized numerical linear algebra, reducing computational costs.
problem Computational bottleneck in randomization step for large-scale linear algebra.
method Near constant-time linear random projections from LightOn OPUs.
result Significant acceleration of RandNLA algorithms with negligible precision loss.
This chapter is based on lectures on Randomized Numerical Linear Algebra from the 2016 Park City Mathematics Institute summer school on The Mathematics of Data.
These are lecture notes that are based on the lectures from a class I taught on the topic of Randomized Linear Algebra (RLA) at UC Berkeley during the Fall 2013 semester.
RandNLA uses randomness for matrix problems in machine learning.
problem Matrix problems in machine learning.
method Randomized Numerical Linear Algebra.
result New challenges in RandNLA due to hardware trends and advances in ML.
Reconstructing signature features from randomized vector fields in differential equations.
problem Reconstructing signature features from controlled differential equations with random vector fields.
method Using controlled ordinary differential equations driven by continuous bounded variation curves, the study explores the extent to which signature features can be reconstructed from the non-linear flow of these equations.
result The number of signature features that can be reconstructed from the non-linear flow of controlled ordinary differential equations with random vector fields is exponential in the hidden dimension, under certain conditions.
Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.
problem Training neural networks to global optimality via convex optimization.
method Randomized algorithms in Clifford's Geometric Algebra for hypercomplex vector spaces.
result Convex optimization and geometric algebra improve LLMs' robustness and reliability in transfer learning.
We investigate the computational complexity of several basic linear algebra primitives, including largest eigenvector computation and linear regression, in the computational model that allows access to the data via a matrix-vector product oracle. We show that for polynomial accuracy, Θ(d) calls to the oracle are nece…
This paper offers a new algebraic perspective of GCCA using subspace intersection.
problem Finding common variables across multiple feature representations.
method Subspace intersection approach based on a (bi-)linear generative model.
result GCCA is equivalent to subspace intersection, with conditions for identifiable common subspace.
This paper speeds up K-FAC for deep learning by focusing on only a few eigen-modes.
problem Time-consuming computation of Kronecker factors in K-FAC for large layers.
method Theoretical analysis and randomized numerical linear algebra to approximate eigen-spectrum decay.
result Reduces time complexity from cubic to quadratic in layer width, improving efficiency.
Projection-cost preservation is a low-rank approximation guarantee which ensures that the cost of any rank-k projection can be preserved using a smaller sketch of the original data matrix. We present a general structural result outlining four sufficient conditions to achieve projection-cost preservation. These condit…
Surveying probabilistic real algebraic geometry.
problem Classical problems in real algebraic geometry.
method Probabilistic perspective on classical topics.
result Modern approach to Hilbert's Sixteenth Problem.
Capacity control, the bias/variance dilemma, and learning unknown functions from data, are all concerned with identifying effective and consistent fits of unknown geometric loci to random data points. A geometric locus is a curve or surface formed by points, all of which possess some uniform property. A geometric locus…
Computes expected number of real intersection points of essential variety with random linear spaces.
problem Computing the expected number of real intersection points of the essential variety with random linear spaces.
method Two probability distributions for linear spaces: invariant under orthogonal group action and one motivated from computer vision. Used Monte Carlo simulation for the latter.
result Expected number of real intersection points lies in the interval (3.95 - 0.05, 3.95 + 0.05) with high probability.
Dimension reduction is the process of embedding high-dimensional data into a lower dimensional space to facilitate its analysis. In the Euclidean setting, one fundamental technique for dimension reduction is to apply a random linear map to the data. This dimension reduction procedure succeeds when it preserves certain …
FCM efficiently approximates committor function with interpretable kernel model.
problem Approximating committor function in stochastic systems.
method Kernel-based approach using randomized linear algebra.
result FCM outperforms neural networks in accuracy and training speed.
We develop an HMC algorithm to easily marginalize random effects in LMMs.
problem Bayesian inference in LMMs is challenging, especially marginalizing random effects.
method Developed an HMC algorithm to marginalize random effects in LMMs efficiently.
result Marginalization is always beneficial when applicable and improves various models, especially cognitive science models.
Efficient kernel methods for large datasets using GPU acceleration.
problem Handling large-scale nonparametric learning problems efficiently.
method Preconditioned gradient solver, GPU acceleration, parallelization, out-of-core linear algebra, numerical precision optimization.
result Dramatic speedups on datasets with billions of points, maintaining state-of-the-art performance.
SALSA efficiently approximates leverage scores for big data, improving ARMA model fitting.
problem Efficiently approximating leverage scores for large matrices.
method Sequential approximate leverage-score algorithm (SALSA) using randomized numerical linear algebra.
result SALSA approximates leverage scores within (1+O(ε)) with high probability. Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields. The aim of the paper is to make modern inference methods (such as variational inference or gradient-based sampling) available for Gaussian models with band…
Study Poisson algebras for Hamiltonian systems linearization.
problem Linearize dynamics along Poisson submanifolds.
method Use contravariant derivative to characterize Poisson algebras.
result Infinitesimal Poisson algebras provide a framework for Hamiltonization.
New algorithm trains neural networks in near-linear time, overcoming slow convergence issues.
problem Slow convergence and computational overhead in training deep neural networks.
method Reformulates Gauss-Newton iteration as an ℓ2-regression problem and uses Fast-JL dimension reduction.
result Achieves an O(mn)-time algorithm for training ReLU networks, near-linear in dimension.
Simplified proof for dimension reduction of polygonal curves.
problem Preserving the continuous Fréchet distance of polygonal curves.
method Sparse oblivious subspace embeddings for generalized dissimilarity measures.
result Generalized dimension reduction technique works for various distance measures.
Accelerates signature kernel computation for sequences.
problem Severe computational bottleneck in computing signature kernel.
method Random Fourier features to accelerate signature kernel computation.
result Uniform approximation guarantees for unbiased estimator with linear computation time.
This chapter introduces quaternion machine learning for 3D rotations.
problem Lack of quaternion machine learning for 3D rotations.
method Augmented statistics, widely linear models, quaternion calculus, mean square estimation.
result Foundation for quaternion machine learning.
Conjugate gradient methods improve efficiency for high-dimensional GLMMs.
problem Efficiency bottleneck in computing high-dimensional GLMM precision matrices.
method Combining spectral analysis and random graph theory with conjugate gradient methods.
result CG-based methods achieve linear scaling in cost with model parameters and observations.
Investigates linearity of group amalgams and new examples of non-linear groups.
problem Linearity of group amalgams and examples of non-linear groups.
method Investigates linearity of amalgams of subgroups of algebraic groups.
result Establishes linearity of certain 'doubles' of linear groups and finds new non-linear examples.
CoLA automates efficient numerical linear algebra for complex matrix structures.
problem Efficiently solving large-scale linear algebra problems with complex matrix structures.
method Combining linear operator abstraction with compositional dispatch rules.
result Automatic and efficient numerical algorithms for various linear algebra operations.
Corrects bias in random sampling matrices for improved ML methods.
problem Inversion bias in random sampling matrices hampers ML applications.
method Corrects inversion bias for various random sampling methods.
result Establishes local convergence rates for sub-sampled Newton methods.
The technological applications of hidden Markov models have been extremely diverse and successful, including natural language processing, gesture recognition, gene sequencing, and Kalman filtering of physical measurements. HMMs are highly non-linear statistical models, and just as linear models are amenable to linear a…
The paper analyzes heavy-tailed multivariate distributions in non-stationary systems using random matrix theory.
problem Risk assessment for rare events in complex, non-stationary systems.
method Generalized scalar product between correlation matrices, model for non-stationary fluctuations.
result Formulae for multivariate distributions with reduced parameters, facilitating applications.
The statistical analysis of Randomized Numerical Linear Algebra (RandNLA) algorithms within the past few years has mostly focused on their performance as point estimators. However, this is insufficient for conducting statistical inference, e.g., constructing confidence intervals and hypothesis testing, since the distri…
Survey on strong convergence in random matrices and its applications.
problem Understanding convergence of random matrices to operators.
method Analysis of operator norms of noncommutative polynomials.
result New insights and applications in random graphs, geometry, and operator algebras.
Several important families of computational and statistical results in machine learning and randomized algorithms rely on uniform bounds on quadratic forms of random vectors or matrices. Such results include the Johnson-Lindenstrauss (J-L) Lemma, the Restricted Isometry Property (RIP), randomized sketching algorithms, …
Researchers develop Malliavin calculus for signatures, simplifying option Greeks computation.
problem Lack of tractability and explicit representations in Malliavin calculus.
method Focus on finite linear combinations of time-extended Brownian motion signatures, derive explicit formulas for Malliavin derivative, and compute Greeks for path-dependent options.
result Closed-form expressions for classical operators of Malliavin calculus, providing algebraic formulations.
This paper optimizes sampling for least-squares approximation.
problem Optimizing sampling for least-squares approximation in arbitrary linear spaces.
method Introducing the Christoffel function to construct near-optimal random sampling strategies.
result The number of samples scales log-linearly in the dimension of the approximation space.
Paper reviews algebraic research in machine learning theory.
problem Understanding phase transitions in machine learning models.
method Algebraic approaches in statistical mechanics.
result Algebraic methods are essential for analyzing machine learning models with singularities.
Random feature mapping (RFM) is a popular method for speeding up kernel methods at the cost of losing a little accuracy. We study kernel ridge regression with random feature mapping (RFM-KRR) and establish novel out-of-sample error upper and lower bounds. While out-of-sample bounds for RFM-KRR have been established by …
Unified determinants via a single equation.
problem Defining determinants with all known properties.
method Proposing a single equation implying all known properties of determinants.
result Unified definition of determinants with all properties.
A new algorithm approximates logistic regression probabilities efficiently.
problem Efficiently approximating probabilities in logistic regression for large datasets.
method Randomized sampling-based algorithm with leverage scores.
result Accurate approximations to estimated probabilities with smaller sample sizes.
Investigates integrable systems with linear periodic integral for e(3) Lie algebra.
problem Analyzes singularities and topological properties of integrable systems.
method Examines singularities of Liouville foliation, bifurcation diagram, transformations of Liouville tori, and isoenergy surfaces.
result Discovers topological properties of integrable systems with linear periodic integral.
Fisher discriminant analysis (FDA) is a widely used method for classification and dimensionality reduction. When the number of predictor variables greatly exceeds the number of observations, one of the alternatives for conventional FDA is regularized Fisher discriminant analysis (RFDA). In this paper, we present a simp…
We show that formal isomorphism of intransitive linear Lie equations along transversal to the orbits can be extended to neighborhoods of these transversal. In analytic cases, the word formal is dropped from theorems. Also, we associate an intransitive Lie algebra with each intransitive linear Lie equation, and from the…
A new method improves convergence in low-rank approximation.
problem Efficiently solving large-scale numerical linear algebra problems.
method Error-Powered Sketched Inverse Iteration (EPSI) Method.
result Convergence rate improves at least linearly with sketch size.
We present a new, unifying approach following some recent developments on the complexity of neural networks with piecewise linear activations. We treat neural network layers with piecewise linear activations as tropical polynomials, which generalize polynomials in the so-called (max,+) or tropical algebra, with pos…
New basis and Schur-Weyl duality for loop Hecke algebra defined.
problem Define a new basis for the loop Hecke algebra.
method Use higher linear rewriting theory and combinatorics of Dyck paths.
result Yields a conjecture of Damiani-Martin-Rowell and provides a representation theoretic interpretation.
Efficiently solves inverse PDE problems with Gaussian processes.
problem Solving inverse problems in linear PDEs with noisy data.
method Gaussian process regression with algebraic priors.
result High accuracy and computational efficiency achieved.
New method solves optimal stopping problems using rough path signatures.
problem Optimal stopping problems in finance and other fields.
method Using rough path signatures and deep neural networks.
result Solves optimal stopping problems efficiently under minimal assumptions.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner …