Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogo…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “matrix scaling”
We propose a new method of learning a sparse nonnegative-definite target matrix. Our primary example of the target matrix is the inverse of a population covariance or correlation matrix. The algorithm first estimates each column of the target matrix by the scaled Lasso and then adjusts the matrix estimator to be symmet…
We study the problem of large-scale network embedding, which aims to learn latent representations for network mining applications. Previous research shows that 1) popular network embedding benchmarks, such as DeepWalk, are in essence implicitly factorizing a matrix with a closed form, and 2)the explicit factorization o…
Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.
problem Recovering a low-rank matrix from incomplete data with high condition number.
method Preconditioned Stochastic Gradient Descent (SGD) for huge-scale online optimization.
result Preconditioned SGD converges to ε-accuracy in O(log(1/ε)) iterations, compared to O(κlog(1/ε)) for unpreconditioned SGD.
PAC-Bayesian matrix completion with a spectral scaled Student prior offers efficient inference.
problem Matrix completion with underlying low-rank structure.
method Spectral scaled Student prior and PAC-Bayesian bounds.
result Minimax-optimal oracle inequality for model misspecification and general sampling distribution.
The Random Parameters model was proposed to explain the structure of the covariance matrix in problems where most, but not all, of the eigenvalues of the covariance matrix can be explained by Random Matrix Theory. In this article, we explore other properties of the model, like the scaling of its PDF as one take larger …
We uncover scaling laws and statistical structure in complex datasets.
problem Understanding universal traits in complex datasets.
method Analogizing data to physical systems, using statistical physics and RMT.
result Real-world datasets and Gaussian data with long-range correlations share the same RMT universality class.
The correlation matrix is the key element in optimal portfolio allocation and risk management. In particular, the eigenvectors of the correlation matrix corresponding to large eigenvalues can be used to identify the market mode, sectors and style factors. We investigate how these eigenvalues depend on the time scale of…
We consider the problem of matrix completion on an matrix. We introduce the problem of Interpretable Matrix Completion that aims to provide meaningful insights for the low-rank matrix using side information. We show that the problem can be reformulated as a binary convex optimization problem. We design Opt…
CAST improves spectral clustering for multi-scale data by integrating reachability similarity.
problem Applying spectral clustering to multi-scale data where clusters vary in size and density.
method CAST integrates reachability similarity with distance-based similarity to derive a coefficient matrix, then applies trace Lasso regularization.
result CAST provides excellent performance and robustness across various multi-scale data test cases.
We introduce a notion of "effective dimension" of a statistical model based on the number of cubes of size needed to cover the model space when endowed with the Fisher Information Matrix as metric, being the number of observations. The number of observations fixes a natural scale or resolution. The eff…
Paper combines scalable BMF algorithms for web-scale datasets.
problem High computational cost of Bayesian Matrix Factorization.
method Combines Posterior Propagation and asynchronous distributed implementation.
result Substantial improvements in scalability on web-scale datasets.
Method selects number of communities in weighted networks.
problem Selecting the number of communities in weighted networks.
method Proposes a novel weighted DCSBM and uses a sequential testing framework with spectral clustering and matrix scaling.
result Method is consistent in estimating the true number of communities under mild conditions.
Novel LRMC tackles missing data and outliers in large-scale low-rank data recovery.
problem Missing data and extreme outliers in low-rank data analysis.
method Learned Robust Matrix Completion (LRMC) using deep unfolding and flexible neural network framework.
result LRMC achieves optimum performance with low computational complexity and linear convergence.
Matrix Factorization (MF) on large scale matrices is computationally as well as memory intensive task. Alternative convergence techniques are needed when the size of the input matrix is higher than the available memory on a Central Processing Unit (CPU) and Graphical Processing Unit (GPU). While alternating least squar…
Exploiting low-rank structure of the user-item rating matrix has been the crux of many recommendation engines. However, existing recommendation engines force raters with heterogeneous behavior profiles to map their intrinsic rating scales to a common rating scale (e.g. 1-5). This non-linear transformation of the rating…
Paper proposes fast, robust methods for low-rank matrix recovery.
problem Estimating low-rank matrices from incomplete or corrupted data.
method Scaled subgradient methods for nonsmooth, nonconvex formulations.
result Methods converge almost dimension-free and condition-number independent.
Random matrix theory predicts neural representations generalize well.
problem Understanding why neural representations generalize well in practice.
method Applied random matrix theory to kernel regression and neural networks.
result GCV estimator accurately predicts generalization risk in overparameterized settings.
Representation learning is typically applied to only one mode of a data matrix, either its rows or columns. Yet in many applications, there is an underlying geometry to both the rows and the columns. We propose utilizing this coupled structure to perform co-manifold learning: uncovering the underlying geometry of both …
Paper proposes a novel method to improve matrix completion with median loss for large datasets.
problem Matrix completion with absolute deviation loss for large-scale data.
method Proposes a refinement step using pseudo data to improve inefficient estimators of median matrix completion.
result Turns inefficient estimators into a rate (near-)optimal matrix completion procedure.
Study reveals an equivalence principle for the spectrum of random inner-product kernel matrices in polynomial scaling.
problem Understanding the spectrum of random kernel matrices in polynomial scaling regimes.
method Investigates random matrices with nonlinear kernel functions applied to inner products of uniformly distributed vectors.
result The spectrum of the random kernel matrix is asymptotically equivalent to a simpler matrix model through free additive convolution.
Multiresolution Matrix Factorization (MMF) was recently introduced as an alternative to the dominant low-rank paradigm in order to capture structure in matrices at multiple different scales. Using ideas from multiresolution analysis (MRA), MMF teased out hierarchical structure in symmetric matrices by constructing a se…
We analyze the eigenvalue distribution of a neural network's kernel under specific scaling.
problem Analyzing the eigenvalue distribution of the Neural Tangent Kernel (NTK) of a neural network.
method Asymptotic analysis of the NTK matrix under given scaling conditions.
result The eigenvalue distribution is described as a free multiplicative convolution of the Marchenko-Pastur distribution and a deterministic distribution.
We address the problem of minimizing a convex function over the space of large matrices with low rank. While this optimization problem is hard in general, we propose an efficient greedy algorithm and derive its formal approximation guarantees. Each iteration of the algorithm involves (approximately) finding the left an…
Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projectionscs.LG
Bayesian deep learning avoids underfitting by projecting onto null space of generalized Gauss-Newton matrix.
problem Bayesian deep learning often underfits, leading to less accurate predictions than point estimates.
method Proposes a matrix-free algorithm to project onto the null space of the generalized Gauss-Newton matrix, ensuring Bayesian predictions do not underfit.
result The method scales to large models, including vision transformers with 28 million parameters, and avoids underfitting.
ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.
problem Efficiently solving ill-conditioned low-rank matrix estimation problems.
method Scaled Gradient Descent (ScaledGD) with adaptive pre-conditioners.
result Linear convergence rate independent of condition number, low per-iteration cost.
Sparse matrix factorization is a popular tool to obtain interpretable data decompositions, which are also effective to perform data completion or denoising. Its applicability to large datasets has been addressed with online and randomized methods, that reduce the complexity in one of the matrix dimension, but not in bo…
Boolean matrix factorization and Boolean matrix completion from noisy observations are desirable unsupervised data-analysis methods due to their interpretability, but hard to perform due to their NP-hardness. We treat these problems as maximum a posteriori inference problems in a graphical model and present a message p…
A Bayesian Graphical Approach for Large-Scale Portfolio Management with Fewer Historical Dataq-fin.PM
Bayesian method improves portfolio management with limited data.
problem Estimating covariance or precision matrix for large portfolios is challenging.
method Bayesian graphical LASSO for precision matrix estimation.
result The Bayesian approach outperforms non-Bayesian methods in stability and precision matrix estimation.
Low-rank matrix approximations are often used to help scale standard machine learning algorithms to large-scale problems. Recently, matrix coherence has been used to characterize the ability to extract global information from a subset of matrix entries in the context of these low-rank approximations and other sampling-…
A new method speeds up ALS for recommender systems by subsampling key elements.
problem High computational cost of ALS for large-scale datasets.
method Core-elements subsampling method for efficient ALS approximation.
result Achieves similar accuracy with significantly reduced computational time.
New method decomposes corrupted data matrices into sparse and low-rank components.
problem Decomposing corrupted data matrices into sparse and low-rank components.
method Discrete optimization approach with alternating minimization, semidefinite relaxation, and branch-and-bound algorithm.
result High-quality solutions and meaningful bounds for SLR problems.
This paper optimizes matrix-based Renyi's entropy computation for large datasets.
problem Efficiently calculating matrix-based Renyi's entropy for large-scale applications.
method Develops randomized approximations for matrix-based Renyi's entropy with arbitrary α orders.
result Achieves a significant reduction in time complexity from O(n^3) to O(n^2sm), where s, m << n.
Muon outperforms GD in associative memory learning by balancing frequency components.
problem Training dynamics and scaling behavior of Muon in associative memory learning.
method Study of Muon in a linear associative memory model with softmax retrieval and hierarchical frequency spectrum over query-answer pairs.
result Muon achieves exponential speedup over GD in noiseless case and superior scaling efficiency in noisy case.
Recurrent Neural Networks (RNNs) are designed to handle sequential data but suffer from vanishing or exploding gradients. Recent work on Unitary Recurrent Neural Networks (uRNNs) have been used to address this issue and in some cases, exceed the capabilities of Long Short-Term Memory networks (LSTMs). We propose a simp…
Large-scale generalized linear array models (GLAMs) can be challenging to fit. Computation and storage of its tensor product design matrix can be impossible due to time and memory constraints, and previously considered design matrix free algorithms do not scale well with the dimension of the parameter vector. A new des…
The matrix completion problem consists of finding or approximating a low-rank matrix based on a few samples of this matrix. We propose a new algorithm for matrix completion that minimizes the least-square distance on the sampling set over the Riemannian manifold of fixed-rank matrices. The algorithm is an adaptation of…
Study uncovers scaling laws and spectral properties of shallow neural networks.
problem Understanding scaling laws and spectral properties of shallow neural networks.
method Leveraging connections with matrix compressed sensing and LASSO, derived a phase diagram for excess risk.
result Uncovered crossovers between scaling regimes and plateau behaviors, validated empirical observations.
Matrix H-theory models stock market fluctuations using hierarchical multivariate distributions.
problem Understanding collective behavior in stock market fluctuations.
method Matrix H-theory framework for multivariate stochastic processes with hierarchical structure.
result Matrix H-theory effectively describes stock market fluctuations using Meijer G-functions.
Recovering low-rank and sparse matrices from incomplete or corrupted observations is an important problem in machine learning, statistics, bioinformatics, computer vision, as well as signal and image processing. In theory, this problem can be solved by the natural convex joint/mixed relaxations (i.e., l_{1}-norm and tr…
Improved stability for large-scale Bayesian sampling.
problem Reducing instability in Langevin dynamics for large datasets.
method Introducing a modified CCAdL thermostat with a scaling and squaring method and a truncated Taylor series approximation.
result Significantly improved numerical stability and accuracy over existing methods.
Study on SGD dynamics and scaling laws for training quadratic neural networks in high dimensions.
problem Optimizing and understanding the training dynamics of quadratic neural networks in high-dimensional settings.
method Sharp analysis of SGD dynamics, combining matrix Riccati differential equations and matrix monotonicity arguments.
result Derivation of scaling laws for prediction risk, highlighting power-law dependencies on optimization time, sample size, and model width.
In this paper, we examine the problem of approximating a general linear dimensionality reduction (LDR) operator, represented as a matrix with , by a partial circulant matrix with rows related by circular shifts. Partial circulant matrices admit fast implementations via Fourier tra…
Fast Convolutive Nonnegative Matrix Factorization Through Coordinate and Block Coordinate Updatescs.LG
Identifying recurring patterns in high-dimensional time series data is an important problem in many scientific domains. A popular model to achieve this is convolutive nonnegative matrix factorization (CNMF), which extends classic nonnegative matrix factorization (NMF) to extract short-lived temporal motifs from a long …
New insights into learning rates and batch sizes for neural networks using random matrix theory.
problem Understanding how batch size affects learning rates in neural networks.
method Random matrix theory applied to spiked, field-dependent random matrices.
result Analytical expressions for maximal learning rates as a function of batch size.
In this paper, we propose an efficient and scalable low rank matrix completion algorithm. The key idea is to extend orthogonal matching pursuit method from the vector case to the matrix case. We further propose an economic version of our algorithm by introducing a novel weight updating rule to reduce the time and stora…
SNS accelerates Sinkhorn algorithm with sparse Newton iterations.
problem Slow runtime of Sinkhorn algorithm for optimal transport.
method Early stopping and Newton-type subroutine for matrix scaling steps.
result SNS converges orders of magnitude faster than Sinkhorn.
Researchers use quantum chaos and RMT to analyze turbulence, revealing unique scaling laws.
problem Understanding the statistical structure and scaling laws of turbulence.
method Applied tools from quantum chaos and Random Matrix Theory to analyze turbulence datasets.
result Turbulence Gram matrices exhibit power-law scalings distinct from classical chaos and random data.