arXiv research

Singular Value Decomposition (SVD) constitutes a bridge between the linear algebra concepts and multi-layer neural networks---it is their linear analogy. Besides of this insight, it can be used as a good initial guess for the network parameters, leading to substantially better optimization results.

Study evaluates thresholds for removing noise from DNN weights using random matrix theory.

Fast and accurate methods for low-rank learning problems.

Decomposing tensors into orthogonal factors is a well-known task in statistics, machine learning, and signal processing. We study orthogonal outer product decompositions where the factors in the summands in the decomposition are required to be orthogonal across summands, by relating this orthogonal decomposition to the…

In this article, we consider the sparse tensor singular value decomposition, which aims for dimension reduction on high-dimensional high-order data with certain sparsity structure. A method named Sparse Tensor Alternating Thresholding for Singular Value Decomposition (STAT-SVD) is proposed. The proposed procedure featu…

We compared the regular Singular Value Decomposition (SVD), truncated SVD, Krylov method and Randomized PCA, in terms of time and space complexity. It is well-known that Krylov method and Randomized PCA only performs well when k << n, i.e. the number of eigenpair needed is far less than that of matrix size. We compared…

Derives a primal-dual MLSVD formulation for multilinear data.

Singular value decomposition (SVD) is the mathematical basis of principal component analysis (PCA). Together, SVD and PCA are one of the most widely used mathematical formalism/decomposition in machine learning, data mining, pattern recognition, artificial intelligence, computer vision, signal processing, etc. In recen…

Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…

We extend the randomized singular value decomposition (SVD) algorithm \citep{Halko2011finding} to estimate the SVD of a shifted data matrix without explicitly constructing the matrix in the memory. With no loss in the accuracy of the original algorithm, the extended algorithm provides for a more efficient way of matrix…

We extend the Kamada-Miyazawa polynomial to virtual singular links, which is valued in $\mathbb{Z}[A^2, A^{-2}, h]$. The decomposition of the resulting polynomial into two components, one in $\mathbb{Z}[A^2, A^{-2}]$ and the other in $\mathbb{Z}[A^2, A^{-2}]h$ yields the decomposition of the Kauffman-Jones polynomial o…

The paper updates SVD of evolving matrices using projection techniques.

Informed by recent work on tensor singular value decomposition and circulant algebra matrices, this paper presents a new theoretical bridge that unifies the hypercomplex and tensor-based approaches to singular value decomposition and robust principal component analysis. We begin our work by extending the principal comp…

New technique stabilizes singular values in concatenated matrices.

A low rank matrix X has been contaminated by uniformly distributed noise, missing values, outliers and corrupt entries. Reconstruction of X from the singular values and singular vectors of the contaminated matrix Y is a key problem in machine learning, computer vision and data science. In this paper we show that common…

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

New insights into X-ray transform on hyperbolic disk, with functional relations and range characterizations.

In this note, we report the back propagation formula for complex valued singular value decompositions (SVD). This formula is an important ingredient for a complete automatic differentiation(AD) infrastructure in terms of complex numbers, and it is also the key to understand and utilize AD in tensor networks.

Paper proposes a new optimization framework for learning eigenfunctions of operators.

In this paper, we develop the notion of entropy for uniform hypergraphs via tensor theory. We employ the probability distribution of the generalized singular values, calculated from the higher-order singular value decomposition of the Laplacian tensors, to fit into the Shannon entropy formula. We show that this tensor …

Deterministic bounds for tensor singular values and vectors, differing from matrix cases.

Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributi…

Fast matrix algorithms have become the fundamental tools of machine learning in big data era. The generalized matrix regression problem is widely used in the matrix approximation such as CUR decomposition, kernel matrix approximation, and stream singular value decomposition (SVD), etc. In this paper, we propose a fast …

Physics-inspired methods optimize SVD compression of LLMs.

New method cleans cross-covariance matrices for better financial forecasting.

This work studies the linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). Searching this approximation in a data-driven approach is formalised as attempting to solve a low-rank constrained optimisation problem. This problem is non-convex and state-of-the-art algor…

A new algorithm speeds up matrix operations in Neural Networks.

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically…

New method uses SVD entropy to price artworks.

For a one-parameter family of simple metrics of constant curvature ($4κ$ for $κ\in (-1,1)$) on the unit disk $M$, we first make explicit the Pestov-Uhlmann range characterization of the geodesic X-ray transform, by constructing a basis of functions making up its range and co-kernel. Such a range characterization also t…

Two multifidelity trust-region methods use low-fidelity models for efficient optimization.

From linear classifiers to neural networks, image classification has been a widely explored topic in mathematics, and many algorithms have proven to be effective classifiers. However, the most accurate classifiers typically have significantly high storage costs, or require complicated procedures that may be computation…

New method for causal effect estimation with hidden confounders.

In this paper, we study the possibility of inferring early warning indicators (EWIs) for periods of extreme bitcoin price volatility using features obtained from Bitcoin daily transaction graphs. We infer the low-dimensional representations of transaction graphs in the time period from 2012 to 2017 using Bitcoin blockc…

High throughput biomedical measurements normally capture multiple overlaid biologically relevant signals and often also signals representing different types of technical artefacts like e.g. batch effects. Signal identification and decomposition are accordingly main objectives in statistical biomedical modeling and data…

Principal component regression (PCR) is a two-stage procedure: the first stage performs principal component analysis (PCA) and the second stage constructs a regression model whose explanatory variables are replaced by principal components obtained by the first stage. Since PCA is performed by using only explanatory var…

Paper improves tensor completion by reducing sample entries needed.

Proof of complex geometry theorem for specific singular spaces.

The study analyzes XRP transaction networks to understand market dynamics.

Generalizes randomized SVD for better matrix approximations using Gaussian vectors.

With the abundance of data in recent years, interesting challenges are posed in the area of recommender systems. Producing high quality recommendations with scalability and performance is the need of the hour. Singular Value Decomposition(SVD) based recommendation algorithms have been leveraged to produce better result…

A new method selects regions of interest in GC-MS data without prior target selection.

Ranky solves SVD for large sparse matrices in distributed systems.

Tensor decompositions have rich applications in statistics and machine learning, and developing efficient, accurate algorithms for the problem has received much attention recently. Here, we present a new method built on Kruskal's uniqueness theorem to decompose symmetric, nearly orthogonally decomposable tensors. Unlik…

Two subset germs of Euclidean spaces are called blow-spherically equivalent, if their spherical modifications are homeomorphic and the homeomorphism induces homeomorphic tangent links. Blow-spherical equivalence is stronger than the topological equivalence but weaker than the Lipschitz equivalence. We introduce the thi…

New method estimates high-dimensional GoM models efficiently.

Many modern big data applications feature large scale in both numbers of responses and predictors. Better statistical efficiency and scientific insights can be enabled by understanding the large-scale response-predictor association network structures via layers of sparse latent factors ranked by importance. Yet sparsit…

We regard pre-trained residual networks (ResNets) as nonlinear systems and use linearization, a common method used in the qualitative analysis of nonlinear systems, to understand the behavior of the networks under small perturbations of the input images. We work with ResNet-56 and ResNet-110 trained on the CIFAR-10 dat…