arXiv research

TensorSketch is an oblivious linear sketch introduced in Pagh'13 and later used in Pham, Pagh'13 in the context of SVMs for polynomial kernels. It was shown in Avron, Nguyen, Woodruff'14 that TensorSketch provides a subspace embedding, and therefore can be used for canonical correlation analysis, low rank approximation…

MLP-Mixer achieves better performance through sparsity and wider architecture.

An algorithm for efficient computation of equivariant neural network layers.

We consider the problem of matrix approximation and denoising induced by the Kronecker product decomposition. Specifically, we propose to approximate a given matrix by the sum of a few Kronecker products of matrices, which we refer to as the Kronecker product approximation (KoPA). Because the Kronecker product is an ex…

This paper uncovers the low-rank structure of neural network Hessians.

A parallel algorithm learns efficient Kronecker product dictionaries.

Shampoo optimizes preconditioners for faster convergence in machine learning.

MCCA extracts shared structure from multiple tensor datasets.

This paper improves compression of large NLP models using doped Kronecker Products.

Kronecker product kernel provides the standard approach in the kernel methods literature for learning from graph data, where edges are labeled and both start and end vertices have their own feature representations. The methods allow generalization to such new edges, whose start and end vertices do not appear in the tra…

A Kronecker product model is the set of visible marginal probability distributions of an exponential family whose sufficient statistics matrix factorizes as a Kronecker product of two matrices, one for the visible variables and one for the hidden variables. We estimate the dimension of these models by the maximum rank …

Discovering the underlying low dimensional structure of high dimensional data has attracted a significant amount of researches recently and has shown to have a wide range of applications. As an effective dimension reduction tool, singular value decomposition is often used to analyze high dimensional matrices, which are…

Coupled nonlinear integrable systems are generated from usual zero curvature equation. The relevant Maurer-Cartan forms are constructed by combining suitably chosen matrices (nilpotent, Hadamard, idempotent and k-idempotent) and Lie algebraic elements via Kronecker product. In each case a closure type property among th…

A fast algorithm speeds up training of pairwise kernels.

Subspace clustering is a useful technique for many computer vision applications in which the intrinsic dimension of high-dimensional data is often smaller than the ambient dimension. Spectral clustering, as one of the main approaches to subspace clustering, often takes on a sparse representation or a low-rank represent…

New methods improve Fisher Matrix approximations for neural networks at low cost.

We study the Kronecker product regression problem, in which the design matrix is a Kronecker product of two or more matrices. Given $A_i \in \mathbb{R}^{n_i \times d_i}$ for $i=1,2,\dots,q$ where $n_i \gg d_i$ for each $i$, and $b \in \mathbb{R}^{n_1 n_2 \cdots n_q}$, let $\mathcal{A} = A_1 \otimes A_2 \otimes \cdots \…

This paper presents a new method for estimating high dimensional covariance matrices. The method, permuted rank-penalized least-squares (PRLS), is based on a Kronecker product series expansion of the true covariance matrix. Assuming an i.i.d. Gaussian random sample, we establish high dimensional rates of convergence to…

In this paper we consider the use of the space vs. time Kronecker product decomposition in the estimation of covariance matrices for spatio-temporal data. This decomposition imposes lower dimensional structure on the estimated covariance matrix, thus reducing the number of samples required for estimation. To allow a sm…

We describe the first sub-quadratic sampling algorithm for the Multiplicative Attribute Graph Model (MAGM) of Kim and Leskovec (2010). We exploit the close connection between MAGM and the Kronecker Product Graph Model (KPGM) of Leskovec et al. (2010), and show that to sample a graph from a MAGM it suffices to sample sm…

New algorithm estimates matrix-valued regression parameters efficiently.

This paper studies iteration convergence of Kronecker graphical lasso (KGLasso) algorithms for estimating the covariance of an i.i.d. Gaussian random sample under a sparse Kronecker-product covariance model and MSE convergence rates. The KGlasso model, originally called the transposable regularized covariance model by …

Paper proposes new gradient codes for robust distributed machine learning.

Proposes a novel graph learning framework for robust graph topology learning from graph signals.

A matrix completion problem is to recover the missing entries in a partially observed matrix. Most of the existing matrix completion methods assume a low rank structure of the underlying complete matrix. In this paper, we introduce an alternative and more general form of the underlying complete matrix, which assumes a …

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained e…

Study learns linear system dynamics from noisy bilinear data.

Paper develops SKPD framework for signal region detection in image regression.

Gaussian Conditional Random Fields (GCRF), as a structured regression model, is designed to achieve higher regression accuracy than unstructured predictors at the expense of execution time, taking into account the objects similarities and the outputs of unstructured predictors simultaneously. As most structural models,…

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

The state-sum invariants for knots and knotted surfaces defined from quandle cocycles are described using the Kronecker product between cycles represented by colored knot diagrams and a cocycle of a finite quandle used to color the diagram. Such an interpretation is applied to evaluating the invariants. Algebraic inter…

A scalable method for efficient inference in Gaussian process regression networks.

Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes m…

In this work we consider the problem of detecting anomalous spatio-temporal behavior in videos. Our approach is to learn the normative multiframe pixel joint distribution and detect deviations from it using a likelihood based approach. Due to the extreme lack of available training samples relative to the dimension of t…

Novel Bayesian framework for spatio-temporal neuroimaging data.

Scalable Gaussian processes with latent Kronecker structure for large datasets.

It is shown that the multiplicative monoids of Temperley-Lieb algebras generated out of the basis are isomorphic to monoids of endomorphisms in categories where an endofunctor is adjoint to itself. Such a self-adjunction is found in a category whose arrows are matrices, and the functor adjoint to itself is based on the…

New methods estimate covariance for matrix data without assuming fixed size or specific distributions.

The aim of this paper is to open the problem of construction of a nonlinear connection $Γ=(M^{(i)}_{(α)β}, N^{(i)}_{(α)j})$ on the jet bundle of first order $J^1(T,M)$, which to be canonically produced by a Kronecker product vertical metrical d-tensor $G^{(α)(β)}_{(i)(j)}=h^{αβ}g_{ij}$, possibly provided by multi-time …

It is shown that the multiplicative monoids of Brauer's centralizer algebras generated out of the basis are isomorphic to monoids of endomorphisms in categories where an endofunctor is adjoint to itself, and where, moreover, a kind of symmetry involving the self-adjoint functor is satisfied. As in a previous paper, of …

Bayesian method estimates Kronecker graphical models from autoregressive processes.

KTVGL models tensor time series data for interpretable dynamic network estimation.

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size. As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained …

DKN adapts to medical imaging data with limited samples and interpretable models.

Kronecker trend filtering improves lattice data smoothing.

Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner t…

Determinantal Point Processes (DPPs) are probabilistic models over all subsets a ground set of $N$ items. They have recently gained prominence in several applications that rely on "diverse" subsets. However, their applicability to large problems is still limited due to the $\mathcal O(N^3)$ complexity of core tasks suc…

We consider the problem of detecting whether a tensor signal having many missing entities lies within a given low dimensional Kronecker-Structured (KS) subspace. This is a matched subspace detection problem. Tensor matched subspace detection problem is more challenging because of the intertwined signal dimensions. We s…