TensorSketch solves Kronecker product regression and non-negative regression.
problem Solving regression problems with Kronecker product matrices.
method Extending TensorSketch to other norms for Kronecker product regression.
result Solving Kronecker product regression and non-negative regression in sublinear time.
Estimates the dimension of Kronecker product models using Jacobian rank and tropical morphism.
problem Estimating the dimension of Kronecker product models.
method Using Jacobian rank and tropical morphism to describe the limit of the model.
result Combinatorial conditions for the expected dimension and proof for binary restricted Boltzmann machine.
Efficiently trains Kronecker product kernel methods for graph data.
problem Learning from graph data with labeled edges and feature representations.
method Generalizes vec trick approach to non-complete training graphs.
result Order of magnitude improvements in training and prediction time.
KoPA approximates matrices using Kronecker products for better flexibility.
problem Matrix approximation and denoising with Kronecker product decomposition.
method Approximate a matrix as a sum of Kronecker products of smaller matrices using extended information criteria for configuration selection.
result KoPA selects the true configuration with high probability under suitable conditions.
Efficient subspace clustering using Kronecker product reduces computational complexity.
problem Efficiency and scalability issues in traditional subspace clustering methods for large datasets.
method Proposes a subspace clustering model based on the Kronecker product to reduce computational complexity.
result Significantly improved efficiency compared to state-of-the-art methods on public datasets.
Scalable Gaussian processes with latent Kronecker structure for large datasets.
problem Limited scalability of Gaussian processes for large datasets.
method Leveraging latent Kronecker structure, projecting kernel matrix onto latent Kronecker product, using iterative linear system solvers and pathwise conditioning.
result Outperforms state-of-the-art sparse and variational GPs on real-world datasets with up to five million examples.
This paper improves compression of large NLP models using doped Kronecker Products.
problem Accuracy loss when compressing large NLP tasks with Kronecker Products.
method Doping Kronecker Products with an overlay matrix to recover accuracy, and a new regularization scheme called co matrix dropout regularization (CMR).
result Compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score.
The present paper discusses that a prescribed Gauss-Kronecker curvature problem on the product of unit spheres.
Bayesian method estimates Kronecker graphical models from autoregressive processes.
problem Estimating Kronecker graphical models from autoregressive Gaussian processes.
method Bayesian approach to estimate Kronecker graphical models.
result Effectiveness demonstrated through numerical experiments and real-world data application.
Paper proposes a new method for approximating high-dimensional matrices using Kronecker products.
problem Discovering low-dimensional structure in high-dimensional data.
method Hybrid Kronecker Product Approximation (hKoPA) and estimation procedures.
result The proposed methods provide flexible and effective dimension reduction.
Machine learning predicts Kronecker coefficients with high accuracy.
problem Predicting Kronecker coefficients from tensor products of symmetric group representations.
method Training machine learning models (NN, CNN, GBDT) to classify Kronecker coefficients as zero or non-zero.
result Trained models achieve high accuracy (≈0.98) in classifying Kronecker coefficients. Kronecker DPPs enable efficient sampling and learning for large DPP problems.
problem Efficient sampling and learning for large Determinantal Point Processes (DPPs).
method Introducing KronDPP, a DPP model with a tensor product kernel matrix, enabling fast exact sampling. Overcoming challenges in learning parameters efficiently.
result Efficient algorithms for learning parameters of KronDPP, overcoming the difficulty of leveraging Kronecker product structure.
Efficiently models learning curves using Gaussian processes with latent Kronecker structure.
problem Joint modeling of machine learning model performance across hyper-parameters and training progress.
method Imposes latent Kronecker structure to leverage efficient product kernels and handle missing values.
result Matches the performance of a Transformer on a learning curve prediction task.
This paper studies iteration convergence of Kronecker graphical lasso (KGLasso) algorithms for estimating the covariance of an i.i.d. Gaussian random sample under a sparse Kronecker-product covariance model and MSE convergence rates. The KGlasso model, originally called the transposable regularized covariance model by …
A parallel algorithm learns efficient Kronecker product dictionaries.
problem Sparse representation of 2D signals like images and hyperspectral data.
method Highly parallelizable algorithm for learning separable dictionaries.
result Competitive sparse representations at lower computational cost.
Shampoo optimizes preconditioners for faster convergence in machine learning.
problem Improving convergence speed in machine learning optimization.
method Explicit connection between Shampoo's Kronecker product approximation and optimal matrix approximations.
result The square of Shampoo's approximation is equivalent to a single power iteration step for optimal Kronecker product approximation.
This paper compresses RNNs for IoT devices by 15-38x using Kronecker products.
problem Resource constraints on IoT devices make RNNs difficult to deploy.
method Kronecker product (KP) for compressing RNN layers by 15-38x with minimal accuracy loss.
result Kronecker product can compress RNNs by 50x when quantized to 8-bits.
MCCA extracts shared structure from multiple tensor datasets.
problem Extracting shared structure from multiple tensor datasets.
method Multilinear common component analysis (MCCA) using Kronecker products of mode-wise covariance matrices.
result MCCA constructs a common basis that retains information from multiple tensor datasets.
New method for matrix completion using Kronecker product approximation.
problem Matrix completion with low Kronecker rank structure.
method Alternative matrix representation using Kronecker product, identification through mean squared error and modified cross-validation.
result Consistency of the method under suitable signal-to-noise ratio conditions.
In this paper we consider the use of the space vs. time Kronecker product decomposition in the estimation of covariance matrices for spatio-temporal data. This decomposition imposes lower dimensional structure on the estimated covariance matrix, thus reducing the number of samples required for estimation. To allow a sm…
EiGLasso speeds up sparse Kronecker-sum covariance estimation.
problem Sparse Kronecker-sum inverse covariance estimation challenges in scalability and parameter identification.
method Newton's method combined with eigendecomposition of sample and feature graphs, approximating Hessian for speed.
result Two to three orders-of-magnitude speed-up on simulated and real-world data.
A fast algorithm speeds up training of pairwise kernels.
problem Training pairwise kernels efficiently for large datasets.
method Generalized vec trick for Kronecker product kernels.
result Pairwise kernels can be expressed as sums of Kronecker products.
New integrable systems are created using matrix operations and Lie algebra elements.
problem Generating coupled nonlinear integrable systems from zero curvature equation.
method Constructing Maurer-Cartan forms using Kronecker product and specific matrices (nilpotent, Hadamard, idempotent, k-idempotent).
result Found a closure property among chosen matrices crucial for coupling and nonlinearity.
Improved GCRF model for multi-scale networks with reduced computation time.
problem Scaling issues in GCRF for large networks.
method Factor graph decomposition using Kronecker product, new Laplacian eigenvalue estimations.
result High prediction accuracy with improved computational complexity.
New method for scalable stochastic neural networks using Kronecker Flow.
problem Scaling stochastic neural networks to high dimensions.
method Kronecker Flow, a scalable parameterization of noise generation.
result Competitive performance on various tasks compared to existing methods.
Faster algorithms for Kronecker product regression and low rank approximation.
problem Efficiently solving Kronecker product regression and low rank approximation problems.
method Developed faster algorithms for Kronecker product regression and low rank approximation.
result Significantly faster algorithms for Kronecker product regression and low rank approximation.
How can we model networks with a mathematically tractable model that allows for rigorous analysis of network properties? Networks exhibit a long list of surprising properties: heavy tails for the degree distribution; small diameters; and densification and shrinking diameters over time. Most present network models eithe…
Proposes a method to classify with matrix-valued predictors using penalized likelihood.
problem Classification with matrix-valued predictors.
method Penalized likelihood method with Kronecker product decomposition for precision matrix estimation.
result Outperforms competitors in classification accuracy, even when assumptions are violated.
We investigate complete minimal hypersurfaces in the Euclidean space , with Gauss-Kronecker curvature identically zero. We prove that, if f:M3→R4 is a complete minimal hypersurface with Gauss-Kronecker curvature identically zero, nowhere vanishing second fundamental form and scalar curvature b…
New methods improve Fisher Matrix approximations for neural networks at low cost.
problem High cost of solving Fisher Information Matrix (FIM) in neural networks.
method Direct minimization via Kronecker product singular value decomposition.
result Improved approximations to FIM provide more accurate and faster optimization.
Kronecker trend filtering improves lattice data smoothing.
problem Estimating smooth functions on lattice data.
method Penalized least squares with Kronecker products of univariate trend filtering penalties.
result Kronecker trend filtering outperforms linear smoothers in high dimensions.
This paper presents a new method for estimating high dimensional covariance matrices. The method, permuted rank-penalized least-squares (PRLS), is based on a Kronecker product series expansion of the true covariance matrix. Assuming an i.i.d. Gaussian random sample, we establish high dimensional rates of convergence to…
DKN adapts to medical imaging data with limited samples and interpretable models.
problem Medical imaging data's unique nature makes general methods like CNN unsuitable.
method DKN uses a Kronecker product structure to adapt to low sample size and provide interpretable models.
result DKN achieves prediction power comparable to CNN and provides model interpretability.
STARK learns structured dictionaries for tensor data.
problem Representing multidimensional data with structured dictionaries.
method Solves a convex relaxation of a nonconvex rank-1 tensor recovery problem.
result Empirical results show promising performance for tensors of any order.
Detects missing tensor signals in a KS subspace with high probability.
problem Detecting tensor signals with many missing entities in a KS subspace.
method Projecting the signal onto the KS subspace and bounding residual energy.
result Reliable detection is possible if the missing signal cardinality exceeds KS subspace dimensions.
KTVGL models tensor time series data for interpretable dynamic network estimation.
problem Estimating time-varying dependencies in multi-mode tensor time series data.
method Kronecker Time-Varying Graphical Lasso (KTVGL) for mode-specific dynamic network estimation.
result KTVGL produces interpretable modeling results and higher edge estimation accuracy than existing methods.
Paper sets limits on tensor data dictionary learning sample complexity.
problem Estimating atomic elements for tensor data with sparse representation.
method Proves minimax lower bound on sample complexity for Kronecker-structured dictionaries.
result Sample complexity for tensor data can be significantly lower than for unstructured data.
New framework for scalable Gaussian processes reduces complexity to O(n).
problem Handling large datasets with Gaussian processes efficiently.
method Decomposing covariance matrices, local kernel interpolation, and BTTB structure.
result Significant reduction in computational complexity to O(n) and O(1) for predictions.
Paper introduces online second order methods for non-convex stochastic optimization.
problem Non-convex stochastic optimization problems.
method Enhanced preconditioned stochastic gradient descent (PSGD) with improved implementations.
result Demonstrates PSGD's advantages in generalization and convergence speed.
We describe the first sub-quadratic sampling algorithm for the Multiplicative Attribute Graph Model (MAGM) of Kim and Leskovec (2010). We exploit the close connection between MAGM and the Kronecker Product Graph Model (KPGM) of Leskovec et al. (2010), and show that to sample a graph from a MAGM it suffices to sample sm…
In this work we consider the problem of detecting anomalous spatio-temporal behavior in videos. Our approach is to learn the normative multiframe pixel joint distribution and detect deviations from it using a likelihood based approach. Due to the extreme lack of available training samples relative to the dimension of t…
KFC approximates Fisher matrix for faster SGD in convolutional nets.
problem Efficiently computing natural gradient for large convolutional models.
method Structured probabilistic model and Kronecker decomposition for Fisher matrix approximation.
result Approximate natural gradient with KFC trains convolutional nets faster than SGD.
A new optimization method reduces memory and compute requirements for deep learning.
problem Memory and compute constraints in second-order stochastic optimizers for deep learning.
method Proposes KrAD, a novel factorization to approximate inverse Fisher matrix without inversion, leading to KrADagrad.
result Improves performance over Shampoo for 32-bit precision and comparable/generalization on real datasets.
Adaptive regularization improves neural network performance on small datasets.
problem Improving neural network performance on limited data.
method Adaptive regularization using a matrix-variate normal prior with a Kronecker product structure.
result The method leads to networks with smaller stable ranks and spectral norms, suggesting better generalization.
A new method for optimizing deep neural networks using TKFAC.
problem Optimizing deep neural networks with second-order methods.
method Proposes Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) for Fisher information matrix approximation.
result TKFAC improves performance on deep network architectures compared to state-of-the-art algorithms.
Paper compresses RNNs for resource-constrained devices.
problem Difficulty deploying RNNs on resource-constrained devices.
method Uses Kronecker product (KP) to compress RNN layers.
result KP compresses RNN layers by 16-38x with minimal accuracy loss.
New methods use Kronecker-factored approximations for faster deep learning optimization.
problem Optimizing deep learning models with rich curvature information.
method Approximate Hessian using Kronecker products for efficient quasi-Newton methods.
result New methods outperform first-order methods and perform comparably to second-order methods.
Proposes a novel graph learning framework for robust graph topology learning from graph signals.
problem Graph learning for revealing node relationships in data entities.
method Functional learning with smoothness-promoting graph learning, incorporating Kronecker product kernel.
result Improves robustness against missing and incomplete information in graph signals.