The paper explains implicit regularization in hierarchical tensor factorization and deep CNNs.
problem Understanding implicit regularization in complex neural network architectures.
method Theoretical analysis using dynamical systems to overcome challenges in hierarchy.
result Established implicit regularization towards low hierarchical tensor rank, equivalent to locality in CNNs.
Proposes a nonparametric tensor factorization for sparse data.
problem Handling sparse tensor data with structural and interpretability benefits.
method Hierarchical Gamma processes and Poisson random measures for tensor-valued process, Dirichlet processes for sampling entry indices, Gaussian processes for values.
result Demonstrates superior performance on benchmark datasets.
New hierarchical tensor decomposition model for complex data.
problem Lack of natural generalization of hierarchical NMF to tensors.
method Proposes a new hierarchical nonnegative tensor decomposition (HNTF) model.
result Model more naturally illuminates topic hierarchy.
Visual objects are composed of a recursive hierarchy of perceptual wholes and parts, whose properties, such as shape, reflectance, and color, constitute a hierarchy of intrinsic causal factors of object appearance. However, object appearance is the compositional consequence of both an object's intrinsic and extrinsic c…
Bayesian hierarchical tensor factorization model for international trade flows
problem Sparse semi-continuous tensor data modeling
method Bayesian hierarchical tensor factorization with Poisson and Gamma models
result Identifies multiway dependence in trade flows
CANDECOMP/PARAFAC (CP) tensor factorization of incomplete data is a powerful technique for tensor completion through explicitly capturing the multilinear latent factors. The existing CP algorithms require the tensor rank to be manually specified, however, the determination of tensor rank remains a challenging problem e…
Introduces nondecreasing rank for matrices and tensors, developing methods and applications.
problem Finding low-rank approximations for matrices and tensors with monotonic constraints.
method Developed a variant of hierarchical alternating least squares algorithm for finding low ND rank approximations.
result Low ND rank factorizations can be found and interpreted for real-world datasets.
NCPF model improves traffic data imputation with neural and tensor methods.
problem Pervasive missing data in traffic analysis due to sensor failures and gaps.
method Neural Canonical Polyadic Factorization (NCPF) integrating CP decomposition and deep learning.
result NCPF outperforms state-of-the-art baselines in urban traffic datasets.
This paper aims at the problem of link pattern prediction in collections of objects connected by multiple relation types, where each type may play a distinct role. While common link analysis models are limited to single-type link prediction, we attempt here to capture the correlations among different relation types and…
New method approximates high-dimensional probability densities efficiently.
problem Approximating high-dimensional probability densities accurately and efficiently.
method Hierarchical tensor-network approach using randomized SVD and linear equations.
result The method effectively approximates high-dimensional densities with linear complexity.
Bayesian TNKMs automatically infer model complexity and feature relevance.
problem Manual tuning of TN rank and feature dimensions is error-prone and computationally expensive.
method Bayesian approach with hierarchical priors on TN factors for automatic rank and feature selection.
result Superior performance in prediction accuracy, uncertainty quantification, interpretability, and scalability.
We introduce Bayesian multi-tensor factorization, a model that is the first Bayesian formulation for joint factorization of multiple matrices and tensors. The research problem generalizes the joint matrix-tensor factorization problem to arbitrary sets of tensors of any depth, including matrices, can be interpreted as u…
It has long been conjectured that hypotheses spaces suitable for data that is compositional in nature, such as text or images, may be more efficiently represented with deep hierarchical networks than with shallow ones. Despite the vast empirical evidence supporting this belief, theoretical justifications to date are li…
Neural NMF discovers hierarchical topics in multilayer data.
problem Detecting latent hierarchical structure in multilayer data.
method Recursive application of nonnegative matrix factorization (NMF) in layers with backpropagation optimization.
result Neural NMF outperforms other hierarchical NMF methods in synthetic and real-world datasets.
Efficiently price high-dimensional Bermudan options using tensor compression.
problem High-dimensional option pricing with computational complexity.
method Hierarchical tensor compression for Monte Carlo and dual martingale methods.
result Tensor compression alleviates the curse of dimensionality for Bermudan option pricing.
Boxhead dataset tests autoencoder disentanglement in hierarchical data.
problem Evaluate disentanglement in hierarchical data.
method Introduced Boxhead dataset with hierarchically structured factors, evaluated autoencoder models.
result Hierarchical models outperform single-layer VAEs in disentangling factors.
Paper compresses RNNs using HT decomposition for better performance.
problem Large model sizes of RNNs in sequence analysis.
method Hierarchical Tucker (HT) tensor decomposition for model compression.
result HT-LSTM achieves better compression and accuracy than state-of-the-art methods.
Tensor factorization arises in many machine learning applications, such knowledge base modeling and parameter estimation in latent variable models. However, numerical methods for tensor factorization have not reached the level of maturity of matrix factorization methods. In this paper, we propose a new method for CP te…
Proposes FATTNN for tensor-on-tensor regression with improved prediction and reduced computation.
problem Tensor-on-tensor regression with complex tensor structures and nonlinear relationships.
method Integrates tensor factor models into deep neural networks to handle nonlinearity and reduce data dimensionality.
result Significant improvements in prediction accuracy and computational efficiency over traditional methods.
We propose a nonparametric Bayesian factor regression model that accounts for uncertainty in the number of factors, and the relationship between factors. To accomplish this, we propose a sparse variant of the Indian Buffet Process and couple this with a hierarchical model over factors, based on Kingman's coalescent. We…
Proposes polynomial neural networks for improved function approximation in various tasks.
problem Improving function approximation in various tasks like image generation, face verification, and 3D mesh representation learning.
method Introduces polynomial neural networks (Π-Nets) and three tensor decompositions to reduce parameter count and enhance expressiveness. result Demonstrates that Π-Nets can produce state-of-the-art results in challenging tasks without non-linear activation functions. Deep tensor factorization benefits from implicit regularization with polynomial growth.
problem Tensor factorization's implicit regularization effect in deep networks is not well understood.
method Investigated the implicit regularization in deep tensor factorization, showing polynomial growth.
result Implicit regularization in deep tensor factorization grows polynomially with depth, improving estimation accuracy and convergence.
Optimal tensor PCA for estimating factors and loadings in high-dimensional panel data.
problem Estimating factors and loadings in high-dimensional panel data with non-negligible correlations.
method Tensor Principal Component Analysis (TPCA) for estimating factors and loadings in a tensor factor model.
result Simple TPCA is optimal for strong factors and can be improved for weak factors with alternating least-squares iterations.
The paper tackles tensor factorization and completion from noisy data.
problem Sparse nonnegative tensor factorization and completion from partial and noisy observations.
method Minimizes the sum of maximum likelihood estimation and tensor ℓ0 norm with nonnegativity constraints. result Error bounds and minimax lower bounds are established for the proposed model.
Proposes a new tensor factorization model for better link prediction in knowledge graphs.
problem Lack of information in treating missing and non-existing relations equally in tensor factorization models.
method Introduces a binary tensor factorization model with probit link to address the issue.
result Shows improved prediction accuracy and interpretability compared to existing models.
Paper speeds up tensor factorization algorithms.
problem Improving convergence speed of tensor factorization algorithms.
method Proposes an extrapolation strategy between block updates.
result HER significantly accelerates convergence speed.
Streaming tensor factorization is a powerful tool for processing high-volume and multi-way temporal data in Internet networks, recommender systems and image/video data analysis. Existing streaming tensor factorization algorithms rely on least-squares data fitting and they do not possess a mechanism for tensor rank dete…
Efficiently factorizes coupled matrix tensor data for better accuracy and speed.
problem Poor computation efficiency in existing N-CMTF algorithms.
method Column-wise element selection to prevent frequent gradient updates.
result More accurate and computationally efficient factorization.
Most popular word embedding techniques involve implicit or explicit factorization of a word co-occurrence based matrix into low rank factors. In this paper, we aim to generalize this trend by using numerical methods to factor higher-order word co-occurrence based arrays, or \textit{tensors}. We present four word embedd…
Spectral methods of moments provide a powerful tool for learning the parameters of latent variable models. Despite their theoretical appeal, the applicability of these methods to real data is still limited due to a lack of robustness to model misspecification. In this paper we present a hierarchical approach to methods…
New algorithm for online tensor factorization with provable guarantees.
problem Factorizing structured tensors with unknown factors and non-convex optimization.
method Online CP/PARAFAC decomposition via dictionary learning with incoherence and sparsity constraints.
result Exact recovery of tensor factors at a linear rate under mild conditions.
TATD predicts missing entries in time-evolving tensors by exploiting temporal dependency and sparsity.
problem Predict missing entries in time-evolving tensors with temporal dependency and sparsity issues.
method TATD (Time-Aware Tensor Decomposition) integrates temporal dependency and time-varying sparsity through a smoothing regularization with Gaussian kernel and alternating optimization.
result TATD achieves state-of-the-art accuracy for decomposing temporal tensors.
The popular Alternating Least Squares (ALS) algorithm for tensor decomposition is efficient and easy to implement, but often converges to poor local optima---particularly when the weights of the factors are non-uniform. We propose a modification of the ALS approach that is as efficient as standard ALS, but provably rec…
Matrix factorizations and their extensions to tensor factorizations and decompositions have become prominent techniques for linear and multilinear blind source separation (BSS), especially multiway Independent Component Analysis (ICA), NonnegativeMatrix and Tensor Factorization (NMF/NTF), Smooth Component Analysis (Smo…
Gradient descent in tensor factorization favors low-rank solutions.
problem Tackling implicit regularization in tensor factorization problems.
method Gradient descent with small random initialization for overparametrized tensor factorization.
result Gradient descent leads to implicit regularization towards low tubal rank solutions.
SPIDER uses deep neural networks for streaming tensor factorization.
problem Lack of effective approach for deep tensor factorization of streaming data.
method Bayesian neural networks with spike-and-slab prior, Taylor expansions, moment matching, and EPI framework.
result Effective incremental updates for latent factors and NN weights.
The paper analyzes implicit regularization in tensor factorization using neural networks.
problem Understanding implicit regularization in tensor factorization.
method Dynamical systems perspective and gradient descent analysis.
result Gradient descent induces a form of greedy low tensor rank search.
A new method uses CPD to efficiently model feature interactions in non-sequential data.
problem Efficiently modeling feature interactions in non-sequential data with high computational and memory costs.
method Implicitly represent model parameters as a tensor, factorize into a compact Tensor Train (TT) format, and use Canonical Polyadic (CP) Decomposition for invariance to feature ordering.
result The proposed CP-based predictor outperforms other TN-based predictors on sparse data and matches neural network performance on dense non-sequential tasks.
Paper proposes C-STM for multimodal neuroimaging data classification.
problem Multimodal neuroimaging data fusion for better classification.
method Coupled Support Tensor Machine (C-STM) using latent factors from ACMTF.
result C-STM achieves better classification performance than single-mode classifiers.
Paper introduces ZIPTF and C-ZIPTF for better tensor factorization of zero-inflated count data.
problem Inefficient tensor factorization for zero-inflated count data, especially in scRNA-seq.
method Zero Inflated Poisson Tensor Factorization (ZIPTF) and Consensus Zero Inflated Poisson Tensor Factorization (C-ZIPTF).
result ZIPTF and C-ZIPTF improve tensor factorization accuracy and consistency for zero-inflated count data.
In this paper, we exhibit the tradeoffs between the (training) sample, computation and storage complexity for the problem of supervised classification using signal subspace estimation. Our main tool is the use of tensor subspaces, i.e. subspaces with a Kronecker structure, for embedding the data into lower dimensions. …
Probabilistic Temporal Tensor Factorization (PTTF) is an effective algorithm to model the temporal tensor data. It leverages a time constraint to capture the evolving properties of tensor data. Nowadays the exploding dataset demands a large scale PTTF analysis, and a parallel solution is critical to accommodate the tre…
Tensor factorizations have become increasingly popular approaches for various learning tasks on structured data. In this work, we extend the RESCAL tensor factorization, which has shown state-of-the-art results for multi-relational learning, to account for the binary nature of adjacency tensors. We study the improvemen…
Non-negative matrix factorization models based on a hierarchical Gamma-Poisson structure capture user and item behavior effectively in extremely sparse data sets, making them the ideal choice for collaborative filtering applications. Hierarchical Poisson factorization (HPF) in particular has proved successful for scala…
NNEinFact fits any nonnegative tensor factorization quickly and accurately.
problem Limited user-friendly tools for fitting tailored nonnegative tensor factorizations.
method NNEinFact is an einsum-based multiplicative update algorithm that fits any nonnegative tensor factorization.
result NNEinFact converges to a stationary point of the loss, supports missing data, and fits tensors with hundreds of millions of entries in seconds.
Causal deep learning tackles causal inference using tensor factor analysis.
problem Addressing causal questions in data using neural networks.
method Tensor factor analysis and neural network architectures (causal capsules, tensor transformer, multilinear projection algorithm).
result Derives deep neural networks for causal inference with tensor factor analysis.
We propose a family of novel hierarchical Bayesian deep auto-encoder models capable of identifying disentangled factors of variability in data. While many recent attempts at factor disentanglement have focused on sophisticated learning objectives within the VAE framework, their choice of a standard normal as the latent…
Efficient NTF algorithm for large sparse tensors.
problem Sparse multi-dimensional data and limitations of existing NTF algorithms.
method Saturating Coordinate Descent with element selection based on Lipschitz continuity.
result Proposes a scalable NTF algorithm for large tensors.