arXiv research

This paper solves matrix blind joint block diagonalization with noise.

Paper proposes ABDR for convex subspace clustering with adaptive block diagonal representation.

New theory allows simultaneous block-diagonalization of commuting operator fields.

Randomized block-diagonal preconditioning improves parallel learning convergence.

We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…

Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive co…

Localized sketching improves matrix multiplication and ridge regression complexity.

Efficiently approximates Sparse PCA with significant speedups and minor error.

A Semi-Hidden Markov Model (SHMM) for bursty error channels is defined by a state transition probability matrix $A$, a prior probability vector $Π$, and the state dependent output symbol error probability matrix $B$. Several processes are utilized for estimating $A$, $Π$ and $B$ from a given empirically obtained or sim…

New insights into Hessian structure of neural networks reveal two forces.

Gaussian graphical models are widely utilized to infer and visualize networks of dependencies between continuous variables. However, inferring the graph is difficult when the sample size is small compared to the number of variables. To reduce the number of parameters to estimate in the model, we propose a non-asymptoti…

New model handles complex non-linear relationships with hidden graph structures.

This paper tackles model selection for MoE models in high-dimensional data.

New method for estimating financial covariance matrices efficiently.

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…

Subspace clustering is a useful technique for many computer vision applications in which the intrinsic dimension of high-dimensional data is often smaller than the ambient dimension. Spectral clustering, as one of the main approaches to subspace clustering, often takes on a sparse representation or a low-rank represent…

Develops efficient quasi-Newton methods for training deep neural networks.

A deep neural network is a hierarchical nonlinear model transforming input signals to output signals. Its input-output relation is considered to be stochastic, being described for a given input by a parameterized conditional probability distribution of outputs. The space of parameters consisting of weights and biases i…

Improved sparse Gaussian processes using structured scaling matrices and Power-EP framework.

DKLM learns adaptive kernels for robust nonlinear subspace clustering.

Ensembles of neural networks improve training dynamics and performance.

Shrunk sample covariance matrix is a factor model of a special form combining some (typically, style) risk factor(s) and principal components with a (block-)diagonal factor covariance matrix. As such, shrinkage, which essentially inherits out-of-sample instabilities of the sample covariance matrix, is not an alternativ…

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, …

In real-world applications, not all instances in multi-view data are fully represented. To deal with incomplete data, Incomplete Multi-view Learning (IML) rises. In this paper, we propose the Joint Embedding Learning and Low-Rank Approximation (JELLA) framework for IML. The JELLA framework approximates the incomplete d…

A quantum generalization of Natural Gradient Descent is presented as part of a general-purpose optimization framework for variational quantum circuits. The optimization dynamics is interpreted as moving in the steepest descent direction with respect to the Quantum Information Geometry, corresponding to the real part of…

It is shown that, in four dimensions, it is possible to introduce coordinates so that an analytic metric locally takes block diagonal form. i.e. one can find coordinates such that $g_{αβ} = 0$ for $(α, β) \in S$ where $S = {(1, 3), (1, 4), (2, 3), (2, 4)}$. We call a coordinate system in which the metric takes this for…

Characterizes Anosov reducible representations in terms of eigenvalues.

The paper identifies redundant columns in matrices for feature selection and clustering.

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for whic…

Proposes TFCL to mitigate negative transfer in MTL by collaborating across features and tasks.

The paper defines a Chern-Simons invariant for stably trivial vector bundles and uses it to obstruct conformal immersions.

We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the we…

We use a cluster ensemble to determine the number of clusters, k, in a group of data. A consensus similarity matrix is formed from the ensemble using multiple algorithms and several values for k. A random walk is induced on the graph defined by the consensus matrix and the eigenvalues of the associated transition proba…

The paper introduces a penalized matrix estimation procedure aiming at solutions which are sparse and low-rank at the same time. Such structures arise in the context of social networks or protein interactions where underlying graphs have adjacency matrices which are block-diagonal in the appropriate basis. We introduce…

We propose a new, infinite class of brackets generalizing the Frölicher--Nijenhuis bracket. This class can be reduced to a family of generalized Nijenhuis torsions recently introduced. In particular, the Haantjes bracket, the first example of our construction, is relevant in the characterization of Haantjes moduli of o…

We introduce the notion of Haantjes algebra: It consists of an assignment of a family of operator fields on a differentiable manifold, each of them with vanishing Haantjes torsion. They are also required to satisfy suitable compatibility conditions. Haantjes algebras naturally generalize several known interesting geome…

One of the longstanding open problems in spectral graph clustering (SGC) is the so-called model order selection problem: automated selection of the correct number of clusters. This is equivalent to the problem of finding the number of connected components or communities in an undirected graph. We propose automated mode…

We present a simple algorithm to forecast vector time series, that is robust against missing data, in both training and inference. It models seasonal annual, weekly, and daily baselines, and a Gaussian process for the seasonally-adjusted residuals. We develop a custom truncated eigendecomposition to fit a low-rank plus…

There are a multitude of methods to perform multi-set correlated component analysis (MCCA), including some that require iterative solutions. The methods differ on the criterion they optimize and the constraints placed on the solutions. This note focuses perhaps on the simplest version, which can be solved in a single s…

It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently …

Alexandrov spaces with non-negative curvature are characterized by the matrix displacement convexity of an entropy tensor.

Given a constant magnetic field on Euclidean space ${\mathbb R}^p$ determined by a skew-symmetric $(p\times p)$ matrix $Θ$, and a ${\mathbb Z}^p$-invariant probability measure $μ$ on the disorder set $Σ$ which is by hypothesis a Cantor set, where the action is assumed to be minimal, the corresponding Integrated Density…

A new iterative K-FAC algorithm reduces training time and memory usage.

Existing MAP inference algorithms for determinantal point processes (DPPs) need to calculate determinants or conduct eigenvalue decomposition generally at the scale of the full kernel, which presents a great challenge for real-world applications. In this paper, we introduce a class of DPPs, called BwDPPs, that are char…

Second-order optimization speeds up deep hedging for complex options.

ViViT efficiently computes curvature for deep networks without approximations.

We introduce a general framework for estimation of inverse covariance, or precision, matrices from heterogeneous populations. The proposed framework uses a Laplacian shrinkage penalty to encourage similarity among estimates from disparate, but related, subpopulations, while allowing for differences among matrices. We p…

Algorithm solves robust linear regression with block Lewis weights.