arXiv research

Paper proposes ABDR for convex subspace clustering with adaptive block diagonal representation.

Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive co…

This paper solves matrix blind joint block diagonalization with noise.

Randomized block-diagonal preconditioning improves parallel learning convergence.

New theory allows simultaneous block-diagonalization of commuting operator fields.

A fast metric learning framework using Gershgorin disc alignment.

DKLM learns adaptive kernels for robust nonlinear subspace clustering.

We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…

New insights into Hessian structure of neural networks reveal two forces.

The high-order relations between the content in social media sharing platforms are frequently modeled by a hypergraph. Either hypergraph Laplacian matrix or the adjacency matrix is a big matrix. Randomized algorithms are used for low-rank factorizations in order to approximately decompose and eventually invert such big…

Subspace clustering is a useful technique for many computer vision applications in which the intrinsic dimension of high-dimensional data is often smaller than the ambient dimension. Spectral clustering, as one of the main approaches to subspace clustering, often takes on a sparse representation or a low-rank represent…

Efficiently approximates Sparse PCA with significant speedups and minor error.

Localized sketching improves matrix multiplication and ridge regression complexity.

Develops a novel stochastic algorithm for diagonal estimation of large matrices.

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and th…

Gaussian graphical models are widely utilized to infer and visualize networks of dependencies between continuous variables. However, inferring the graph is difficult when the sample size is small compared to the number of variables. To reduce the number of parameters to estimate in the model, we propose a non-asymptoti…

A Semi-Hidden Markov Model (SHMM) for bursty error channels is defined by a state transition probability matrix $A$, a prior probability vector $Π$, and the state dependent output symbol error probability matrix $B$. Several processes are utilized for estimating $A$, $Π$ and $B$ from a given empirically obtained or sim…

Improved sparse Gaussian processes using structured scaling matrices and Power-EP framework.

New method for estimating financial covariance matrices efficiently.

Homogeneous links were introduced by Peter Cromwell, who proved that the projection surface of these links, that given by the Seifert algorithm, has minimal genus. Here we provide a different proof, with a geometric rather than combinatorial flavor. To do this, we first show a direct relation between the Seifert matrix…

A deep neural network is a hierarchical nonlinear model transforming input signals to output signals. Its input-output relation is considered to be stochastic, being described for a given input by a parameterized conditional probability distribution of outputs. The space of parameters consisting of weights and biases i…

ADAHESSIAN optimizes machine learning models with adaptive second-order methods.

New model handles complex non-linear relationships with hidden graph structures.

In real-world applications, not all instances in multi-view data are fully represented. To deal with incomplete data, Incomplete Multi-view Learning (IML) rises. In this paper, we propose the Joint Embedding Learning and Low-Rank Approximation (JELLA) framework for IML. The JELLA framework approximates the incomplete d…

This paper tackles model selection for MoE models in high-dimensional data.

It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently …

Paper proposes efficient methods for clustering and signal recovery in high-dimensional data with block structures.

We introduce the notion of Haantjes algebra: It consists of an assignment of a family of operator fields on a differentiable manifold, each of them with vanishing Haantjes torsion. They are also required to satisfy suitable compatibility conditions. Haantjes algebras naturally generalize several known interesting geome…

Develops efficient quasi-Newton methods for training deep neural networks.

This paper improves neural network generalization by dynamically learning kernel parameters.

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, …

Sharp pseudospectral bounds prevent transient amplification in coupled gradient descent.

New MCMC method learns sparse preconditioner for high-dimensional problems.

Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years. While first-order methods seem to dominate the field, second-order methods are nevertheless attractive as they potentially require fewer communication rounds to converge. However, th…

It is shown that, in four dimensions, it is possible to introduce coordinates so that an analytic metric locally takes block diagonal form. i.e. one can find coordinates such that $g_{αβ} = 0$ for $(α, β) \in S$ where $S = {(1, 3), (1, 4), (2, 3), (2, 4)}$. We call a coordinate system in which the metric takes this for…

New estimators reduce computation for Kendall's tau and conditional Kendall's tau matrices under structural assumptions.

Characterizes Anosov reducible representations in terms of eigenvalues.

Second-order optimization speeds up deep hedging for complex options.

A new metric learning framework for signed graphs using Gershgorin disc alignment.

Three methods for tuning HMC diagonal scale matrices compared.

The paper identifies redundant columns in matrices for feature selection and clustering.

Paper develops methods for solving complex stochastic equations using Malliavin calculus.

Develops large-sample theory for non-stationary source separation.

Advanced optimization algorithms such as Newton method and AdaGrad benefit from second order derivative or second order statistics to achieve better descent directions and faster convergence rates. At their heart, such algorithms need to compute the inverse or inverse square root of a matrix whose size is quadratic of …

Method estimates M-matrices in graphical models with improved accuracy.

Algorithm solves robust linear regression with block Lewis weights.

Adaptive stochastic gradient methods such as AdaGrad have gained popularity in particular for training deep neural networks. The most commonly used and studied variant maintains a diagonal matrix approximation to second order information by accumulating past gradients which are used to tune the step size adaptively. In…

The adaptive gradient online learning method known as AdaGrad has seen widespread use in the machine learning community in stochastic and adversarial online learning problems and more recently in deep learning methods. The method's full-matrix incarnation offers much better theoretical guarantees and potentially better…