Modular method simplifies curvature computation in neural nets.
problem Efficient computation of curvature matrices for training neural nets.
method Modular backpropagation for block-diagonal approximations.
result Compact notation and easy integration into machine learning libraries.
Efficiently approximates Sparse PCA with significant speedups and minor error.
problem Sparse Principal Component Analysis (Sparse PCA) is NP-hard and computationally expensive.
method Approximates the covariance matrix with block-diagonal form, solves sub-problems in each block, and reconstructs the solution.
result Significant computational speedups with minor additive error.
Improved Hessian-free method for neural networks reduces computational cost.
problem High computational cost and model-dependent algorithmic variations in second-order methods.
method Block-diagonal approximation of the generalized Gauss-Newton matrix, conjugate gradient updates for each block.
result Better convergence and generalization compared to original Hessian-free and Adam methods.
Localized sketching improves matrix multiplication and ridge regression complexity.
problem Efficiently approximate matrix multiplication and ridge regression with limited data availability.
method Localized sketching matrices for block diagonal structure, reducing sample complexity.
result Localized sketching achieves sample complexity matching global sketching methods.
Framework for incomplete multi-view learning improves efficiency and clustering accuracy.
problem Incomplete representation in multi-view data.
method Joint Embedding Learning and Low-Rank Approximation (JELLA) framework.
result Improves efficiency and clustering accuracy in incomplete multi-view data.
This paper solves matrix blind joint block diagonalization with noise.
problem Identifying the diagonalizer and block diagonal structure of matrices under noise.
method Bi-block diagonalization method.
result The method can identify the exact solution under certain conditions.
Improved sparse Gaussian processes using structured scaling matrices and Power-EP framework.
problem Scaling Gaussian processes for large datasets.
method Structured diagonal scaling matrix and Power-EP framework.
result Structured approximations improve performance without increasing computational cost.
Develops efficient quasi-Newton methods for training deep neural networks.
problem Training deep neural networks with large-scale Hessian matrices.
method Approximates Hessian as block-diagonal Kronecker product of smaller matrices, applies damping.
result Outperforms or matches state-of-the-art methods in autoencoder models.
Paper proposes ABDR for convex subspace clustering with adaptive block diagonal representation.
problem Subspace clustering with block diagonal structure for noisy data.
method ABDR explicitly pursues block diagonality without sacrificing convexity, using a specially designed convex regularizer.
result Experimental results show ABDR outperforms state-of-the-arts.
New theory allows simultaneous block-diagonalization of commuting operator fields.
problem Normal forms of operator fields.
method Generalized Nijenhuis torsions and generalized Haantjes algebra.
result Simultaneous block-diagonalization of commuting operator fields.
Gaussian graphical models are widely utilized to infer and visualize networks of dependencies between continuous variables. However, inferring the graph is difficult when the sample size is small compared to the number of variables. To reduce the number of parameters to estimate in the model, we propose a non-asymptoti…
The paper simplifies the Fisher information matrix for random deep networks, speeding up learning.
problem Learning deep neural networks efficiently with large parameter spaces.
method Statistical neurodynamical method to reveal Fisher information properties, proving unit-wise block diagonal structure and explicit inverse.
result Explicit natural gradient formula without matrix inversion, speeding up learning.
Randomized block-diagonal preconditioning improves parallel learning convergence.
problem Improving convergence of gradient-based optimization methods in parallel settings.
method Randomization of coordinates during optimization to repartition tasks.
result Randomization significantly improves convergence of block-diagonal preconditioned methods.
Bayesian online learning method improves neural network performance.
problem Overcoming catastrophic forgetting in neural networks.
method Kronecker factored online Laplace approximation for Bayesian online learning.
result Achieves over 90% test accuracy across 50 MNIST tasks.
New adaptive methods improve deep learning performance.
problem Training deep networks efficiently and effectively.
method Block-diagonal matrix adaptation for gradient updates.
result Block-diagonal methods outperform adaptive diagonal methods and vanilla SGD.
New insights into Hessian structure of neural networks reveal two forces.
problem Understanding the Hessian structure of neural networks.
method Analyzing the static and dynamic forces, comparing limit distributions using random matrix theory.
result The Hessian structure arises from a combination of static and dynamic forces, with C being a primary driver. Quantum Natural Gradient uses quantum geometry for optimization.
problem Optimizing variational quantum circuits efficiently.
method Quantum generalization of Natural Gradient Descent using Quantum Information Geometry.
result Efficient algorithm for computing metric tensor approximations.
New model handles complex non-linear relationships with hidden graph structures.
problem Modeling non-linear relationships with hidden graph-structured interactions.
method Block-diagonal localized mixture of polynomial experts (BLoMPE) regression model with penalized maximum likelihood selection criterion.
result Strong theoretical guarantee for finite-sample oracle inequality.
Paper proposes a new Markov model for efficient PLC system design.
problem Efficient estimation of Markov model parameters for bursty error channels.
method Introduced a Block Diagonal Markov model and a modified Baum-Welch algorithm.
result Efficient estimation of state transition matrix Λ for PLC system design. This paper tackles model selection for MoE models in high-dimensional data.
problem Model selection for Gaussian-gated localized MoE and block-diagonal covariance localized MoE regression models in high-dimensional data.
method Penalized maximum likelihood estimation framework with non-asymptotic risk bounds.
result Established non-asymptotic risk bounds for model selection in MoE models.
New method for estimating financial covariance matrices efficiently.
problem Noisy covariance matrix estimation in high-dimensional financial data.
method Cluster financial time series into groups, apply shrinkage to ensure positive definiteness.
result Proposed methods provide reliable estimates and outperform other estimators.
ViViT efficiently computes curvature for deep networks without approximations.
problem Efficiently computing curvature for deep networks without approximations.
method Leverages the GGN's low-rank structure without further approximations.
result ViViT allows for efficient computation of eigenvalues, eigenvectors, and directional derivatives.
Efficient Gauss-Newton optimisation for neural networks.
problem Optimizing deep learning models efficiently and effectively.
method Block-diagonal approximation of Gauss-Newton matrix for neural networks.
result The resulting algorithm outperforms state-of-the-art first-order optimisation methods.
A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
New methods improve Fisher Matrix approximations for neural networks at low cost.
problem High cost of solving Fisher Information Matrix (FIM) in neural networks.
method Direct minimization via Kronecker product singular value decomposition.
result Improved approximations to FIM provide more accurate and faster optimization.
Efficient subspace clustering using Kronecker product reduces computational complexity.
problem Efficiency and scalability issues in traditional subspace clustering methods for large datasets.
method Proposes a subspace clustering model based on the Kronecker product to reduce computational complexity.
result Significantly improved efficiency compared to state-of-the-art methods on public datasets.
This study explains why approximate NGD works well in wide neural networks.
problem Understanding why NGD with approximate Fisher information converges fast in wide neural networks.
method Analyzing asymptotic training dynamics in function space via the neural tangent kernel.
result NGD with approximate Fisher information achieves the same fast convergence as exact NGD under specific conditions.
We analyze the Hessian spectra of large models up to 100B parameters.
problem Accurate Hessian spectra of large foundation models are difficult to obtain.
method We use shard-local finite-difference Hessian vector products and stochastic Lanczos quadrature.
result We produce the first large-scale spectral density estimates of foundation models.
DKLM learns adaptive kernels for robust nonlinear subspace clustering.
problem Nonlinear structures in data and challenges with kernel-based clustering.
method Data-driven kernel learning with adaptive weighting and optimal block-diagonal affinity matrix.
result DKLM enhances robustness and preserves manifold structure in nonlinear space.
New methods use Kronecker-factored approximations for faster deep learning optimization.
problem Optimizing deep learning models with rich curvature information.
method Approximate Hessian using Kronecker products for efficient quasi-Newton methods.
result New methods outperform first-order methods and perform comparably to second-order methods.
A new distributed optimization method for ERM problems.
problem Efficiently solving ERM problems with nonsmooth regularization in a distributed setting.
method Second-order distributed optimization using successive quadratic approximations and Hessian approximation.
result Global linear convergence for a broad range of non-strongly convex problems.
Ensembles of neural networks improve training dynamics and performance.
problem Improving neural network performance through model size increase.
method Defining collegial ensembles (CE) as multiple independent models trained as a single model, and using theoretical results on NTK to optimize architecture search.
result CE dynamics simplify and scale favorably, resembling wide models, and can be efficiently implemented using group convolutions and block diagonal layers.
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
Shrunk sample covariance matrix is a factor model of a special form combining some (typically, style) risk factor(s) and principal components with a (block-)diagonal factor covariance matrix. As such, shrinkage, which essentially inherits out-of-sample instabilities of the sample covariance matrix, is not an alternativ…
It is shown that, in four dimensions, it is possible to introduce coordinates so that an analytic metric locally takes block diagonal form. i.e. one can find coordinates such that gαβ=0 for (α,β)∈S where S=(1,3),(1,4),(2,3),(2,4). We call a coordinate system in which the metric takes this for…
New neural network scaling improves robustness.
problem Improving robustness of neural networks.
method Introduces a new rescaling technique for neural network training.
result Scales stepsizes to improve robustness.
BEGIN network models binary data without parametric assumptions.
problem Conditional independence in non-parametric families of binary data.
method BEGIN network models binary data using sparse linear representations and block factorizations.
result BEGIN network captures conditional independence for arbitrary binary and multinomial variables.
Characterizes Anosov reducible representations in terms of eigenvalues.
problem Understanding Anosov representations in reducible settings.
method Characterizes Anosov representations using eigenvalue magnitudes of irreducible block factors.
result Connected components of character varieties do not contain reducible representations for many non-elementary hyperbolic groups.
The paper identifies redundant columns in matrices for feature selection and clustering.
problem Identifying redundant columns in matrices for feature selection and clustering.
method Proves that after re-ordering columns, a matrix can be block-diagonalized revealing linearly dependent columns.
result Identifies redundant columns in matrices, aiding in feature selection and clustering.
We present a method based on the orthogonal symmetric non-negative matrix tri-factorization of the normalized Laplacian matrix for community detection in complex networks. While the exact factorization of a given order may not exist and is NP hard to compute, we obtain an approximate factorization by solving an optimiz…
Proposes TFCL to mitigate negative transfer in MTL by collaborating across features and tasks.
problem Negative transfer in Multi-Task Learning (MTL) due to dissimilar tasks.
method Task-Feature Collaborative Learning (TFCL) with heterogeneous block-diagonal structure regularizer and optimization method.
result Global convergence and block-diagonal structure recovery guarantees.
The paper defines a Chern-Simons invariant for stably trivial vector bundles and uses it to obstruct conformal immersions.
problem Obstructing conformal immersions of Riemannian manifolds.
method Defining a Chern-Simons invariant for stably trivial vector bundles and using it to derive an obstruction.
result An obstruction for conformally immersing a n-dimensional Riemannian manifold in a translation manifold of dimension n+1.
We use a cluster ensemble to determine the number of clusters, k, in a group of data. A consensus similarity matrix is formed from the ensemble using multiple algorithms and several values for k. A random walk is induced on the graph defined by the consensus matrix and the eigenvalues of the associated transition proba…
New algorithm finds clusters in graphs faster than existing methods.
problem Finding communities in graph data efficiently.
method Sparse solution of Laplacian linear system, thresholding, compressive sensing.
result Algorithm finds clusters in O(nlog(n)n_0) operations for a single cluster and fewer than O(n^2ln(n)) for all clusters.
The paper introduces a penalized matrix estimation procedure aiming at solutions which are sparse and low-rank at the same time. Such structures arise in the context of social networks or protein interactions where underlying graphs have adjacency matrices which are block-diagonal in the appropriate basis. We introduce…
Parallelizes DEC on curved meshes using group actions.
problem Efficiently solving DEC operators on curved and 3D meshes.
method Universal block-diagonalization framework for d and ⋆ operators, exploiting group actions. result Block-diagonal structure inherited by operators, enabling parallel solvers.
One of the longstanding open problems in spectral graph clustering (SGC) is the so-called model order selection problem: automated selection of the correct number of clusters. This is equivalent to the problem of finding the number of connected components or communities in an undirected graph. We propose automated mode…
New brackets generalize Haantjes moduli and ensure integrability of operators.
problem Characterizing and integrating operators using Haantjes moduli.
method Introducing a new infinite class of brackets and proving integrability conditions.
result Vanishing of higher-level Nijenhuis torsions ensures integrability and block-diagonal form.