BLAST optimizes deep model inference by learning efficient matrix structures.
problem Efficient inference of large-scale deep neural networks.
method Introduces BLAST matrix for adaptive structured weights.
result Significant compression and performance improvements for various model sizes.
Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computation…
We discuss a weighted estimation of correlation and covariance matrices from historical financial data. To this end, we introduce a weighting scheme that accounts for similarity of previous market conditions to the present one. The resulting estimators are less biased and show lower variance than either unweighted or e…
Paper studies deep diagonal circulant neural networks and introduces training techniques.
problem Understanding and training deep neural networks with structured weight matrices.
method Theoretical analysis and practical training techniques including initialization and non-linearity use.
result Deep diagonal circulant networks outperform other structured models in accuracy and weight efficiency.
The low displacement rank (LDR) framework for structured matrices represents a matrix through two displacement operators and a low-rank residual. Existing use of LDR matrices in deep learning has applied fixed displacement operators encoding forms of shift invariance akin to convolutions. We introduce a class of LDR ma…
We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent network…
New method clusters hypergraphs using weighted random walks and Laplacians.
problem Clustering hypergraph data with edge-dependent weights.
method Random walks with edge-dependent vertex weights, constructing hypergraph Laplacians for clustering.
result Proposed methods outperform existing hypergraph clustering algorithms.
The paper proves a distribution claim for neural network Jacobians.
problem Distribution of singular values in deep neural networks.
method Free probability and random matrix theory techniques.
result Singular value distribution matches for specific cases.
Spectral portfolio theory links neural networks to wealth dynamics via SGD weight matrices.
problem Understanding wealth dynamics from neural network training.
method Direct identification of weight matrices as portfolio allocation matrices, linking SGD forces to portfolio dynamics.
result Spectral properties of SGD weight matrices transition between additive and multiplicative regimes, influencing wealth dynamics.
Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
New bound for neural networks with full-rank weights, independent of network width.
problem Understanding generalization of neural networks with full-rank weight matrices.
method Using Koopman operators to derive a tighter generalization bound for full-rank weight matrices.
result The bound is tighter than existing norm-based bounds when condition numbers are small.
Study of discrete period matrices on embedded graphs, relating to Riemann surfaces.
problem Understanding discrete conformal structures on surfaces via period matrices.
method Combinatorial interpretation of period matrices, using homological quasi-trees and Laplacian determinants.
result Derived a combinatorial analogue of the Weil-Petersson potential and related it to homological quasi-trees.
This work examines how low-rank weights improve adversarial robustness in neural networks.
problem Improving adversarial robustness in neural networks.
method The study investigates the impact of low-rank structure on adversarial robustness through compression measures.
result Promoting low-rank structure in weight matrices enhances adversarial robustness in neural networks.
SGD and weight decay encourage neural networks to learn low-rank weight matrices.
problem The bias of SGD towards low-rank weight matrices in neural networks.
method The study investigates the effect of SGD and weight decay on the rank of weight matrices in neural networks, both theoretically and empirically.
result Training with SGD and weight decay induces a bias towards rank minimization in weight matrices, which becomes more pronounced with smaller batch sizes and stronger weight decay.
Bayesian method infers transition matrices from incomplete graph data with topological constraints.
problem Inference of transition matrices from incomplete graph data with topological constraints.
method Bayesian approach using repeated interactions and a topological prior.
result Higher accuracy in inferring transition probabilities, improving downstream tasks.
Study evaluates thresholds for removing noise from DNN weights using random matrix theory.
problem Removing noise from deep neural network weights for better approximation.
method Model weights as signal + noise, use random matrix theory to estimate thresholds, evaluate using cosine similarity.
result Proposed threshold estimation method improves approximation quality.
This work compresses heavy-tailed weight matrices for tighter generalization bounds.
problem Empirical evidence linking heavy-tailed weight matrices to test set accuracy but lack of formal relationship with generalization bounds.
method Utilized the compression framework to show that heavy-tailed matrices can be compressed, resulting in sparse weight matrices.
result Demonstrated a non-vacuous generalization bound for compressed networks with heavy-tailed weight matrices.
New bounds for CNNs show better generalization than previous models.
problem Improving understanding of CNNs' generalization ability.
method Proposed tighter generalization bounds for CNNs by exploiting the sparse and permutation structure of weight matrices and spectral norms of convolution operations.
result Theoretical and experimental results show tighter bounds for CNNs than existing bounds.
New framework improves GAN training by controlling weight spectra.
problem Training GANs is challenging due to instability and poor generalization.
method Proposes a new reparameterization approach for the discriminator's weight matrices to control spectra.
result Spectrum control enhances GANs' generalization ability and image quality.
Efficiently learns DAG structures without cycles.
problem Learning DAG structures efficiently with constraints.
method Proposes DAG-NoCurl, a two-step algorithm to find and project acyclic graphs.
result Significantly improves efficiency over existing methods, often by more than one order of magnitude.
WE constructs GP kernels for mixed inputs using weighted EDMs.
problem Limitation of standard GP models in handling categorical variables.
method WEGP constructs kernel function using weighted EDMs for categorical inputs.
result WEGP improves GP model accuracy in both synthetic and real-world optimization problems.
Heavy-tailed regularization improves deep neural network performance.
problem Improving generalization of deep neural networks.
method Introducing Heavy-Tailed Regularization, using differentiable penalty terms and Bayesian statistics.
result Heavy-tailed regularization outperforms conventional regularization techniques.
New method clusters weighted directed networks using motifs.
problem Clustering directed networks fails to consider higher-order structure and edge weights.
method Motif-based weighted spectral clustering with new matrix formulae.
result Scalable and effective clustering on large graphs and real-world data.
The difficulty of classification affects the weight matrices' heavy tail appearance in deep learning networks.
problem Understanding the spectral properties of weight matrices in deep learning networks.
method Spectral analysis of weight matrices in different modules of DNNs, classification difficulty as a driving factor for heavy tail appearance.
result Higher classification difficulty leads to more frequent appearance of heavy tails in weight matrices spectra.
Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To cir…
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
Gradient descent aligns neural feature matrices with pre-activation tangent features.
problem Understanding neural feature learning mechanisms.
method Analytical proof of alignment between weight matrices and pre-activation tangent features.
result Derivative alignment occurs almost surely in high-dimensional settings.
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
We introduce a covariance matrix estimator that both takes into account the heteroskedasticity of financial returns (by using an exponentially weighted moving average) and reduces the effective dimensionality of the estimation (and hence measurement noise) via techniques borrowed from random matrix theory. We calculate…
A theory of feature geometry using spectral analysis of weight matrices.
problem Current methods decompose neural network activations into sparse linear features, losing geometric structure.
method Develops a theory by analyzing the spectra of weight-derived matrices, introducing the frame operator.
result Features collapse onto single eigenspaces, organizing into tight frames, and admit discrete classification.
Recurrent Neural Networks (RNNs) are becoming increasingly important for time series-related applications which require efficient and real-time implementations. The recent pruning based work ESE suffers from degradation of performance/energy efficiency due to the irregular network structure after pruning. We propose bl…
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
Study uses graph Laplacians to analyze surface links.
problem Analyzing virtual genus of surface links.
method Laplacian matrices of weighted graphs in surfaces are used to define invariants.
result Obtained information about virtual genus.
One reflection suffices for orthogonal weights, reducing GPU usage.
problem Efficiently computing orthogonal weight matrices without high GPU utilization.
method Use an auxiliary neural network to compute one reflection instead of many.
result One reflection is sufficient for orthogonal weights, improving GPU utilization.
Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to repre…
We show that the adjacency matrices of the intersection graphs of chord diagrams satisfy the 2-term relations of Bar-Natan and Garoufalides [bg], and hence give rise to weight systems. Among these weight systems are those associated with the Conway and HOMFLYPT polynomials. We extend these ideas to looking at a space o…
New ICA method improves on existing techniques.
problem Finding independent components in data.
method Multiple-weighted Independent Component Analysis (MWeICA) based on approximate diagonalization of weighted covariance matrices.
result MWeICA achieves better results than state-of-the-art ICA methods with similar computational time.
The paper studies matrix normalization and graph balancing using a new functional and gradient descent.
problem Matrix normalization and graph balancing.
method A new functional called the non-normal energy, and gradient descent.
result Gradient descent of the non-normal energy converges to balanced graphs and preserves spectra and realness of weights.
Study of metrics on positive-definite matrices from power potential, linking to power means.
problem Understanding metrics on positive-definite matrices derived from power potential.
method Explicit expressions for geodesics and distance function derived from Hessian of power potential.
result Geodesics and distance function converge to weighted matrix geometric mean as β tends to zero.
Study heavy-tailed weights' impact on neural network's spectral distribution.
problem Analyzing spectral distribution of conjugate kernel matrices with heavy-tailed weights.
method Computed limiting eigenvalue distribution through moments, considering heavy-tailed distributions and nonlinear activation functions.
result Heavy-tailed weights induce strong correlations, leading to fundamentally different spectral behavior.
E2M predicts metric space outputs using deep learning.
problem Predicting non-Euclidean outputs like distributions and matrices.
method Weighted Fréchet means over learned weights.
result E2M achieves state-of-the-art performance across various outputs.
New method estimates spatial weights matrix for lattice data, improving prediction accuracy.
problem Estimating spatial dependence structure for regular lattice data.
method Adaptive lasso with cross-sectional resampling to estimate sparse spatial weights matrix.
result Improves prediction accuracy of nitrogen dioxide concentrations.
Study reveals how attention helps in signal recovery from sequence models using random matrix theory.
problem Signal recovery from sequence models with attention mechanisms.
method Analysis of sample covariance matrices constructed from pooled sequence representations with attention weights.
result Optimal attention weights maximize signal-to-noise ratio and improve signal recovery.
Holonomy-preserving transformations help recover Alexander polynomials from graph zeta functions.
problem Recovering Alexander polynomials from graph zeta functions.
method Introducing holonomy to preserve zeta functions of matrix-weighted graphs and extending to group elements and quandles.
result Holonomy-preserving transformations correspond to transformations of group presentations and preserve the twisted Alexander polynomial.
Low-rank structure emerges in neural networks during learning.
problem Understanding the evolution of synaptic connectivity over learning.
method Investigated the rank of 3-tensor formed by weight matrices throughout learning.
result Inferred weights are low-tensor-rank and evolve in a fixed low-dimensional subspace.
Study on random matrices in deep neural networks with IID entries.
problem Distribution of singular values in product of random matrices for deep neural networks.
method Random matrix theory with a streamlined approach for non-Gaussian data.
result Generalization of macroscopic universality property to non-Gaussian data.
New method corrects quantization errors in LLMs using low-rank matrices.
problem Correcting quantization errors in large language models.
method Introducing low-rank weight matrices to correct quantized activations in LLMs.
result Reduces accuracy gap with original model by more than 50% using low-rank matrices.
New method clusters matrix-valued data by latent variables.
problem Clustering matrix-valued data with hidden structure.
method Latent variable model with hierarchical clustering.
result Algorithm attains clustering consistency in high dimensions.