The paper analyzes how low-rank layers in neural networks improve generalization.
problem Understanding how low-rank layers affect generalization in neural networks.
method Applying Maurer's chain rule for Gaussian complexity to analyze rank and spectral norm constraints.
result Deep networks with low-rank layers achieve better generalization than those with full-rank layers.
New insights into training deep networks with low rank layers.
problem Efficiency in training deep neural networks.
method Analysis of techniques for training in low rank space.
result Falsified common beliefs in training deep networks.
Proposes a model for identifying edges in low-rank dynamical networks.
problem Inability of conventional methods to handle low-rank dynamical networks.
method Low rank dynamical network model with causal Wiener filtering.
result Consistent method for estimating all network edges.
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
New algorithm improves deep learning models' robustness without sacrificing accuracy.
problem Low-rank methods compromise model robustness against adversarial perturbations.
method Robust low-rank training via approximate orthonormal constraints.
result Ensures well-conditioning and better adversarial robustness without sacrificing model accuracy.
LR-EDNN reduces PDE solver complexity by limiting network weights to low-rank subspace.
problem Efficiently solving time-dependent PDEs with deep neural networks.
method Low-rank constraint on network weights using SVD for efficient parameter updates.
result LR-EDNN achieves comparable accuracy to full EDNN with fewer parameters and lower cost.
New method for initializing low-rank neural networks improves performance.
problem Training low-rank neural networks efficiently and accurately.
method Inspired by function approximation, proposes a novel low-rank initialization framework.
result Demonstrates significant gap between spectral and low-rank initialization approaches.
This work examines how low-rank weights improve adversarial robustness in neural networks.
problem Improving adversarial robustness in neural networks.
method The study investigates the impact of low-rank structure on adversarial robustness through compression measures.
result Promoting low-rank structure in weight matrices enhances adversarial robustness in neural networks.
New method finds efficient low-rank neural networks during training.
problem High memory and computational demands of neural networks.
method Restricts weight matrices to a low-rank manifold and updates low-rank factors.
result Significantly reduced time and memory resources required for training and evaluation.
The paper proposes a method to learn Bayesian networks with low rank conditional probability tables.
problem Learning the structure of Bayesian networks efficiently.
method Introduces low rankness for conditional probability tables, connects to Fourier transformation, and proposes a polynomial time algorithm.
result Correctly recovers the true directed structure of a low rank Bayesian network with few queries and polynomial samples.
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
Tensor regression networks improve neural network compression and regularization.
problem Improving neural network compression and regularization with low-rank tensor approximations.
method Investigating various low-rank tensor approximations in tensor regression networks.
result Tensor regression networks with Global Average Pooling layer outperformed in deep CNNs, while shallow CNNs with tensor regression and dropout achieved lower test error.
Efficiently compress neural networks with MUSCO method.
problem Compression of deep neural networks.
method Iterative approach alternating low-rank factorization with rank selection and fine-tuning.
result Improves compression rate while maintaining accuracy.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
New method fits low-rank models for egocentrically sampled networks.
problem Statistical modeling of egocentrically sampled partial networks.
method Graph spectral properties-based approach for low-rank models.
result Consistent recovery of missing subnetworks for sparse networks.
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
Proposes a low-rank deep CNN for multi-task learning.
problem Multi-task learning with deep neural networks.
method Low-rank deep network with nuclear norm and sparsity penalties.
result Improves performance on multiple tasks compared to standard models.
Improved GCNs for non-sparse graphs with low-rank filters.
problem Training and evaluation of GCNs on large non-sparse graphs is computationally expensive.
method Introduced low-rank filters and a reduced-order GCN architecture.
result Significant runtime acceleration and improved accuracy achieved.
The paper proposes using low rank assumption to improve causal structure learning in DAGs.
problem Challenges in learning causal structures in high-dimensional, non-sparse DAGs.
method Exploits low rank assumption of DAG adjacency matrix to adapt causal structure learning methods.
result Maximum rank is highly related to hubs, suggesting low rank for scale-free networks.
Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
BoostNE learns multiple network embeddings from coarse to fine.
problem Complex node interactions cannot be fully captured by a single low-rank embedding matrix.
method BoostNE proposes a multi-level network embedding framework using gradient boosting.
result BoostNE outperforms existing network embedding methods on various datasets.
LOCUS separates brain network connectivity matrices efficiently.
problem High dimensionality, latent sources, and spurious findings in analyzing brain connectivity matrices.
method LOCUS: low-rank structure with uniform sparsity, iterative Node-Rotation algorithm.
result LOCUS achieves more efficient and accurate source separation for connectivity matrices.
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value. Proposes a low-rank PGD attack for more efficient adversarial training.
problem Efficiency and effectiveness of adversarial training methods.
method Variation of PGD to compute low-rank perturbations.
result Low-rank PGD attacks are more efficient and comparable to traditional attacks.
Decomposable-Net compresses neural networks without retraining for various sizes.
problem Performance degradation when changing model size after training.
method Decomposes weight matrices via SVD and adjusts ranks for different sizes.
result Maintains and improves performance across multiple model sizes.
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
Low-rank training improves neural network training on edge devices with non-volatile memory.
problem Training neural networks on edge devices with non-volatile memory, especially in terms of write density and auxiliary memory.
method Low-rank training scheme to address write density and auxiliary memory limitations.
result The low-rank training technique outperforms standard SGD in accuracy and weight writes.
Paper proposes a new technique to compress CNNs while maintaining accuracy.
problem CNNs struggle with traditional low-rank approximation methods, leading to degraded accuracy.
method Introduces a training technique that finds a flat minimum in low-rank approximation without a decomposed structure.
result CNN models can be compressed with higher accuracy and lower computation than conventional methods.
We consider the problem of modeling multivariate time series with parsimonious dynamical models which can be represented as sparse dynamic Bayesian networks with few latent nodes. This structure translates into a sparse plus low rank model. In this paper, we propose a Gaussian regression approach to identify such a mod…
Proposes a model to clean and analyze network data.
problem Noisy and missing connections in network data.
method Generalized linear model with low rank effects.
result Efficient algorithm for fitting the model and demonstrating its effectiveness.
The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.
problem Investigating low-rank structure in gradients of neural networks under relaxed assumptions.
method Spiked data model, relaxation of isotropy assumptions, analysis of mean-field and neural-tangent-kernel scalings.
result Gradient of input weights is approximately low rank, dominated by two rank-one terms.
Paper proposes low-rank gradient approximation to save memory for deep neural network training.
problem Memory limitation on mobile devices for deep neural network training.
method Approximating gradient matrices using low-rank parameterization.
result Reduces training memory by about 33.0% for Adam optimization and 4.5% relative lower word error rate on ASR personalization task.
Novel beamforming method reduces errors in wireless networks.
problem Mitigating channel errors in wireless networks with relays.
method Low-rank and cross-correlation techniques for robust distributed beamforming.
result The proposed LRCC-RDB technique significantly improves SINR performance.
Given the superposition of a low-rank matrix plus the product of a known fat compression matrix times a sparse matrix, the goal of this paper is to establish deterministic conditions under which exact recovery of the low-rank and sparse components becomes possible. This fundamental identifiability issue arises with tra…
CoLoRA models predict PDE solutions quickly and accurately with minimal data.
problem Efficiently modeling PDE solutions with limited data.
method Continuous low-rank adaptation of neural networks trained on offline data.
result Predictions are orders of magnitude faster and more accurate than classical methods.
Tensorial Neural Networks improve neural network compression and performance.
problem Efficiently compressing neural networks while maintaining or improving performance.
method Introducing tensor operations on high-order operands to solve hierarchical nonlinear tensor decomposition using stochastic gradient descent.
result TNNs achieve up to 5% test accuracy improvement on CIFAR10 compared to state-of-the-art compression methods.
We propose a method to infer stochastic low-rank RNNs from neural data.
problem Fitting low-rank RNNs to noisy, stochastic neural data.
method Variational sequential Monte Carlo methods for stochastic low-rank RNNs.
result Lower dimensional latent dynamics compared to state-of-the-art methods.
This work improves robustness guarantees for neural networks using low rank representations.
problem Certified robustness to adversarial perturbations in neural networks.
method Low rank representations to provide improved robustness guarantees.
result Improved robustness guarantees for ℓ∞ perturbations using natural low rank representations. SVD training reduces DNN rank and computation load without SVD per step.
problem High memory and computational load in deep neural networks.
method Explicitly achieves low-rank DNNs during training without SVD per step, using orthogonality regularization and sparsity-inducing regularizers.
result Significantly reduces DNN rank and computation load compared to existing methods.
Distributed algorithm finds global solutions for low-rank matrices.
problem Finding global solutions for low-rank matrices in distributed systems.
method Distributed Gradient Descent (DGD+) with LOCAL variables.
result DGD+LOCAL converges to global minimizer with exact consensus.
LoRA enhances model adaptability without increasing parameters.
problem Theoretical understanding of LoRA's adaptability.
method Theoretical analysis of LoRA's expressive power.
result LoRA can adapt any model to a smaller target model with a specific rank.
Flat minima lead to better generalization in low-rank matrix recovery models.
problem Understanding why flat minima generalize well in overparameterized models.
method Analysis of overparameterized matrix and bilinear sensing, robust PCA, covariance matrix estimation, and neural networks with quadratic activation functions.
result Flat minima, measured by the trace of the Hessian, exactly recover the ground truth in low-rank matrix recovery models under standard statistical assumptions.
Flora uses random projections to achieve high-rank updates with low memory usage.
problem Excessive memory usage in large neural networks during training.
method Flora approximates LoRA using random projections to enable high-rank updates with sublinear space complexity.
result Flora achieves high-rank updates with significantly reduced memory usage compared to LoRA.
The paper explains neural network generalization via low-rank Jacobian structure.
problem Understanding why neural networks generalize well despite having more parameters than training data.
method Develops a theory leveraging the low-rank structure of the Jacobian matrix.
result Neural networks can generalize well due to a low-dimensional information space in the Jacobian.
Proposes a new model for image restoration combining deep learning and total variation.
problem Restoring images from limited data with low-rank constraints insufficient.
method Regularized Deep Matrix Factorized (RDMF) model using deep neural network's low-rank bias and total variation.
result Outperforms state-of-the-art models in image restoration from few observations.
DeepTensor uses deep networks to efficiently decompose tensors with improved performance and robustness.
problem Efficiently decomposing tensors with deep learning to capture nonlinear structures.
method Low-rank tensor decomposition using deep generative networks trained to minimize approximation error.
result DeepTensor outperforms classical methods like SVD and PCA in various applications, including image denoising and 3D MRI.
A new method matches point sets of low-rank networks via their Laplace transforms.
problem Matching nodes in unseeded, low-rank networks without known correspondences.
method Transform-based unsupervised point registration via minimizing discrepancy between Laplace transforms.
result First consistency guarantee and explicit error rate for general low-rank models.
Dropout improves neural network performance by promoting low-rank solutions.
problem Improving neural network generalization through regularization.
method Analyzing Dropout, DropBlock, and DropConnect as regularizers for linear networks and extending to deep networks.
result Dropout, DropBlock, and DropConnect induce low-rank solutions and can be computed in closed form.