On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Low-rank weights”

Reweighted Solutions for Weighted Low Rank Approximationcs.DS

New algorithm for weighted low rank approximation with provable guarantees.

problem Weighted low rank approximation (WLRA) is computationally hard.

method Reweights the low rank solution using the weight matrix itself.

result Provably optimal approximation guarantees for WLRA.

Low rank approximation Weighted low rank approximation Approximation guarantees Weight matrix Relaxed solutions

2024-06-04abs ↗pdf ↗

Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscs.LG

Weight Decay induces low-rank weight matrices in neural networks, improving generalization.

problem Improving generalization in neural networks.

method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.

result The weight matrix of a trained NN is approximately rank-two.

Weight Decay Neural Networks Low-rank Generalization Stochastic Gradient Descent

2024-10-03abs ↗pdf ↗

Weighted Low Rank Matrix Approximation and Accelerationstat.ML

Paper develops a new weighted low-rank matrix approximation technique.

problem Matrix completion with missing data.

method Element-wise weighted generalization of low-rank matrix approximation.

result Proposes an algorithm and acceleration techniques for solving the weighted problem.

matrix approximation matrix completion low-rank recommender systems GLM optimization

2021-09-22abs ↗pdf ↗

Low-Rank Correction for Quantized LLMsstat.ML

New method corrects quantization errors in LLMs using low-rank matrices.

problem Correcting quantization errors in large language models.

method Introducing low-rank weight matrices to correct quantized activations in LLMs.

result Reduces accuracy gap with original model by more than 50% using low-rank matrices.

model compression quantization errors large language models LLMs low-rank correction

2024-12-10abs ↗pdf ↗

Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscs.LG

LoRAs enable efficient adaptation of large models; this paper explores processing LoRA weights with machine learning.

problem Efficient processing of low-rank weight decompositions in large finetuned models.

method Developed symmetry-aware invariant and equivariant LoL models to process LoRA weights.

result LoL models can predict CLIP scores, finetuning data attributes, and accuracy on downstream tasks.

LoRAs Low-rank weights Machine learning Invariant models Equivariant models

2024-10-05abs ↗pdf ↗

Robust low-rank training via approximate orthonormal constraintscs.LG

New algorithm improves deep learning models' robustness without sacrificing accuracy.

problem Low-rank methods compromise model robustness against adversarial perturbations.

method Robust low-rank training via approximate orthonormal constraints.

result Ensures well-conditioning and better adversarial robustness without sacrificing model accuracy.

deep learning robustness low-rank orthonormal constraints adversarial perturbations

2023-06-02abs ↗pdf ↗

Provable Emergence of Deep Neural Collapse and Low-Rank Bias in

L^2

-Regularized Nonlinear Networkscs.LG

The paper connects neural collapse and low-rank bias in networks with L2 regularization.

problem Understanding the emergence of low-rank bias and neural collapse in L2-regularized networks.

method Unified theoretical framework linking TCV and rank of weight matrices, proving global optimality of DNC1, and establishing a benign landscape property.

result Zero TCV across intermediate layers minimizes representation cost under natural architectural constraints, and DNC1 is globally optimal.

Deep Neural Collapse Low-Rank Bias L2 Regularization Total Cluster Variation Weight Decay

2024-02-06abs ↗pdf ↗

Weighted Low-rank Approximation via Stochastic Gradient Descent on Manifoldsmath.OC

Stochastic gradient descent on manifolds improves low-rank approximation.

problem Efficiently approximate large matrices with lower rank.

method Stochastic gradient descent on a manifold.

result Algorithm outperforms Euclidean space methods on Netflix Prize data.

low-rank approximation manifold optimization stochastic gradient descent Netflix Prize

2025-02-20abs ↗pdf ↗

Low-Rank Adaptation of Evolutionary Deep Neural Networks for Efficient Learning of Time-Dependent PDEsstat.ML

LR-EDNN reduces PDE solver complexity by limiting network weights to low-rank subspace.

problem Efficiently solving time-dependent PDEs with deep neural networks.

method Low-rank constraint on network weights using SVD for efficient parameter updates.

result LR-EDNN achieves comparable accuracy to full EDNN with fewer parameters and lower cost.

PDEs Deep Learning Evolutionary Neural Networks Low-Rank Partial Differential Equations

2025-09-19abs ↗pdf ↗

Low Rank Gradients and Where to Find Themcs.LG

The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.

problem Investigating low-rank structure in gradients of neural networks under relaxed assumptions.

method Spiked data model, relaxation of isotropy assumptions, analysis of mean-field and neural-tangent-kernel scalings.

result Gradient of input weights is approximately low rank, dominated by two rank-one terms.

neural networks gradient structure low-rank spiked data neural tangent kernel

2025-10-01abs ↗pdf ↗

Recovery guarantee of weighted low-rank approximation via alternating minimizationcs.LG

Many applications require recovering a ground truth low-rank matrix from noisy observations of the entries, which in practice is typically formulated as a weighted low-rank approximation problem and solved by non-convex optimization heuristics such as alternating minimization. In this paper, we provide provable recover…

2016-02-06abs ↗pdf ↗

Gradient dynamics for low-rank fine-tuning beyond kernelscs.LG

LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.

problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.

method Generalized student-teacher setting with i.i.d. samples and online gradient descent.

result Gradient descent converges to the teacher model in

dk^{O(1)}

iterations under certain conditions.

LoRA gradient dynamics low-rank perturbations fine-tuning student-teacher

2024-11-23abs ↗pdf ↗

On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networkscs.LG

Recently, there has been an abundance of works on designing Deep Neural Networks (DNNs) that are robust to adversarial examples. In particular, a central question is which features of DNNs influence adversarial robustness and, therefore, can be to used to design robust DNNs. In this work, this problem is studied throug…

2019-01-29abs ↗pdf ↗

Matrix Approximation under Local Low-Rank Assumptioncs.LG

Matrix approximation is a common tool in machine learning for building accurate prediction models for recommendation systems, text mining, and computer vision. A prevalent assumption in constructing matrix approximations is that the partially observed matrix is of low-rank. We propose a new matrix approximation model w…

2013-01-15abs ↗pdf ↗

Mitigating Forgetting in Low Rank Adaptationcs.LG

LaLoRA prevents forgetting in LoRA fine-tuning.

problem Catastrophic forgetting in fine-tuned models.

method LaLoRA applies Laplace approximation to LoRA weights for regularization.

result Improved learning-forgetting trade-off with controllable regularization strength.

catastrophic forgetting parameter-efficient fine-tuning Low-Rank Adaptation Laplace approximation weight-space regularization

2025-12-19abs ↗pdf ↗

Robust Low-Rank Tensor Completion based on M-product with Weighted Correlated Total Variation and Sparse Regularizationstat.ML

New method improves tensor completion by selectively preserving important elements.

problem Recovering corrupted high-dimensional tensor data with missing entries and noise.

method Tensor weighted correlated total variation (TWCTV) regularizer with ADMM algorithm.

result Superior performance in image completion, denoising, and background subtraction tasks.

tensor completion low-rank tensor M-product total variation sparse regularization

2026-04-15abs ↗pdf ↗

Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theorystat.ML

Study evaluates thresholds for removing noise from DNN weights using random matrix theory.

problem Removing noise from deep neural network weights for better approximation.

method Model weights as signal + noise, use random matrix theory to estimate thresholds, evaluate using cosine similarity.

result Proposed threshold estimation method improves approximation quality.

deep learning random matrix theory cosine similarity singular value decomposition

2025-12-15abs ↗pdf ↗

Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelsstat.ML

TFB simplifies Bayesian LLM uncertainty estimation without extra training.

problem Estimating uncertainty in LLM responses remains challenging.

method Training-Free Bayesianization (TFB) that transforms low-rank adapters into Bayesian ones without additional training.

result TFB achieves superior uncertainty estimation and generalization compared to existing methods.

Bayesian inference Large Language Models Low-rank adapters Uncertainty quantification Training-free

2024-12-07abs ↗pdf ↗

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processesstat.ML

This paper extends the Gaussian process interpretation of deep networks to more varied weight distributions.

problem Understanding the impact of different weight initialization schemes on deep learning dynamics.

method Extending the Gaussian process interpretation to PSEUDO-IID weight distributions, including sparse and low-rank networks.

result PSEUDO-IID initialized networks are effectively equivalent up to variance, enabling tractable posterior distributions.

Gaussian Processes Deep Learning Weight Initialization Sparse Networks Low-Rank Networks

2023-10-25abs ↗pdf ↗

SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcs.LG

SGD and weight decay encourage neural networks to learn low-rank weight matrices.

problem The bias of SGD towards low-rank weight matrices in neural networks.

method The study investigates the effect of SGD and weight decay on the rank of weight matrices in neural networks, both theoretically and empirically.

result Training with SGD and weight decay induces a bias towards rank minimization in weight matrices, which becomes more pronounced with smaller batch sizes and stronger weight decay.

SGD weight decay neural networks rank minimization mini-batch

2022-06-12abs ↗pdf ↗

Generalized Nonconvex Nonsmooth Low-Rank Minimizationcs.CV

As surrogate functions of $L_0$ -norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…

2014-04-29abs ↗pdf ↗

Nonlinear Initialization Methods for Low-Rank Neural Networkscs.LG

New method for initializing low-rank neural networks improves performance.

problem Training low-rank neural networks efficiently and accurately.

method Inspired by function approximation, proposes a novel low-rank initialization framework.

result Demonstrates significant gap between spectral and low-rank initialization approaches.

neural networks low-rank ReLU networks spectral initialization function approximation

2022-02-02abs ↗pdf ↗

CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equationscs.LG

CoLoRA models predict PDE solutions quickly and accurately with minimal data.

problem Efficiently modeling PDE solutions with limited data.

method Continuous low-rank adaptation of neural networks trained on offline data.

result Predictions are orders of magnitude faster and more accurate than classical methods.

partial differential equations neural networks parameterized models low-rank approximation data-driven

2024-02-22abs ↗pdf ↗

Koopman-based generalization bound: New aspect for full-rank weightscs.LG

New bound for neural networks with full-rank weights, independent of network width.

problem Understanding generalization of neural networks with full-rank weight matrices.

method Using Koopman operators to derive a tighter generalization bound for full-rank weight matrices.

result The bound is tighter than existing norm-based bounds when condition numbers are small.

Koopman operators generalization bound neural networks full-rank weights operator-theoretic analysis

2023-02-12abs ↗pdf ↗

Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecs.LG

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

problem Understanding the dynamics of gradient descent in deep ReLU networks.

method Analysis of escape directions and singular values of weight matrices.

result The first singular value of the

\ell

-th layer weight matrix is at least

\ell^{\frac{1}{4}}

larger than any other singular value.

Deep Learning Gradient Descent ReLU Networks Saddle Points Low-Rank Bias

2025-05-27abs ↗pdf ↗

IMPACT: Importance-Aware Activation Space Reconstructioncs.LG

IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.

problem Resource constraints in deploying large language models (LLMs).

method IMPACT integrates activation importance into low-rank compression, optimizing for both size and accuracy.

result IMPACT achieves up to 55.4% greater model size reduction while maintaining comparable or better accuracy.

large language models compression activation importance model size reduction accuracy preservation

2025-07-04abs ↗pdf ↗

Matrix Completion under Low-Rank Missing Mechanismstat.ML

Matrix completion is a modern missing data problem where both the missing structure and the underlying parameter are high dimensional. Although missing structure is a key component to any missing data problems, existing matrix completion methods often assume a simple uniform missing mechanism. In this work, we study ma…

2018-12-19abs ↗pdf ↗

On the Initialisation of Wide Low-Rank Feedforward Neural Networksstat.ML

Optimizes wide low-rank neural networks for reduced parameters and cost.

problem Reducing the number of learnable parameters in wide neural networks.

method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.

result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.

neural networks low-rank initialization edge-of-chaos variance

2023-01-31abs ↗pdf ↗

On Generalization Bounds for Neural Networks with Low Rank Layerscs.LG

The paper analyzes how low-rank layers in neural networks improve generalization.

problem Understanding how low-rank layers affect generalization in neural networks.

method Applying Maurer's chain rule for Gaussian complexity to analyze rank and spectral norm constraints.

result Deep networks with low-rank layers achieve better generalization than those with full-rank layers.

Neural Networks Generalization Low Rank Rank Constraints Gaussian Complexity

2024-11-20abs ↗pdf ↗

An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscs.LG

Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.

problem Efficient training and deployment of large-scale deep learning models.

method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.

result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.

low-rank structures deep learning gradient descent optimization fine-tuning

2025-03-25abs ↗pdf ↗

Dual Convexified Convolutional Neural Networkscs.LG

DCCNNs reduce computational overhead and ambiguity in convolutional neural networks.

problem Reducing computational overhead and ambiguity in convolutional neural networks.

method Introducing a primal learning problem and constructing a dual convex training program, using Fenchel conjugates and Karush-Kuhn-Tucker conditions.

result Eliminates ambiguity and reduces computational overhead in constructing a large kernel matrix.

convolutional neural networks dual convexification low-rank structure Karush-Kuhn-Tucker conditions Fenchel conjugates

2022-05-27abs ↗pdf ↗

On Low Rank Directed Acyclic Graphs and Causal Structure Learningcs.LG

The paper proposes using low rank assumption to improve causal structure learning in DAGs.

problem Challenges in learning causal structures in high-dimensional, non-sparse DAGs.

method Exploits low rank assumption of DAG adjacency matrix to adapt causal structure learning methods.

result Maximum rank is highly related to hubs, suggesting low rank for scale-free networks.

causal structure learning directed acyclic graphs low rank assumption hubs scale-free networks

2020-06-10abs ↗pdf ↗

Low-Rank Generalized Linear Bandit Problemsstat.ML

An algorithm tackles low-rank linear bandit problems with improved regret bounds.

problem Low-rank linear bandit problems where rewards are inner products with an unknown low-rank matrix.

method Combines online-to-confidence-set conversion and exponentially weighted average forecaster with a covering of low-rank matrices.

result Achieves

\widetilde{O}((d_1+d_2)^{3/2}\sqrt{rT})

regret, improving over standard bounds when

r \ll \min\{d_1,d_2\}

bandit problems low-rank matrices regret bounds online-to-confidence-set exponentially weighted average

2020-06-04abs ↗pdf ↗

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscs.LG

We analyze the computational limits of LoRA for transformer models using fine-grained complexity theory.

problem Computational efficiency of LoRA fine-tuning for transformer models.

method Fine-grained complexity theory, identifying phase transitions, almost linear algorithms.

result Existence of almost linear algorithms for LoRA adaptation based on specific norms.

Transformer models Low-Rank Adaptation Fine-tuning Complexity theory Algorithm efficiency

2024-06-05abs ↗pdf ↗

DeepTwist: Learning Model Compression via Occasional Weight Distortioncs.LG

Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model com…

2018-10-30abs ↗pdf ↗

Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcs.LG

SVD training reduces DNN rank and computation load without SVD per step.

problem High memory and computational load in deep neural networks.

method Explicitly achieves low-rank DNNs during training without SVD per step, using orthogonality regularization and sparsity-inducing regularizers.

result Significantly reduces DNN rank and computation load compared to existing methods.

deep neural networks low-rank approximation SVD training orthogonality regularization sparsity-inducing regularizers

2020-04-20abs ↗pdf ↗

Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioningcs.LG

This paper addresses the challenging task of video captioning which aims to generate descriptions for video data. Recently, the attention-based encoder-decoder structures have been widely used in video captioning. In existing literature, the attention weights are often built from the information of an individual modali…

2019-11-01abs ↗pdf ↗

Deep Neural Regression Collapsecs.LG

This paper extends neural collapse to regression problems, revealing key features and structures.

problem Understanding the structure learned by deep neural networks in regression tasks.

method Established Neural Regression Collapse (NRC) across different models, analyzing feature and weight alignments.

result Deep neural regression models exhibit a collapsed feature space, aligning with target dimensions and covariances.

Neural Regression Collapse Deep Neural Networks Feature Alignment Low Rank Targets Weight Decay

2026-03-25abs ↗pdf ↗

Structured low-rank matrix completion for forecasting in time series analysisstat.ME

In this paper we consider the low-rank matrix completion problem with specific application to forecasting in time series analysis. Briefly, the low-rank matrix completion problem is the problem of imputing missing values of a matrix under a rank constraint. We consider a matrix completion problem for Hankel matrices an…

2018-02-22abs ↗pdf ↗

Low Tensor Rank Learning of Neural Dynamicsq-bio.NC

Low-rank structure emerges in neural networks during learning.

problem Understanding the evolution of synaptic connectivity over learning.

method Investigated the rank of 3-tensor formed by weight matrices throughout learning.

result Inferred weights are low-tensor-rank and evolve in a fixed low-dimensional subspace.

neural networks learning dynamics tensor rank reinforcement learning neuroscience

2023-08-22abs ↗pdf ↗

OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkagemath.ST

The truncated singular value decomposition (SVD) of the measurement matrix is the optimal solution to the_representation_ problem of how to best approximate a noisy measurement matrix using a low-rank matrix. Here, we consider the (unobservable)_denoising_ problem of how to best approximate a low-rank signal matrix bur…

2013-06-25abs ↗pdf ↗

Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscs.LG

New method finds efficient low-rank neural networks during training.

problem High memory and computational demands of neural networks.

method Restricts weight matrices to a low-rank manifold and updates low-rank factors.

result Significantly reduced time and memory resources required for training and evaluation.

neural networks low-rank dynamic model order reduction matrix differential equations approximation

2022-05-26abs ↗pdf ↗

Compressing Large Language Models using Low Rank and Low Precision Decompositioncs.LG

CALDERA compresses large language models by approximating weight matrices with low-rank, low-precision factors.

problem The large sizes of Large Language Models (LLMs) hinder deployment on edge devices.

method CALDERA approximates weight matrices

\mathbf{W}

\mathbf{Q} + \mathbf{L}\mathbf{R}

, where

\mathbf{L}

and

\mathbf{R}

are low-rank factors quantized to low precision.

result CALDERA achieves better zero-shot performance than existing techniques, especially with low bit precision.

Large Language Models Compression Low-Rank Approximation Low-Precision Edge Devices

2024-05-29abs ↗pdf ↗

CNN Acceleration by Low-rank Approximation with Quantized Factorscs.CV

New method accelerates CNNs for mobile devices by approximating tensors and quantizing weights.

problem Efficiently compress and accelerate CNNs for mobile devices.

method Low-rank tensor approximation in Tucker format combined with quantization of weights and activations.

result Our method significantly improves CNN performance on various classification tasks.

CNN acceleration low-rank approximation quantization mobile devices computer vision

2020-06-16abs ↗pdf ↗

HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMsstat.ML

Unified framework HASSLE-free decomposes large model weights into sparse and low-rank components.

problem Efficiently compress large foundation models to reduce inference costs.

method Designs a unified framework for sparse plus low-rank matrix decomposition with a local layer-wise reconstruction error objective.

result HASSLE-free framework significantly outperforms state-of-the-art methods in compression and evaluation benchmarks.

compression foundation models sparse plus low-rank LLMs reconstruction error

2025-02-02abs ↗pdf ↗

PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Trainingcs.LG

PLUMAGE improves large model training efficiency and stability.

problem Accelerator memory and networking constraints during large model training.

method Probabilistic Low rank Unbiased Minimum Variance Gradient Estimator (PLUMAGE) that resolves bias and variance issues.

result PLUMAGE reduces training loss by 28% on average across the GLUE benchmark.

large model training bottlenecks optimizer state memory constraints gradient estimation

2025-05-23abs ↗pdf ↗

Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Additioncs.LG

RNNs solve modular addition tasks using low rank and sparse Fourier structures.

problem Solving modular addition tasks with recurrent neural networks.

method Identified low rank structures and sparse Fourier representations in RNN weights.

result RNNs robust to removing individual frequencies but degrade with more ablation.

Recurrent Neural Networks Fourier Multiplication Modular Addition Low Rank Structure Sparse Representation

2025-03-28abs ↗pdf ↗

Learning Parameters for Weighted Matrix Completion via Empirical Estimationstat.ML

Recently theoretical guarantees have been obtained for matrix completion in the non-uniform sampling regime. In particular, if the sampling distribution aligns with the underlying matrix's leverage scores, then with high probability nuclear norm minimization will exactly recover the low rank matrix. In this article, we…

2014-12-31abs ↗pdf ↗