New method corrects quantization errors in LLMs using low-rank matrices.
problem Correcting quantization errors in large language models.
method Introducing low-rank weight matrices to correct quantized activations in LLMs.
result Reduces accuracy gap with original model by more than 50% using low-rank matrices.
Paper analyzes convergence of PAM method for low-rank factorization models.
problem Convergence analysis of PAM method with subspace correction for low-rank factorization models.
method Majorized proximal alternating minimization (PAM) method with subspace correction.
result Established full convergence of PAM method under KL property and column ℓ2,0-norm condition. For the problems of low-rank matrix completion, the efficiency of the widely-used nuclear norm technique may be challenged under many circumstances, especially when certain basis coefficients are fixed, for example, the low-rank correlation matrix completion in various fields such as the financial market and the low-ra…
A new method combines Laplace and Variational Bayes for scalable inference.
problem Complex models and large datasets make exact inference infeasible.
method Low-Rank Variational Bayes Correction (VBC) using Laplace method and Variational Bayes correction in a lower dimension.
result The method ensures scalability in both model complexity and data size.
Improved SVD-based NMF initialization reduces initial error and is faster.
problem Improving NMF initialization to reduce convergence error and computational cost.
method Nonnegative SVD with low-rank correction (NNSVD-LRC) that considers discarded SVD factors.
result Significantly reduces initial error with negligible additional computational cost.
Fine-tunes LLMs to correct bias in predictions.
problem LLMs exhibit bias in predictions from data.
method Supervised fine-tuning with Low-Rank Adaptation (LoRA).
result Fine-tuning corrects bias in both controlled and real-world settings.
PSI-LinUCB improves scalability for large recommender systems.
problem Efficiently training and inferring for large action spaces in recommender systems.
method Represent inverse design matrix as diagonal + low-rank correction, derive stable rank-1 and batched updates, use projector-splitting integrator.
result Demonstrated effectiveness on recommender system datasets, achieving scalable training and inference.
Recovery of low-rank matrices has recently seen significant activity in many areas of science and engineering, motivated by recent theoretical results for exact reconstruction guarantees and interesting practical applications. A number of methods have been developed for this recovery problem. However, a principled meth…
Survey of structured low-rank algorithms for MR signal recovery.
problem Recovering multidimensional signals from few non-uniform measurements.
method Structured low-rank matrix completion formulation.
result Performance guarantees and fast algorithms for large-scale MR problems.
Review of robust PCA and matrix completion methods.
problem Robust Principal Component Analysis and matrix completion with outliers.
method Various provably correct, fast, and practical solutions to RPCA and matrix completion.
result Exhaustive review of recent literature on RPCA and dynamic RPCA.
This paper describes a suite of algorithms for constructing low-rank approximations of an input matrix from a random linear image of the matrix, called a sketch. These methods can preserve structural properties of the input matrix, such as positive-semidefiniteness, and they can produce approximations with a user-speci…
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
HERA improves PLL by integrating heterogeneous loss and sparse-low-rank regularization.
problem Learning from data with partial labels.
method Combines heterogeneous loss and sparse-low-rank regularization.
result Achieves superior performance on artificial and real-world data.
Develops a curvature-corrected tangent space method for manifold-valued data.
problem Generalizing real-valued data approximation to manifold-valued data.
method Systematic approach to developing global-geometry aware, computationally feasible approximation schemes.
result Proposes CC-tHOSVD for low-rank approximation of manifold-valued data.
New method extends low-rank MDPs to continuous action spaces.
problem Limited applicability of current low-rank MDP methods to continuous action spaces.
method Extending FLAMBE algorithm to continuous action spaces with Hölder smoothness conditions.
result Similar PAC bound achieved for continuous actions with polynomial dependence on smoothness order.
Improved LoRA+ adapts large models more efficiently.
problem Suboptimal feature learning in LoRA for large models.
method Different learning rates for LoRA adapter matrices A and B.
result Improves performance and speed by 1-2% and up to 2X, respectively.
Efficiently implements MEG for low-rank matrix optimization problems.
problem Optimization over spectrahedron with low-rank matrices.
method Matrix Exponentiated Gradient (MEG) method with efficient implementations.
result Methods converge from a warm-start initialization with similar rates to full-SVD-based counterparts.
Many computer vision tasks involve processing large amounts of data contaminated by outliers, which need to be detected and rejected. While outlier detection methods based on robust statistics have existed for decades, only recently have methods based on sparse and low-rank representation been developed along with guar…
Paper proposes a new optimization framework for learning eigenfunctions of operators.
problem Computing eigenvalue decomposition of high-dimensional operators.
method Operator SVD with Neural Networks via Nested Low-Rank Approximation.
result Proposed method efficiently learns top-L singular values and functions in the correct order.
Sharp condition found for Burer-Monteiro method to work for MaxCut-type SDPs.
problem MaxCut-type semidefinite programs with low-rank solutions.
method Sharp condition on Laplacian matrix conditioning for global minimizers of non-convex problem.
result Any second-order critical point is a global minimizer under the given condition.
Proposes CC-NMDF for analyzing manifold-valued data.
problem Nonlinear structure in manifold-valued data requires new analysis methods.
method Curvature-corrected nonnegative manifold data factorization (CC-NMDF) with an iterative algorithm.
result Demonstrates CC-NMDF on real-world diffusion tensor MRI data.
Framework for incomplete multi-view learning improves efficiency and clustering accuracy.
problem Incomplete representation in multi-view data.
method Joint Embedding Learning and Low-Rank Approximation (JELLA) framework.
result Improves efficiency and clustering accuracy in incomplete multi-view data.
A k-space deep learning method corrects EPI ghost artifacts without a reference scan.
problem Nyquist ghost artifacts in EPI MRI due to phase mismatch between even and odd echoes.
method Structured low-rank Hankel matrix approaches combined with data-driven Hankel matrix decomposition and deep convolutional neural networks.
result The proposed k-space deep learning method outperforms existing methods in image quality and computing time.
New algorithm solves low-rank phase retrieval with fewer measurements than previously possible.
problem Recovering low-rank matrices from phaseless projections.
method Developed an alternating minimization algorithm (AltMinLowRaP) with provable correctness and geometric convergence.
result The algorithm solves low-rank phase retrieval with mq≥Cnr4log(1/ε) measurements, achieving ε accuracy with high probability. We consider the problem of matrix column subset selection, which selects a subset of columns from an input matrix such that the input can be well approximated by the span of the selected columns. Column subset selection has been applied to numerous real-world data applications such as population genetics summarization,…
This work introduces 'Artificial Entanglement' to understand LLMs' fine-tuning effectiveness.
problem Understanding the effectiveness of parameter-efficient fine-tuning methods for large language models.
method Adopting a quantum-information-inspired perspective, the study measures 'Artificial Entanglement' in neural networks.
result LoRA and FFT induce distinct internal entanglement signatures but not external ones, suggesting a 'no-hair' property.
Enhances knowledge graph completion with mixed geometry tensor factorization.
problem Capturing nuanced distributional properties in knowledge graphs.
method Combines Euclidean and hyperbolic geometries for tensor factorization.
result Improves link prediction accuracy with fewer parameters.
Total least squares regression sped up with input sparsity time.
problem Efficiently solving total least squares regression problems.
method Input sparsity time algorithm for solving implicit linear systems.
result Running time of ildeO(nnz(A)+nnz(B))+poly(n/ε)⋅d. Asynchronous federated modeling improves spatial data sharing without centralizing raw data.
problem Privacy and bandwidth constraints in distributed spatial data.
method Asynchronous federated modeling using low-rank Gaussian process approximations with block-wise optimization and adaptive strategies.
result Asynchronous federated modeling achieves synchronous performance and outperforms it in heterogeneous settings.
A new method for robust matrix completion overcomes limitations of existing approaches.
problem Robust matrix completion from corrupted entries, especially under overparameterization and ill-conditioning.
method Factorization-based iterative algorithm combining Gauss-Newton linearization and outlier removal.
result Theoretical guarantees of exact recovery for suitable assumptions.
Paper proposes a method for estimating complex low-rank matrices from phase-only measurements.
problem Estimating complex low-rank matrices from magnitude-only measurements.
method A hierarchical prior model with a Gaussian-Wishart distribution is used to promote low-rankness. A variational EM algorithm is developed to solve the problem.
result The proposed method is less sensitive to initialization and performs well with random initialization.
Low-rank modeling generally refers to a class of methods that solve problems by representing variables of interest as low-rank matrices. It has achieved great success in various fields including computer vision, data mining, signal processing and bioinformatics. Recently, much progress has been made in theories, algori…
Unified theory and debiasing framework for random oblique projections in high dimensions.
problem Systematic statistical bias in random oblique projections induced by sampling.
method Unified non-asymptotic theory and debiasing framework.
result Sharp bias--variance characterizations and improved approximation accuracy.
Paper proposes a new technique to compress CNNs while maintaining accuracy.
problem CNNs struggle with traditional low-rank approximation methods, leading to degraded accuracy.
method Introduces a training technique that finds a flat minimum in low-rank approximation without a decomposed structure.
result CNN models can be compressed with higher accuracy and lower computation than conventional methods.
SGD with mini-batches can solve convex low-rank matrix problems efficiently.
problem Solving large-scale convex low-rank matrix problems efficiently.
method Stochastic Gradient Descent with mini-batches and low-rank projections.
result SGD with mini-batches produces low-rank iterates with high probability.
New spectral methods improve matrix estimation in RL with low-rank structure.
problem Estimating matrices with low-rank structure in reinforcement learning.
method Spectral-based matrix estimation approaches.
result Spectral methods efficiently recover singular subspaces and minimize entry-wise error.
Novel method for efficient low-rank matrix estimation and bandit algorithms.
problem Low-rank matrix estimation and bandit problems.
method LowPopArt method for low-rank matrix estimation and novel experimental design criterion.
result Improved recovery guarantees and regret bounds for low-rank bandit algorithms.
Paper improves MVSC using tensor low-rank modeling.
problem Improving multi-view spectral clustering.
method Structured tensor low-rank norm for MVSC optimization.
result Proposed method outperforms state-of-the-art methods.
Contrastive UCB improves RL by learning feature representations efficiently.
problem Improving feature learning in RL for online decision making.
method Proposes UCB-based contrastive learning algorithms for RL in MDPs and MGs.
result Proves sample efficiency in learning optimal policies and Nash equilibria.
This paper solves tensor robust principal component analysis via scaled gradient descent.
problem Extracting useful information from tensor data robust to corruptions and ill-conditioning.
method Directly recovers low-rank tensor factors via scaled gradient descent with adaptive thresholding.
result The proposed algorithm converges linearly to the true low-rank tensor at a constant rate independent of the condition number.
Low-rank approach to metric learning from data.
problem Learning a Mahalanobis metric from data.
method Low-rank geometric mean metric learning (GMML) approach.
result Competes effectively with GMML at lower ranks.
Robust PCA method optimizes low-rank matrices with corrupted data.
problem Recover a low-rank matrix from grossly corrupted observations.
method Nonconvex optimization on the manifold of low-rank matrices, using manifold optimization algorithms.
result Proposed algorithms converge to the underlying low-rank matrix linearly with proper initialization.
The paper proposes using low rank assumption to improve causal structure learning in DAGs.
problem Challenges in learning causal structures in high-dimensional, non-sparse DAGs.
method Exploits low rank assumption of DAG adjacency matrix to adapt causal structure learning methods.
result Maximum rank is highly related to hubs, suggesting low rank for scale-free networks.
New method uses low-rank tensor factor analysis for better image restoration.
problem Restoring images from limited data.
method Low-rank tensor factor analysis combined with ADMM.
result The method outperforms traditional approaches, especially at low sampling rates.
Algorithm recovers multiple low-rank matrices from unlabeled data.
problem Learning mixtures of low-rank models from unlabelled data.
method Three-stage meta-algorithm that copes with non-convexity and noise.
result Near-optimal sample and computational complexities under Gaussian designs.
Paper tackles fair low-rank approximation and column subset selection.
problem Minimize loss over sub-populations in machine learning.
method Developed algorithms for fair low-rank approximation and fair column subset selection.
result Achieved polynomial time algorithms for fair low-rank approximation.
Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
New insights into training deep networks with low rank layers.
problem Efficiency in training deep neural networks.
method Analysis of techniques for training in low rank space.
result Falsified common beliefs in training deep networks.