arXiv research

Stochastic gradient descent on manifolds improves low-rank approximation.

Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.

ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.

SGD with mini-batches can solve convex low-rank matrix problems efficiently.

The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.

Paper studies asymmetric matrix sensing, proving gradient descent converges to low-rank solutions.

ScaledGD accelerates ill-conditioned low-rank estimation.

Gradient descent solves asymmetric low-rank matrix sensing without balancing.

Paper develops methods for non-quadratic loss low-rank matrix recovery.

We propose a generic framework based on a new stochastic variance-reduced gradient descent algorithm for accelerating nonconvex low-rank matrix recovery. Starting from an appropriate initial estimator, our proposed algorithm performs projected gradient descent based on a novel semi-stochastic gradient specifically desi…

The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.

New method improves robust low-rank matrix completion for computer vision.

Gradient descent with small random init mimics spectral methods for low-rank matrix recovery.

Paper proposes recycling model updates in federated learning by exploiting low-rank gradient subspaces.

Paper proposes low-rank gradient approximation to save memory for deep neural network training.

Gradient descent achieves exact linear convergence rate for symmetric matrix completion.

Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.

LORENZA improves LLM fine-tuning efficiency and generalization.

LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.

The paper shows how gradient flow on over-parametrized tensor decomposition behaves like deflation.

Improved convergence for overparameterized low-rank matrix sensing.

ScaledGD algorithm estimates low-rank tensors efficiently from corrupted data.

PLUMAGE improves large model training efficiency and stability.

Efficiently implements MEG for low-rank matrix optimization problems.

Weight Decay induces low-rank weight matrices in neural networks, improving generalization.

Gradient descent recovers low-rank matrices from corrupted measurements with double over-parameterization.

FedLoRU improves FL efficiency by using low-rank updates.

Gradient descent solves asymmetric low-rank matrix factorization efficiently.

In this paper, we present GASG21 (Grassmannian Adaptive Stochastic Gradient for $L_{2,1}$ norm minimization), an adaptive stochastic gradient algorithm to robustly recover the low-rank subspace from a large matrix. In the presence of column outliers, we reformulate the batch mode matrix $L_{2,1}$ norm minimization with…

We study the problem of estimating low-rank matrices from linear measurements (a.k.a., matrix sensing) through nonconvex optimization. We propose an efficient stochastic variance reduced gradient descent algorithm to solve a nonconvex optimization problem of matrix sensing. Our algorithm is applicable to both noisy and…

New model reduces matrix factorization bias, yielding truly low-rank solutions.

We study the convergence of a variant of distributed gradient descent (DGD) on a distributed low-rank matrix approximation problem wherein some optimization variables are used for consensus (as in classical DGD) and some optimization variables appear only locally at a single node in the network. We term the resulting a…

Paper proposes a new method to separate low rank and sparse matrices without bias.

Gradient descent promotes low-rank solutions in tensor completion.

New algorithm tackles dynamic query routing to multiple embedding models.

Nonnegative low-rank matrix recovery can have spurious local minima.

Flora uses random projections to achieve high-rank updates with low memory usage.

We study gradient compression methods to alleviate the communication bottleneck in data-parallel distributed optimization. Despite the significant attention received, current compression schemes either do not scale well or fail to achieve the target test accuracy. We propose a new low-rank gradient compressor based on …

We analyze the computational limits of LoRA for transformer models using fine-grained complexity theory.

We propose a unified framework to solve general low-rank plus sparse matrix recovery problems based on matrix factorization, which covers a broad family of objective functions satisfying the restricted strong convexity and smoothness conditions. Based on projected gradient descent and the double thresholding operator, …

As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…

New method avoids spurious critical points for low-rank matrix recovery.

Gradient flow with infinitesimal initialization converges to Greedy Low-Rank Learning for matrix factorization.

Deep ReLU networks escape from the origin via saddle points with a low-rank bias.

Random Feedback Alignment helps solve low-rank matrix factorization problems.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

New findings show DNC is not optimal for deep models, revealing a low-rank bias.

Composite convex optimization problems which include both a nonsmooth term and a low-rank promoting term have important applications in machine learning and signal processing, such as when one wishes to recover an unknown matrix that is simultaneously low-rank and sparse. However, such problems are highly challenging t…