Proposes a model for identifying edges in low-rank dynamical networks.
problem Inability of conventional methods to handle low-rank dynamical networks.
method Low rank dynamical network model with causal Wiener filtering.
result Consistent method for estimating all network edges.
Review of algorithms for linear system approximations.
problem Linear approximation of high-dimensional dynamical systems.
method State-of-the-art algorithms for low-rank DMD.
result Provides additional details for comprehensive understanding.
We propose a method to infer stochastic low-rank RNNs from neural data.
problem Fitting low-rank RNNs to noisy, stochastic neural data.
method Variational sequential Monte Carlo methods for stochastic low-rank RNNs.
result Lower dimensional latent dynamics compared to state-of-the-art methods.
New method finds efficient low-rank neural networks during training.
problem High memory and computational demands of neural networks.
method Restricts weight matrices to a low-rank manifold and updates low-rank factors.
result Significantly reduced time and memory resources required for training and evaluation.
A new method reduces high-dimensional filtering to quadratic complexity.
problem High-dimensional dynamical systems inference and simulation.
method Low-rank Kalman filtering using dynamical low-rank integrator.
result The method reproduces exact Kalman filter in low-rank limit.
AIR-Net adapts low-rank regularization dynamically for better image completion.
problem Fixed low-rank regularization limits adaptability to different images.
method AIR-Net uses adaptive and implicit regularization parameterized by a dynamic Laplacian matrix.
result AIR-Net enhances implicit regularization and outperforms fixed methods in non-uniform missing data scenarios.
We consider the problem of modeling multivariate time series with parsimonious dynamical models which can be represented as sparse dynamic Bayesian networks with few latent nodes. This structure translates into a sparse plus low rank model. In this paper, we propose a Gaussian regression approach to identify such a mod…
SALT models combine ARHMM and SLDS for efficient, interpretable time-series analysis.
problem Efficient modeling of systems with time-varying dynamics and long-range dependencies.
method Switching autoregressive low-rank tensor models parameterized with a low-rank factorization.
result SALT models provide a balance of interpretability and efficiency, outperforming ARHMMs and SLDSs.
This work studies the linear approximation of high-dimensional dynamical systems using low-rank dynamic mode decomposition (DMD). Searching this approximation in a data-driven approach is formalised as attempting to solve a low-rank constrained optimisation problem. This problem is non-convex and state-of-the-art algor…
New approach uses compressible dynamics to train deep models efficiently.
problem Efficient training of deep overparameterized models with low-rank structures.
method Leveraging low-dimensional structures and compressible dynamics within model parameters.
result Improved training efficiency and reduced overfitting in language models.
Riemannian gradient descent escapes some spurious critical points on low-rank matrix manifold.
problem Spurious critical points on the boundary of low-rank matrix manifold.
method Riemannian gradient descent with dynamical low-rank approximation and rescaled gradient flow.
result Riemannian gradient descent escapes some spurious critical points on the boundary of the manifold.
Dynamic Mode Decomposition (DMD) has emerged as a powerful tool for analyzing the dynamics of non-linear systems from experimental datasets. Recently, several attempts have extended DMD to the context of low-rank approximations. This extension is of particular interest for reduced-order modeling in various applicative …
Active learning method for neural population dynamics using optogenetics.
problem Efficiently selecting neurons to stimulate for identifying neural population dynamics.
method Developed active learning procedure for low-rank regression to determine informative photostimulation patterns.
result Demonstrated a two-fold reduction in data required for predictive power using low-rank linear dynamical systems model.
LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.
problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.
method Generalized student-teacher setting with i.i.d. samples and online gradient descent.
result Gradient descent converges to the teacher model in d k O ( 1 ) dk^{O(1)} d k O ( 1 ) iterations under certain conditions. Low-rank matrix factorizations arise in a wide variety of applications -- including recommendation systems, topic models, and source separation, to name just a few. In these and many other applications, it has been widely noted that by incorporating temporal information and allowing for the possibility of time-varying …
LASER compresses recursive model activations by exploiting their low-dimensional structure.
problem Understanding and optimizing the geometric structure of recursive reasoning trajectories.
method Dynamic low-rank basis tracking via matrix-free subspace tracking with a fidelity-triggered reset mechanism.
result Recursive activations occupy a linear, low-dimensional subspace that can be compressed efficiently.
MOSAIC detects change points in dynamic networks with low-rank and sparse changes.
problem Detecting change points in dynamic networks with specific structural properties.
method Eigen-decomposition-based test with screened signals and residual-based adjustment.
result MOSAIC achieves minimax-optimal detection and testing rates.
Sparsity-based approaches have been popular in many applications in image processing and imaging. Compressed sensing exploits the sparsity of images in a transform domain or dictionary to improve image recovery from undersampled measurements. In the context of inverse problems in dynamic imaging, recent research has de…
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ \ell ℓ -th layer weight matrix is at least ℓ 1 4 \ell^{\frac{1}{4}} ℓ 4 1 larger than any other singular value. Research reveals deep networks often learn low-rank structures, leading to more efficient training and fine-tuning.
problem Efficient training and deployment of large-scale deep learning models.
method Complementary theoretical perspectives on low-rank structures during training and convergence, and practical applications of LoRA and masked training.
result Understanding and exploiting low-rank structures can improve efficiency and effectiveness of training and fine-tuning.
FLAMBE tackles RL in low rank MDPs by learning features.
problem Dealing with the curse of dimensionality in RL.
method Develops FLAMBE, a method that engages in exploration and representation learning for RL in low rank transition models.
result FLAMBE efficiently learns features for RL in low rank transition models.
The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.
problem Stability of low-rank implicit regularization in perturbed deep matrix factorization.
method Derives spectral conditions for gradient descent to exhibit a low-rank phase in noiseless settings and analyzes perturbed dynamics.
result Gradient descent converges to a low-rank solution under perturbation, with explicit dependence on perturbation size.
The paper shows how gradient flow on over-parametrized tensor decomposition behaves like deflation.
problem Understanding the training dynamics of gradient flow on tensor decomposition.
method Empirical observation and mathematical proof of gradient flow dynamics for orthogonally decomposable tensors.
result Gradient flow dynamics for orthogonally decomposable tensors follows a tensor deflation process, recovering all tensor components.
New algorithm tackles dynamic query routing to multiple embedding models.
problem Dynamic query routing to multiple embedding models under adversarial conditions.
method Formalized as adversarial contextual linear bandit with low-rank experts, proposed HPG algorithm.
result HPG algorithm achieves linearized policy regret of i l d e O ( s M T ) ilde{\mathcal O}(s\sqrt{M T}) i l d e O ( s M T ) . New algorithm catches moving subspaces in bandit problems.
problem Adapt to changing low-dimensional latent subspaces in bandit settings.
method Piecewise-stationary low-rank linear contextual bandits with CUSUM-style boundary detection.
result Achieves intrinsic rank dynamic regret rate of O ( r T ) O(r\sqrt{T}) O ( r T ) . A low-rank tensor model simplifies multi-dimensional Markov chains.
problem Simplifying the dynamics of multi-dimensional Markov chains.
method Low-rank tensor decomposition for multi-dimensional state spaces.
result Our tensor model requires fewer parameters and samples than conventional methods.
Paper tackles dynamic assortment with dual contexts, improving revenue in e-commerce.
problem Maximizing revenue in e-commerce with personalized recommendations from vast catalogs.
method Low-rank dynamic assortment model and upper confidence bound approach.
result Regret bound of i l d e O ( ( d 1 + d 2 ) r T ) ilde{O}((d_1+d_2)r\sqrt{T}) i l d e O (( d 1 + d 2 ) r T ) for dynamic assortment problem. Principal Components Analysis (PCA) is one of the most widely used dimension reduction techniques. Robust PCA (RPCA) refers to the problem of PCA when the data may be corrupted by outliers. Recent work by Cand{è}s, Wright, Li, and Ma defined RPCA as a problem of decomposing a given data matrix into the sum of a low-ran…
New model reduces matrix factorization bias, yielding truly low-rank solutions.
problem Gradient descent's implicit bias in matrix factorization.
method Introducing a new factorization model with constrained factors and diagonal components.
result The new model consistently exhibits a strong implicit bias, yielding truly low-rank solutions.
Gradient descent promotes low-rank solutions in tensor completion.
problem Implicit regularization in tensor factorization using gradient descent.
method Introduced deep Tucker and TensorTrain (TT) unconstrained factorization to address tensor completion.
result Gradient descent promotes solutions with low-rank.
New model-free algorithms learn representations for low-rank MDPs efficiently.
problem Learning representations in reinforcement learning for low-rank MDPs.
method Developed minimax representation learning objective and interleaved with reward-free exploration.
result Proven sample efficiency and scalability to complex environments.
This paper considers a new framework to detect communities in a graph from the observation of signals at its nodes. We model the observed signals as noisy outputs of an unknown network process, represented as a graph filter that is excited by a set of unknown low-rank inputs/excitations. Application scenarios of this m…
We consider dynamic pricing with many products under an evolving but low-dimensional demand model. Assuming the temporal variation in cross-elasticities exhibits low-rank structure based on fixed (latent) features of the products, we show that the revenue maximization problem reduces to an online bandit convex optimiza…
Analyzes learning dynamics of RNNs under locality constraints.
problem Understanding learning dynamics in RNNs with locality constraints.
method Dynamical systems theory applied to data-aligned linear RNNs.
result RFLO solutions are restricted to low-rank perturbations of initial parameters.
PSI-LinUCB improves scalability for large recommender systems.
problem Efficiently training and inferring for large action spaces in recommender systems.
method Represent inverse design matrix as diagonal + low-rank correction, derive stable rank-1 and batched updates, use projector-splitting integrator.
result Demonstrated effectiveness on recommender system datasets, achieving scalable training and inference.
UA-LQE improves value function learning by selectively erasing uncertain entries in Q-matrix.
problem Improving value function learning in complex reinforcement learning tasks.
method Uncertainty-aware low-rank Q-matrix estimation (UA-LQE) algorithm.
result UA-LQE selectively erases uncertain entries in Q-matrix to improve value function approximation.
Reduced modeling of a computationally demanding dynamical system aims at approximating its trajectories, while optimizing the trade-off between accuracy and computational complexity. In this work, we propose to achieve such an approximation by first embedding the trajectories in a reproducing kernel Hilbert space (RKHS…
LoRA fine-tuning causes forgetting, studied via particle system dynamics.
problem Catastrophic forgetting in LoRA fine-tuning.
method Mean-field self-attention model, partial differential equations, dynamical systems.
result Characterization of phase transitions in forgetting behavior.
A new method for efficient neural network fine-tuning using queryable low-rank update atoms.
problem Rigidity of static low-rank adaptation methods when input and depth-wise computation vary.
method A shared queryable memory of low-rank update atoms, allowing dynamic and context-sensitive adaptation.
result Improves final test performance and training stability compared to standard low-rank adaptation.
Low-rank structure emerges in neural networks during learning.
problem Understanding the evolution of synaptic connectivity over learning.
method Investigated the rank of 3-tensor formed by weight matrices throughout learning.
result Inferred weights are low-tensor-rank and evolve in a fixed low-dimensional subspace.
Algorithm for low-rank matrix bandits with heavy-tailed rewards, achieving nearly optimal regret bound.
problem Stochastic low-rank matrix bandit with heavy-tailed rewards.
method LOTUS algorithm using truncation and dynamic exploration.
result Regret bound of order $ ilde O(d^ rac{3}{2}r^ rac{1}{2}T^ rac{1}{1+δ}/ ilde{D}_{rr})$ without knowing T T T . TensorGuide improves LoRA efficiency and expressivity through joint tensor-train optimization.
problem Limited expressivity and generalization of standard LoRA.
method TensorGuide uses a unified tensor-train structure with controlled Gaussian noise to generate correlated low-rank matrices.
result TensorGuide achieves superior accuracy and scalability with fewer parameters compared to standard LoRA and TT-LoRA.
AdaRL improves robust RL by adaptively adjusting policy complexity.
problem Handling epistemic uncertainty in environment dynamics.
method Bi-level optimization framework with adaptive rank adjustment.
result AdaRL outperforms existing methods on MuJoCo benchmarks.
Flora uses random projections to achieve high-rank updates with low memory usage.
problem Excessive memory usage in large neural networks during training.
method Flora approximates LoRA using random projections to enable high-rank updates with sublinear space complexity.
result Flora achieves high-rank updates with significantly reduced memory usage compared to LoRA.
Survey of structured low-rank algorithms for MR signal recovery.
problem Recovering multidimensional signals from few non-uniform measurements.
method Structured low-rank matrix completion formulation.
result Performance guarantees and fast algorithms for large-scale MR problems.
Often, large, high dimensional datasets collected across multiple modalities can be organized as a higher order tensor. Low-rank tensor decomposition then arises as a powerful and widely used tool to discover simple low dimensional structures underlying such data. However, we currently lack a theoretical understanding …
New method learns low-dimensional representations of nonlinear time series without supervision.
problem Learning low-dimensional representations of nonlinear time series without supervision.
method Based on monotone variational inequality, the method learns representations by assuming sequences arise from a common domain.
result The method can learn the geometry for the entire domain and faithful representations for the dynamics of each individual sequence.