arXiv research

LASER compresses recursive model activations by exploiting their low-dimensional structure.

IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.

New method corrects quantization errors in LLMs using low-rank matrices.

Active learning method for neural population dynamics using optogenetics.

New method recovers matrix column space with active sampling for better results.

New CSC model extracts EEG signals with low noise sensitivity.

Recovery of low-rank matrices has recently seen significant activity in many areas of science and engineering, motivated by recent theoretical results for exact reconstruction guarantees and interesting practical applications. A number of methods have been developed for this recovery problem. However, a principled meth…

Active sampling selects few points for accurate model reduction of high-fidelity systems.

Paper compresses ViTs models by 60% with minimal accuracy loss.

We propose a method to infer stochastic low-rank RNNs from neural data.

Representation of human actions as a sequence of human body movements or action attributes enables the development of models for human activity recognition and summarization. We present an extension of the low-rank representation (LRR) model, termed the clustering-aware structure-constrained low-rank representation (CS…

The paper reveals low-rank structure in neural network gradients, influenced by data and model parameters.

Introduces TT-NF for more compact neural field representations.

Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.

Kolmogorov-Arnold Networks offer improved interpretability and parsimony in science tasks.

Flat minima lead to better generalization in low-rank matrix recovery models.

The paper establishes theoretical foundations for low-rank knowledge distillation in LLMs.

EVA adapts LoRA for faster, more efficient fine-tuning.

This paper introduces a new multivariate convolutional sparse coding based on tensor algebra with a general model enforcing both element-wise sparsity and low-rankness of the activations tensors. By using the CP decomposition, this model achieves a significantly more efficient encoding of the multivariate signal-partic…

Robust tensor CP decomposition involves decomposing a tensor into low rank and sparse components. We propose a novel non-convex iterative algorithm with guaranteed recovery. It alternates between low-rank CP decomposition through gradient ascent (a variant of the tensor power method), and hard thresholding of the resid…

New method for robust PCA with exponential family distributions.

We propose a novel general algorithm LHAC that efficiently uses second-order information to train a class of large-scale l1-regularized problems. Our method executes cheap iterations while achieving fast local convergence rate by exploiting the special structure of a low-rank matrix, constructed via quasi-Newton approx…

New method estimates neuronal connectivity from partially observed data.

New method accelerates CNNs for mobile devices by approximating tensors and quantizing weights.

LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.

Active learning aims to obtain a classifier of high accuracy by using fewer label requests in comparison to passive learning by selecting effective queries. Many active learning methods have been developed in the past two decades, which sample queries based on informativeness or representativeness of unlabeled data poi…

ZDP detects drift in large language models without labels, proving key theorems and metrics.

Study on identifiability of deep polynomial neural networks.

The goal of a recommendation system is to predict the interest of a user in a given item by exploiting the existing set of ratings as well as certain user/item features. A standard approach to modeling this problem is Inductive Matrix Completion where the predicted rating is modeled as an inner product of the user and …

This paper proposes a new method to compress CNNs for medical image analysis, improving efficiency and accuracy.

Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.

Optimizes algorithms for non-concave bandit problems.

This paper extends the Gaussian process interpretation of deep networks to more varied weight distributions.

Dictionary learning and component analysis are part of one of the most well-studied and active research fields, at the intersection of signal and image processing, computer vision, and statistical machine learning. In dictionary learning, the current methods of choice are arguably K-SVD and its variants, which learn a …

How can we correlate neural activity in the human brain as it responds to words, with behavioral data expressed as answers to questions about these same words? In short, we want to find latent variables, that explain both the brain activity, as well as the behavioral responses. We show that this is an instance of the C…

FMMNN combines sine activations with multi-component, multi-layer structure for high-frequency function approximation.

This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.

Data often comes in the form of an array or matrix. Matrix factorization techniques attempt to recover missing or corrupted entries by assuming that the matrix can be written as the product of two low-rank matrices. In other words, matrix factorization approximates the entries of the matrix by a simple, fixed function-…

Residential homes constitute roughly one-fourth of the total energy usage worldwide. Providing appliance-level energy breakdown has been shown to induce positive behavioral changes that can reduce energy consumption by 15%. Existing approaches for energy breakdown either require hardware installation in every target ho…

Low-rank modeling generally refers to a class of methods that solve problems by representing variables of interest as low-rank matrices. It has achieved great success in various fields including computer vision, data mining, signal processing and bioinformatics. Recently, much progress has been made in theories, algori…

SGD with mini-batches can solve convex low-rank matrix problems efficiently.

New spectral methods improve matrix estimation in RL with low-rank structure.

Novel method for efficient low-rank matrix estimation and bandit algorithms.

In this paper, we consider the problem of low-rank phase retrieval whose objective is to estimate a complex low-rank matrix from magnitude-only measurements. We propose a hierarchical prior model for low-rank phase retrieval, in which a Gaussian-Wishart hierarchical prior is placed on the underlying low-rank matrix to …

Paper improves MVSC using tensor low-rank modeling.

In this paper, we unravel a fundamental connection between weighted finite automata~(WFAs) and second-order recurrent neural networks~(2-RNNs): in the case of sequences of discrete symbols, WFAs and 2-RNNs with linear activation functions are expressively equivalent. Motivated by this result, we build upon a recent ext…

Low-rank approximation is an effective model compression technique to not only reduce parameter storage requirements, but to also reduce computations. For convolutional neural networks (CNNs), however, well-known low-rank approximation methods, such as Tucker or CP decomposition, result in degraded model accuracy becau…

The paper proposes using low rank assumption to improve causal structure learning in DAGs.