STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
problem Improving sparsity in DNNs for better accuracy and lower inference cost.
method Soft Threshold Reparameterization (STR) using the soft-threshold operator on DNN weights.
result STR achieves state-of-the-art accuracy and reduces FLOPs by up to 50%.
LOCUS separates brain network connectivity matrices efficiently.
problem High dimensionality, latent sources, and spurious findings in analyzing brain connectivity matrices.
method LOCUS: low-rank structure with uniform sparsity, iterative Node-Rotation algorithm.
result LOCUS achieves more efficient and accurate source separation for connectivity matrices.
Understanding statistical inference under possibly non-sparse high-dimensional models has gained much interest recently. For a given component of the regression coefficient, we show that the difficulty of the problem depends on the sparsity of the corresponding row of the precision matrix of the covariates, not the spa…
Random sampling has become a critical tool in solving massive matrix problems. For linear regression, a small, manageable set of data rows can be randomly selected to approximate a tall, skinny data matrix, improving processing time significantly. For theoretical performance guarantees, each row must be sampled with pr…
Graphical models have become a very popular tool for representing dependencies within a large set of variables and are key for representing causal structures. We provide results for uniform inference on high-dimensional graphical models with the number of target parameters d being possible much larger than sample siz…
Paper proposes efficient algorithm for recovering sparsity pattern from deterministic missing data.
problem Recovering sparsity pattern from datasets with deterministic missing structure.
method Proposes an efficient algorithm for missing value imputation using topological property of censorship filter.
result Consistently recovers the sparsity pattern with high probability in polynomial time and logarithmic sample complexity.
Multiple kernel learning (MKL), structured sparsity, and multi-task learning have recently received considerable attention. In this paper, we show how different MKL algorithms can be understood as applications of either regularization on the kernel weights or block-norm-based regularization, which is more common in str…
Hypothesis tests in models whose dimension far exceeds the sample size can be formulated much like the classical studentized tests only after the initial bias of estimation is removed successfully. The theory of debiased estimators can be developed in the context of quantile regression models for a fixed quantile value…
LEWIS merges LLMs without training, improving performance on specific tasks.
problem Limited performance improvement of merged models on specific benchmarks.
method Guided model merging using layer-wise sparsity and task-vector pruning.
result Improved model performance by up to 11.3% on math-solving tasks.
Fiedler regularization uses spectral graph theory to improve neural network performance.
problem Improving neural network performance by penalizing weights based on connectivity.
method Uses the Fiedler value of the neural network's graph as a regularization tool, providing theoretical and computational methods.
result Demonstrates Fiedler regularization's effectiveness in improving neural network performance.
Robust STAP with coprime arrays reduces clutter using sparse modeling.
problem Limited performance due to training samples support in practical applications.
method Two-stage approach: 1) RD virtual snapshot, 2) RD sparse measurement modeling with OMP-like recovery.
result Robust to prior knowledge errors, good clutter suppression performance.
Paper converts deep networks to flat, equivalent kernel machines.
problem Capacity control and uniform convergence in deep learning.
method Push-forward transformation from deep networks to indefinite kernel machines.
result Flat network weights are Lp-norm regularized (0<p<1).
Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure not only improves generalization but also leads to extremely sparse neural architectures by automatica…
A new model for sparse networks improves consistency of score estimators.
problem Statistical inference in large sparse networks.
method General pairwise comparison model with flexible parametrization.
result Maximum likelihood estimator is uniformly consistent under sparse conditions.
Gaussian multiplicative noise is commonly used as a stochastic regularisation technique in training of deterministic neural networks. A recent paper reinterpreted the technique as a specific algorithm for approximate inference in Bayesian neural networks; several extensions ensued. We show that the log-uniform prior us…
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
Efficient algorithms for sparse parameter recovery in mixture models.
problem Support recovery of high-dimensional sparse latent vectors in mixture models.
method Efficient algorithms with logarithmic sample complexity dependence on dimensionality.
result First guarantees on support recovery for various mixture models.
The paper provides generalization bounds for metric learning using neural network embeddings.
problem Generalization guarantees for metric learning with neural network embeddings.
method Uniform generalization bounds for two regimes: sparse and bounded amplification.
result Dimension-free generalization bounds can be achieved even without sparsity in solutions.
Deep Convolutional Sparse Coding (D-CSC) is a framework reminiscent of deep convolutional neural networks (DCNNs), but by omitting the learning of the dictionaries one can more transparently analyse the role of the activation function and its ability to recover activation paths through the layers. Papyan, Romano, and E…
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…
New Performer model tackles long-sequence protein modeling.
problem Challenges of training complex Transformer models for long sequences.
method Linearly scalable long-context Transformer architecture, Performer.
result Performer provides strong theoretical guarantees and is effective for protein sequence modeling.
Most high-dimensional estimation and prediction methods propose to minimize a cost function (empirical risk) that is written as a sum of losses associated to each data point. In this paper we focus on the case of non-convex losses, which is practically important but still poorly understood. Classical empirical process …
We consider the problem of recovering a signal x∗∈Rn, from magnitude-only measurements yi=∣⟨ai,x∗⟩∣ for i=[m]. Also called the phase retrieval, this is a fundamental challenge in bio-,astronomical imaging and speech processing. The problem abov…
Proposes a co-hub node model for multiview graph learning.
problem Identifying shared graphical structures in heterogeneous datasets.
method Enforces structured sparsity on co-hub nodes across multiple views.
result Demonstrates improved precision and interpretive insight in learning multiview graphs.
We consider the problem of recovering block-sparse signals whose structures are unknown \emph{a priori}. Block-sparse signals with nonzero coefficients occurring in clusters arise naturally in many practical scenarios. However, the knowledge of the block structure is usually unavailable in practice. In this paper, we d…
This paper establishes conditions for sparse signal recovery with sparse measurements.
problem Recovering the support of a sparse signal using noisy projections with sparse measurement matrices.
method Establishes sufficient conditions for successful sparse recovery using sparse measurement matrices.
result A phase transition threshold for sparse recovery in the sparse setting is discovered, revealing a trade-off between sampling complexity and measurement sparsity.
Paper tackles low-rank matrix recovery with column ℓ2,0-norm regularization.
problem Low-rank matrix recovery problems with column sparsity constraints.
method Developed alternating majorization-minimization (AMM) methods with extrapolation and hybrid AMM.
result Global convergence analysis and superior performance in matrix completion problems.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
FSD-CAP improves graph feature imputation under high missing rates.
problem Challenges in imputing missing node features in graphs, especially under high missing rates.
method Two-stage framework: subgraph expansion, fractional diffusion, class-aware propagation.
result Significantly improved imputation quality compared to existing methods, achieving high accuracy on benchmark datasets.
New layer sparsity concept improves neural networks.
problem Improving neural network efficiency and interpretability.
method Formulated layer sparsity, introduced regularization and refitting schemes.
result Generated more compact and accurate neural networks.
In order to alleviate data sparsity and overfitting problems in maximum likelihood estimation (MLE) for sequence prediction tasks, we propose the Generative Bridging Network (GBN), in which a novel bridge module is introduced to assist the training of the sequence prediction model (the generator network). Unlike MLE di…
Proposes a tensor Laplacian-based method for better subspace clustering of non-uniformly distributed data.
problem LRR's inability to handle non-uniform data distribution and local information loss.
method Tensor Laplacian Regularized Low-Rank Representation (TLRR) using hypergraph model and tensor Laplacian algorithm.
result Higher accuracy and precision in subspace clustering compared to state-of-the-art methods.
Enhances privacy in federated learning with Laplacian smoothing.
problem Protecting data privacy in federated learning while maintaining model accuracy.
method Laplacian smoothing for differentially private federated learning (DP-Fed-LS).
result Improves model accuracy with differential privacy guarantee and membership privacy.
SpinSVAR estimates SVAR models with sparse input, improving accuracy and scalability.
problem Estimating SVAR models with sparse input assumptions.
method SpinSVAR models input as independent Laplacian variables, enforcing sparsity and using least absolute error regression.
result SpinSVAR outperforms state-of-the-art methods in accuracy and runtime, identifying significant structural shocks.
Our work is focused on the joint sparsity recovery problem where the common sparsity pattern is corrupted by Poisson noise. We formulate the confidence-constrained optimization problem in both least squares (LS) and maximum likelihood (ML) frameworks and study the conditions for perfect reconstruction of the original r…
New theorem for generalized group sparsity improves consistency and convergence rates.
problem Improving statistical inference in high-dimensional data with element-wise and group-wise sparsity.
method Developed a generalized version of Sparse-Group Lasso and proved a universal theorem for consistency and convergence rates.
result Obtained results on consistency and convergence rates for different forms of double sparsity regularization.
New findings show sparse signals in MRA model require fewer measurements than previously thought.
problem Learning an unknown signal from repeated noisy images under group actions.
method Enhanced probabilistic method and analysis of uniform uncertainty principles.
result Sparse signals exhibit intermediate σ4 sample complexity, improving over traditional σ2. This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL-p% sparsity, analyzes trends with training data, width-depth ratio, and parameter scale. result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
Embeds sparsity in deep neural networks, allowing exact zero parameters.
problem Learning sparse structures in deep networks.
method Embeds sparsity into neural network structure, allowing exact zero parameters during training.
result Can learn both structured and unstructured sparsity.
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
New sparsity attacks degrade DNN efficiency, raising concerns for resource-constrained systems.
problem Vulnerabilities in DNNs through energy and latency attacks.
method Proposed sparsity attacks that modify DNN inputs to reduce activation sparsity, increasing execution time and energy consumption.
result Adversarial sparsity attacks can degrade DNN efficiency by up to 1.82x in image recognition DNNs.
Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are un…
Complexity helps identify sparse risk factors in asset pricing.
problem Tension between feature richness and economic parsimony in high-dimensional asset pricing.
method Expanding feature space and using basis pursuit to discover sparse risk factors.
result Nonlinear feature expansions combined with basis pursuit yield superior out-of-sample performance.
We identify linear dynamical systems under convex constraints with fewer samples.
problem Identifying linear dynamical systems with prior structural information.
method Constrained least squares estimator with error bounds dependent on convex set size.
result Linear dynamical systems can be reliably estimated with fewer samples than unconstrained settings.
Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.
problem Inactivity of most neurons in FFN of Transformers, leading to inefficiency.
method Top-k masking, statistical top-k algorithm, and predictor for activated entries.
result Only 8% of FFN neurons are activated, achieving up to 2.5x wall-time speedup.
We provide adaptive inference methods, based on ℓ1 regularization, for regular (semi-parametric) and non-regular (nonparametric) linear functionals of the conditional expectation function. Examples of regular functionals include average treatment effects, policy effects, and derivatives. Examples of non-regular f…
New method for multiclass classification reduces error bounds.
problem High-dimensional multiclass classification with sparse coefficients.
method Sparse multinomial logistic regression with convex penalties.
result Plug-in classifiers achieve minimax generalization error bounds.