New theorem for generalized group sparsity improves consistency and convergence rates.
problem Improving statistical inference in high-dimensional data with element-wise and group-wise sparsity.
method Developed a generalized version of Sparse-Group Lasso and proved a universal theorem for consistency and convergence rates.
result Obtained results on consistency and convergence rates for different forms of double sparsity regularization.
The paper studies how regularization parameters affect sparsity in deep neural networks.
problem Reducing the complexity of deep neural networks by promoting sparsity.
method Derives ℓ1-norm sparsity-promoting models, characterizes sparsity levels, and develops algorithms for selecting optimal regularization parameters. result Developed algorithms to select regularization parameters for desired sparsity levels in neural networks.
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
Paper develops a method to learn optimal sparsity-promoting regularizers for linear inverse problems.
problem Solving linear inverse problems with sparse solutions.
method Bilevel optimization framework to select an optimal synthesis operator B. result Established well-posedness and theoretical guarantees for the learning process.
Gradient descent implicitly favors group sparsity in neural networks.
problem Understanding implicit regularization in neural networks for structured sparsity.
method Novel neural reparameterization for diagonally grouped linear networks.
result Gradient descent without explicit regularization biases towards group sparsity.
New layer sparsity concept improves neural networks.
problem Improving neural network efficiency and interpretability.
method Formulated layer sparsity, introduced regularization and refitting schemes.
result Generated more compact and accurate neural networks.
Paper proposes a new sparse group k-max regularization for sparsity constraints.
problem Linear inverse problems with sparsity constraints are NP-hard.
method Sparse group k-max regularization, iterative soft thresholding algorithm.
result Approximates l0 norm more closely and enhances group-wise and in-group sparsity.
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
We propose a novel SPARsity and Clustering (SPARC) regularizer, which is a modified version of the previous octagonal shrinkage and clustering algorithm for regression (OSCAR), where, the proposed regularizer consists of a K-sparse constraint and a pair-wise ℓ∞ norm restricted on the K largest componen…
In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradi…
Fiedler regularization uses graph sparsity to improve neural network training.
problem Improving neural network training by respecting graph structure.
method Using the Fiedler value of the neural network's graph as a regularization tool.
result Fiedler regularization outperforms traditional methods like dropout and weight decay.
A new method solves l1-regularized optimization problems efficiently and sparsely.
problem l1-regularized optimization problems in machine learning.
method Orthant Based Proximal Stochastic Gradient Method (OBProx-SG)
result Promotes sparsity of solutions substantially and converges to global optimal solutions.
Paper proposes a new sparsity scheme for high-dimensional VAR models.
problem Estimation of high-dimensional VAR models with sparsity assumptions.
method Regularized estimation procedures for sparse VAR models.
result Threholding extends consistency properties of regularized estimators.
Novel regularization for Vision Transformers improves model generalization and sparsity.
problem Improving generalization and sparsity in Vision Transformers.
method Likelihood-guided variational Ising-based regularization.
result Improved generalization and sparsity in Vision Transformers.
Proposes a neural network for sparsity regularization in inverse problems using Gaussian mixture.
problem Sparsity in inverse problems with limited significant components.
method Probabilistic sparsity prior as a mixture of degenerate Gaussians, trained with neural network.
result Neural network yields lower mean square error than LASSO, group LASSO, and iterative hard thresholding.
RMDA trains structured neural networks with regularization and variance reduction.
problem Training structured neural networks with desired properties.
method RMDA algorithm for structured NNs with regularization and variance reduction.
result RMDA achieves desired structures identical to regularizer's at stationary points.
Convex optimization with sparsity-promoting convex regularization is a standard approach for estimating sparse signals in noise. In order to promote sparsity more strongly than convex regularization, it is also standard practice to employ non-convex optimization. In this paper, we take a third approach. We utilize a no…
Enhances KLR for indefinite kernels with L1-norm regularization.
problem Classifying with indefinite kernels captures more domain-specific information.
method Introduces L1-norm regularization to induce sparsity and a proximal linearized algorithm. result Superior performance in accuracy and sparsity on multiple datasets.
Sparse Bayesian Optimization (SEBO) finds interpretable configurations.
problem Optimizing black-box functions for recommendation systems while maintaining interpretability.
method Regularization-based approaches, including a differentiable relaxation for L0 regularization, and a hyperparameter-free method SEBO. result SEBO efficiently optimizes for sparsity without hyperparameters.
A new K-means method HT K-means uses ℓ0 penalty for sparsity.
problem Cluster center regularization for improved clustering performance.
method HT K-means with ℓ0 penalty for sparsity. result HT K-means outperforms other regularized K-means methods in simulations and real data. Study compares L1 and VG sparsity priors in inverse problems.
problem Sparse regularization in inverse problems with incomplete or corrupted measurements.
method Compared L1 regularization with Variational Garrote (VG), a probabilistic method approximating L0 sparsity.
result VG often achieves lower minimum generalization error and improved stability in strongly underdetermined regimes.
In this paper, we propose ℓp-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓp-norm regularized models…
Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for fu…
Improved iterative hard thresholding for faster, sparser solutions.
problem Finding sparser solutions without sacrificing runtime.
method Adaptive regularization framework applied to iterative hard thresholding.
result Returns solutions with sparsity O(sκ), improving over existing methods. TATD predicts missing entries in time-evolving tensors by exploiting temporal dependency and sparsity.
problem Predict missing entries in time-evolving tensors with temporal dependency and sparsity issues.
method TATD (Time-Aware Tensor Decomposition) integrates temporal dependency and time-varying sparsity through a smoothing regularization with Gaussian kernel and alternating optimization.
result TATD achieves state-of-the-art accuracy for decomposing temporal tensors.
The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.
problem Quantifying the trade-off between smoothness and sparsity in GCN.
method Proposes a novel SGD proximal algorithm for GCNs with an inexact operator to analyze the stability of the ℓp-regularized stochastic learning. result Establishes an explicit theoretical understanding of GCN with ℓp-regularized stochastic learning. Path regularization reveals convex optimization in deep ReLU networks.
problem Understanding the optimization landscape of deep neural networks.
method Introducing path regularization to make the training problem convex and sparsity-inducing.
result Path regularized parallel ReLU networks are a parsimonious convex model in high dimensions.
Soft diamond regularizers improve deep learning performance and sparsity.
problem Improving deep learning performance and sparsity of trained weights.
method New soft diamond synaptic weight priors based on thick-tailed symmetric alpha stable probability curves.
result Soft diamond regularizers outperform state-of-the-art methods in deep learning tasks.
High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) l…
Multiple kernel learning (MKL), structured sparsity, and multi-task learning have recently received considerable attention. In this paper, we show how different MKL algorithms can be understood as applications of either regularization on the kernel weights or block-norm-based regularization, which is more common in str…
This work extends neural networks to automatically select features by stochastically penalizing feature involvement.
problem Feature selection in machine learning models.
method Stochastic regularization to select features instead of layer weights.
result Superior efficiency compared to classical methods with minimal computational overhead.
We present a data dependent generalization bound for a large class of regularized algorithms which implement structured sparsity constraints. The bound can be applied to standard squared-norm regularization, the Lasso, the group Lasso, some versions of the group Lasso with overlapping groups, multiple kernel learning a…
We study the emergence of sparse representations in neural networks. We show that in unsupervised models with regularization, the emergence of sparsity is the result of the input data samples being distributed along highly non-linear or discontinuous manifold. We also derive a similar argument for discriminatively trai…
Paper proposes a new method to optimize deep neural networks with sparse regularization.
problem Difficulty in achieving optimal convergence rates for deep neural networks due to sparsity constraints.
method Introduces a novel penalized estimation method for sparse DNNs, resolving computational and theoretical issues.
result Establishes an oracle inequality for the excess risk of the proposed sparse-penalized DNN estimator and derives convergence rates.
In high dimensional regression settings, sparsity enforcing penalties have proved useful to regularize the data-fitting term. A recently introduced technique called screening rules propose to ignore some variables in the optimization leveraging the expected sparsity of the solutions and consequently leading to faster s…
A new method for differentiable structured sparsity improves neural network performance and sparsity.
problem Non-differentiability of structured sparsity penalties in neural networks.
method Introducing D-Gating, a differentiable approach to structured overparameterization. result The D-Gating objective converges to the L2,2/D-regularized loss and induces sparse learning dynamics. We investigate the learning rate of multiple kernel learning (MKL) with ℓ1 and elastic-net regularizations. The elastic-net regularization is a composition of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total …
New method smooths optimization for sparse regularization.
problem Non-smooth, non-convex optimization problems for sparsity.
method Overparameterization and smooth surrogate penalties.
result Surrogate objective has identical global and local minima.
Sparsity helps reduce diffusion model costs.
problem High computational costs in diffusion models.
method Introduced sparsity concept to reduce input dimensionality.
result Sparsity reduces computational complexity to intrinsic data dimension.
New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.
problem Over-parametrization leads to too many active neurons in neural networks, especially with large data.
method Investigates a nonconvex regularization method for shallow ReLU networks.
result Locally optimal networks are finite even with infinite data, maintaining approximation guarantees and network size bounds.
DNNs with L2 regularization reveal feature learning dynamics and sparsity.
problem Understanding feature learning in DNNs with L2 regularization. method Reformulating loss in terms of layerwise activations and covariances.
result Proving sparsity of local minima in L2-regularized DNNs. During the past years there has been an explosion of interest in learning methods based on sparsity regularization. In this paper, we discuss a general class of such methods, in which the regularizer can be expressed as the composition of a convex function ω with a linear function. This setting includes several metho…
Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it. A now standard method for recovering the unknown signal is to solve a convex optimization problem that enforc…
In text classification, the problem of overfitting arises due to the high dimensionality, making regularization essential. Although classic regularizers provide sparsity, they fail to return highly accurate models. On the contrary, state-of-the-art group-lasso regularizers provide better results at the expense of low s…
Develops a functional mix-of-experts model for multiclass classification.
problem Multiclass classification with univariate functional predictors.
method Functional mix-of-experts model with regularization and sparsity constraints.
result Regularized maximum likelihood estimation yields interpretable coefficient functions.
This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of…
A new filter design improves system identification accuracy.
problem Improving system identification accuracy for various system types.
method Generalized proportionate-type normalized subband adaptive filter (GPtNSAF) using least squares on subband errors with a sparsity penalty.
result GPtNSAF benefits from increasing subbands more than sparsity for quasi-sparse or dispersive systems, and both aspects are complementary for sparse systems.
We study the problem of learning a sparse linear regression vector under additional conditions on the structure of its sparsity pattern. This problem is relevant in machine learning, statistics and signal processing. It is well known that a linear regression can benefit from knowledge that the underlying regression vec…