arXiv research

New theorem for generalized group sparsity improves consistency and convergence rates.

The paper studies how regularization parameters affect sparsity in deep neural networks.

Overfitting is one of the most common problems when training deep neural networks on comparatively small datasets. Here, we demonstrate that neural network activation sparsity is a reliable indicator for overfitting which we utilize to propose novel targeted sparsity visualization and regularization strategies. Based o…

Paper develops a method to learn optimal sparsity-promoting regularizers for linear inverse problems.

Gradient descent implicitly favors group sparsity in neural networks.

New layer sparsity concept improves neural networks.

Paper proposes a new sparse group k-max regularization for sparsity constraints.

Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…

We propose a novel SPARsity and Clustering (SPARC) regularizer, which is a modified version of the previous octagonal shrinkage and clustering algorithm for regression (OSCAR), where, the proposed regularizer consists of a $K$-sparse constraint and a pair-wise $\ell_{\infty}$ norm restricted on the $K$ largest componen…

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradi…

A new method solves l1-regularized optimization problems efficiently and sparsely.

Paper proposes a new sparsity scheme for high-dimensional VAR models.

Novel regularization for Vision Transformers improves model generalization and sparsity.

Proposes a neural network for sparsity regularization in inverse problems using Gaussian mixture.

RMDA trains structured neural networks with regularization and variance reduction.

Convex optimization with sparsity-promoting convex regularization is a standard approach for estimating sparse signals in noise. In order to promote sparsity more strongly than convex regularization, it is also standard practice to employ non-convex optimization. In this paper, we take a third approach. We utilize a no…

Enhances KLR for indefinite kernels with $L_1$-norm regularization.

Sparse Bayesian Optimization (SEBO) finds interpretable configurations.

A new $K$-means method HT $K$-means uses $\ell_0$ penalty for sparsity.

Study compares L1 and VG sparsity priors in inverse problems.

In this paper, we propose $\ell_p$-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the $\ell_p$-norm regularized models…

Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for fu…

Improved iterative hard thresholding for faster, sparser solutions.

TATD predicts missing entries in time-evolving tensors by exploiting temporal dependency and sparsity.

The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.

Path regularization reveals convex optimization in deep ReLU networks.

Soft diamond regularizers improve deep learning performance and sparsity.

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) l…

Multiple kernel learning (MKL), structured sparsity, and multi-task learning have recently received considerable attention. In this paper, we show how different MKL algorithms can be understood as applications of either regularization on the kernel weights or block-norm-based regularization, which is more common in str…

This work extends neural networks to automatically select features by stochastically penalizing feature involvement.

We present a data dependent generalization bound for a large class of regularized algorithms which implement structured sparsity constraints. The bound can be applied to standard squared-norm regularization, the Lasso, the group Lasso, some versions of the group Lasso with overlapping groups, multiple kernel learning a…

We study the emergence of sparse representations in neural networks. We show that in unsupervised models with regularization, the emergence of sparsity is the result of the input data samples being distributed along highly non-linear or discontinuous manifold. We also derive a similar argument for discriminatively trai…

Paper proposes a new method to optimize deep neural networks with sparse regularization.

In high dimensional regression settings, sparsity enforcing penalties have proved useful to regularize the data-fitting term. A recently introduced technique called screening rules propose to ignore some variables in the optimization leveraging the expected sparsity of the solutions and consequently leading to faster s…

A new method for differentiable structured sparsity improves neural network performance and sparsity.

We investigate the learning rate of multiple kernel learning (MKL) with $\ell_1$ and elastic-net regularizations. The elastic-net regularization is a composition of an $\ell_1$-regularizer for inducing the sparsity and an $\ell_2$-regularizer for controlling the smoothness. We focus on a sparse setting where the total …

New method smooths optimization for sparse regularization.

Sparsity helps reduce diffusion model costs.

New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

During the past years there has been an explosion of interest in learning methods based on sparsity regularization. In this paper, we discuss a general class of such methods, in which the regularizer can be expressed as the composition of a convex function $ω$ with a linear function. This setting includes several metho…

Inverse problems and regularization theory is a central theme in contemporary signal processing, where the goal is to reconstruct an unknown signal from partial indirect, and possibly noisy, measurements of it. A now standard method for recovering the unknown signal is to solve a convex optimization problem that enforc…

In text classification, the problem of overfitting arises due to the high dimensionality, making regularization essential. Although classic regularizers provide sparsity, they fail to return highly accurate models. On the contrary, state-of-the-art group-lasso regularizers provide better results at the expense of low s…

Develops a functional mix-of-experts model for multiclass classification.

This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of…

A new filter design improves system identification accuracy.

We study the problem of learning a sparse linear regression vector under additional conditions on the structure of its sparsity pattern. This problem is relevant in machine learning, statistics and signal processing. It is well known that a linear regression can benefit from knowledge that the underlying regression vec…

AGS-CL selectively updates penalties based on node importance for continual learning.