We study the problem of learning high dimensional regression models regularized by a structured-sparsity-inducing penalty that encodes prior structural information on either input or output sides. We consider two widely adopted types of such penalties as our motivating examples: 1) overlapping group lasso penalty, base…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,932 papers · 148 categories
Trend · papers per month
48 results for “sparsity-inducing penalties”
We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlappin…
Regularization vs. Relaxation: A conic optimization perspective of statistical variable selectioncs.LG
Variable selection is a fundamental task in statistical data analysis. Sparsity-inducing regularization methods are a popular class of methods that simultaneously perform variable selection and model estimation. The central problem is a quadratic optimization problem with an l0-norm penalty. Exactly enforcing the l0-no…
This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of…
Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast…
HALO learns to prune neural networks by adaptively shrinking weights.
problem Sparsity and model size in deep neural networks.
method Bayesian hierarchical models and trainable parameters for adaptive sparsification.
result HALO learns to create highly sparse networks with significant performance gains.
In this paper, the estimation problem for sparse reduced rank regression (SRRR) model is considered. The SRRR model is widely used for dimension reduction and variable selection with applications in signal processing, econometrics, etc. The problem is formulated to minimize the least squares loss with a sparsity-induci…
Gradient descent on measure positions and weights solves sparse optimization problems.
problem Sparse optimization of measures with sparsity-inducing penalties.
method Discretize measure, run non-convex gradient descent on positions and weights.
result Global optimization with complexity scaling as log(1/ε), improving over convex methods.
Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection. However, it is quite challenging to solve the nonconvex penalized SVMs due to their nondifferentiability, nonsmoothness and nonconve…
This paper proposes a new interpretation of sparse penalties such as the elastic-net and the group-lasso. Beyond providing a new viewpoint on these penalization schemes, our approach results in a unified optimization strategy. Our experiments demonstrate that this strategy, implemented on the elastic-net, is computatio…
One-bit measurements widely exist in the real world, and they can be used to recover sparse signals. This task is known as the problem of learning halfspaces in learning theory and one-bit compressive sensing (1bit-CS) in signal processing. In this paper, we propose novel algorithms based on both convex and nonconvex s…
In this paper we propose and study a family of sparsity-inducing penalty functions. Since the penalty functions are related to the kinetic energy in special relativity, we call them \emph{kinetic energy plus} (KEP) functions. We construct the KEP function by using the concave conjugate of a -distance function and …
This work addresses the issue of large covariance matrix estimation in high-dimensional statistical analysis. Recently, improved iterative algorithms with positive-definite guarantee have been developed. However, these algorithms cannot be directly extended to use a nonconvex penalty for sparsity inducing. Generally, a…
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural prop…
We consider a regularized least squares problem, with regularization by structured sparsity-inducing norms, which extend the usual and the group lasso penalty, by allowing the subsets to overlap. Such regularizations lead to nonsmooth problems that are difficult to optimize, and we propose in this paper a suit…
Study develops a method to select penalty parameters for sparse neural networks without cross-validation.
problem Selecting optimal penalty parameters for sparse neural networks without cross-validation.
method Established theoretical foundation to bound the infinite norm of the gradient of the loss function at zero.
result Proposed method effectively selects penalty parameters for sparse neural networks.
A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networksstat.ML
ANN learner finds sparse needles in nonlinear haystacks with high probability.
problem Finding sparse features in nonlinear data.
method Generalized LASSO penalty with stochastic gradient descent, warm-start algorithm.
result Phase transition in ANN learner's ability to find needles, better than other learners.
Non-convex sparsity-inducing penalties have recently received considerable attentions in sparse learning. Recent theoretical investigations have demonstrated their superiority over the convex counterparts in several sparse learning settings. However, solving the non-convex optimization problems associated with non-conv…
We consider a class of sparse learning problems in high dimensional feature space regularized by a structured sparsity-inducing norm which incorporates prior knowledge of the group structure of the features. Such problems often pose a considerable challenge to optimization algorithms due to the non-smoothness and non-s…
FIRE extracts interpretable rules from tree ensembles.
problem Building sparse and interpretable rule sets from tree ensembles.
method Optimization-based framework with fusion regularization and sparsity-inducing penalties.
result FIRE outperforms state-of-the-art rule ensemble algorithms.
In this paper, we propose an unifying view of several recently proposed structured sparsity-inducing norms. We consider the situation of a model simultaneously (a) penalized by a set- function de ned on the support of the unknown parameter vector which represents prior knowledge on supports, and (b) regularized in Lp-n…
Sparse regression models are increasingly prevalent due to their ease of interpretability and superior out-of-sample performance. However, the exact model of sparse regression with an constraint restricting the support of the estimators is a challenging (\NP-hard) non-convex optimization problem. In this paper…
Sparse Estimation using Bayesian Hierarchical Prior Modeling for Real and Complex Linear Modelsstat.ML
In sparse Bayesian learning (SBL), Gaussian scale mixtures (GSMs) have been used to model sparsity-inducing priors that realize a class of concave penalty functions for the regression task in real-valued signal models. Motivated by the relative scarcity of formal tools for SBL in complex-valued models, this paper propo…
This paper studies simultaneous feature selection and extraction in supervised and unsupervised learning. We propose and investigate selective reduced rank regression for constructing optimal explanatory factors from a parsimonious subset of input features. The proposed estimators enjoy sharp oracle inequalities, and w…
We consider the problem of estimating a sparse multi-response regression function, with an application to expression quantitative trait locus (eQTL) mapping, where the goal is to discover genetic variations that influence gene-expression levels. In particular, we investigate a shrinkage technique capable of capturing a…
Multi-task learning has shown to significantly enhance the performance of multiple related learning tasks in a variety of situations. We present the fused logistic regression, a sparse multi-task learning approach for binary classification. Specifically, we introduce sparsity inducing penalties over parameter differenc…
New screening rules speed up optimal design calculations.
problem Optimal experimental design problems with sparse solutions.
method Quadratic Lasso reformulation and safe screening rules.
result Significantly faster computation compared to existing methods.
In this paper we discuss Bayesian nonconvex penalization for sparse learning problems. We explore a nonparametric formulation for latent shrinkage parameters using subordinators which are one-dimensional Lévy processes. We particularly study a family of continuous compound Poisson subordinators and a family of discrete…
SpINNEr uses matrix regression to analyze brain connectivity, improving accuracy over other methods.
problem Analyzing multi-dimensional data like brain imaging arrays using traditional scalar regression methods.
method SpINNEr applies matrix regression with nuclear norm and lasso norms to encourage low rank and sparse solutions.
result SpINNEr outperforms other methods in estimating brain connectivity, especially in well-connected regions.
Neural Granger Causalitystat.ML
While most classical approaches to Granger causality detection assume linear dynamics, many interactions in real-world applications, like neuroscience and genomics, are inherently nonlinear. In these cases, using linear models may lead to inconsistent estimation of Granger causal interactions. We propose a class of non…
SIC measures dependency between variables, promoting feature selection.
problem Measuring and selecting features in high-dimensional data.
method Gradient regularized Integral Probability Metric (IPM) with sparsity inducing penalties.
result SIC can be used for reliable and interpretable feature selection.
TSInsight improves interpretability of deep time-series models.
problem Lack of interpretability methods for time-series data.
method Attach auto-encoder to classifier with sparsity-inducing norm, fine-tune based on gradients and reconstruction penalty.
result TSInsight effectively boosts interpretability of deep time-series models.
Bayesian VAR model discovers Granger causality with uncertainty-aware binary graphs.
problem Discovering Granger causal relations from multivariate time-series data.
method Bayesian Vector AutoRegression with factorised Granger-Causal Graphs.
result Our method achieves better performance, especially in low-data regimes.
We consider the problem of sparse variable selection in nonparametric additive models, with the prior knowledge of the structure among the covariates to encourage those variables within a group to be selected jointly. Previous works either study the group sparsity in the parametric setting (e.g., group lasso), or addre…
Efficiently solves Elastic Net in high dimensions with Newton method.
problem Feature selection in high-dimensional data with non-negligible collinearity.
method Semi-smooth Newton Augmented Lagrangian Method.
result Significantly reduces computational cost compared to competitors.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
This work extends neural networks to automatically select features by stochastically penalizing feature involvement.
problem Feature selection in machine learning models.
method Stochastic regularization to select features instead of layer weights.
result Superior efficiency compared to classical methods with minimal computational overhead.
Iteratively reweighted algorithm is a popular algorithm for solving a large class of optimization problems whose objective is the sum of a Lipschitz differentiable loss function and a possibly nonconvex sparsity inducing regularizer. In this paper, motivated by the success of extrapolation techniques in accele…
Regularised canonical correlation analysis was recently extended to more than two sets of variables by the multiblock method Regularised generalised canonical correlation analysis (RGCCA). Further, Sparse GCCA (SGCCA) was proposed to address the issue of variable selection. However, for technical reasons, the variable …
New method links covariates to CTMCs using RKHS, improving state transitions modeling.
problem Traditional multistate models rely on linear relationships, limiting flexibility.
method Nonparametric approach using RKHS, with Frequentist and Bayesian versions.
result Effective in identifying nonlinear transition functions and predicting long-term behaviors.
Improves robustness of information bottleneck framework with sparsity-inducing prior.
problem Fixed-dimensional priors restrict flexibility and restrict robustness.
method Sparsity-inducing spike-slab categorical prior that learns dimension distribution per data point.
result Improves accuracy and robustness compared to traditional priors and other methods.
New method smooths optimization for sparse regularization.
problem Non-smooth, non-convex optimization problems for sparsity.
method Overparameterization and smooth surrogate penalties.
result Surrogate objective has identical global and local minima.
Solves complex machine learning problems with IRW method.
problem Problems with intractable sparsity-inducing norms in machine learning.
method Iteratively Re-Weighted (IRW) method with convergence guarantee.
result IRW method significantly outperforms alternative methods in robust feature selection.
Existing methods for sparse channel estimation typically provide an estimate computed as the solution maximizing an objective function defined as the sum of the log-likelihood function and a penalization term proportional to the l1-norm of the parameter of interest. However, other penalization terms have proven to have…
We consider the empirical risk minimization problem for linear supervised learning, with regularization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on certain subsets of variables, extending the usual -norm and the group -norm by allowing the subsets to overlap. T…
Paper develops sparse learning for heavy-tailed time series with locally stationary dynamics.
problem Sparse learning for high-dimensional heavy-tailed locally stationary time series.
method Additive modeling with kernel smoothing, sparsity-inducing penalized estimation.
result Prediction-error bounds and convergence rates for different sparsity structures.
Given a multivariate data set, sparse principal component analysis (SPCA) aims to extract several linear combinations of the variables that together explain the variance in the data as much as possible, while controlling the number of nonzero loadings in these combinations. In this paper we consider 8 different optimiz…
DICCA maps multi-view data into a shared latent space with interpretable components.
problem Learning from multiple related but distinct data views.
method DICCA extends CCA to deep generative networks and uses sparsity-inducing priors for interpretability.
result DICCA effectively disentangles shared and view-specific variations in multi-view data.