A neural network solves logistic regression with ℓ1 regularization efficiently.
problem Efficiently solving logistic regression with ℓ1 regularization due to non-differentiability of ℓ1 norm. method A simple projection neural network that avoids auxiliary variables and smooth approximations.
result The neural network converges to a solution of the problem with any initial value and outperforms existing methods.
The paper analyzes ℓ1-LinR for Ising model selection using statistical mechanics.
problem Model selection consistency of ℓ1-LinR for Ising models. method Replica method from statistical mechanics, ℓ1-regularized linear regression (ℓ1-LinR). result Model selection consistency with sample complexity $M=\mathcal{O}\left(\log N
ight)$.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. Unified framework for sparse logistic regression with nonconvex regularization.
problem Sparse logistic regression with nonconvex regularization.
method Unified framework, line search criteria for nonconvex terms.
result Effective classification and feature selection at lower computational cost.
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that ℓ1-regularized least squares regression can accurately estimate a sparse linear model from n nois…
New method learns high-dimensional Poisson DAG models from observational data.
problem Learning high-dimensional Poisson DAG models from observational data without strong assumptions.
method Decouples ordering estimation and parent search using ℓ1-regularized regression and mean-variance relationship. result Sample size n=Ω(d2log9p) sufficient for polynomial time algorithm to recover true directed graph. A new algorithm speeds up EEG source localization using ℓ1 regularization.
problem Challenging inverse problem in mapping EEG readings to brain activity.
method Formulated as a graphical generalized elastic net inverse problem, solved with a variable projected algorithm (VPAL).
result VPAL provides faster and more accurate EEG source localization compared to existing methods.
Regularized linear regression improves binary classification performance, especially with ridge and ℓ1 regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while ℓ1 regularization can induce sparsity and ℓ∞ regularization can concentrate weights to two values. This paper shows how to use ℓ1 regularization effectively in training sparse CNNs.
problem Why ℓ1 regularization hasn't been used in sparse deep learning models like CNNs. method Demonstrated that SGD is not suitable for ℓ1 regularization and replaced it with a new training algorithm based on regularized dual averaging (RDA). result Achieved state-of-the-art sparsity for CNNs using RDA with ℓ1 regularization, achieving 95% sparsity for ResNet18 on CIFAR-10. Feature selection is a standard approach to understanding and modeling high-dimensional classification data, but the corresponding statistical methods hinge on tuning parameters that are difficult to calibrate. In particular, existing calibration schemes in the logistic regression framework lack any finite sample guara…
In this paper we consider the task of estimating the non-zero pattern of the sparse inverse covariance matrix of a zero-mean Gaussian random vector from a set of iid samples. Note that this is also equivalent to recovering the underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We present two no…
New neural network training method uses bi-fidelity data to reduce errors.
problem Training neural networks with limited high-fidelity data.
method Bi-fidelity ℓ1-regularization strategies. result Bi-fidelity ℓ1-regularization reduces errors by one order of magnitude. New GLS estimator handles high-dimensional data with autocorrelated errors.
problem High-dimensional regressions with autocorrelated errors.
method LASSO regression, autoregressive model fitting, and whitening.
result The method outperforms unadjusted LASSO in estimating errors driven by autoregressive processes.
In a recent work (arXiv:0910.2517), for nonlinear models with sparse underlying linear structures, we studied the error bounds of ℓ0-regularized estimation. In this note, we show that ℓ1-regularized estimation in some important cases can achieve the same order of error bounds as those in the aforementioned …
Method improves SINDy for noisy nonlinear systems.
problem Recover nonlinear dynamical systems from noisy data.
method Reweighted ℓ1-regularized least squares. result Improved accuracy and robustness in noisy conditions.
We develop an approximate formula for evaluating a cross-validation estimator of predictive likelihood for multinomial logistic regression regularized by an ℓ1-norm. This allows us to avoid repeated optimizations required for literally conducting cross-validation; hence, the computational time can be significantl…
We investigate the learning rate of multiple kernel learning (MKL) with ℓ1 and elastic-net regularizations. The elastic-net regularization is a composition of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total …
Sparse regularization such as ℓ1 regularization is a quite powerful and widely used strategy for high dimensional learning problems. The effectiveness of sparse regularization has been supported practically and theoretically by several studies. However, one of the biggest issues in sparse regularization is that i…
Proposes a method to adapt models in nonstationary environments using ℓ1 regularization.
problem Adapting models to nonstationary environments in machine learning.
method Integrates ℓ1 regularization of differences between source and target parameters.
result Effective balance of stability and plasticity in model adaptation.
Multi-task feature learning aims to identity the shared features among tasks to improve generalization. It has been shown that by minimizing non-convex learning models, a better solution than the convex alternatives can be obtained. Therefore, a non-convex model based on the capped-ℓ1,ℓ1 regularization wa…
In this paper we consider the problem of grouped variable selection in high-dimensional regression using ℓ1−ℓq regularization (1≤q≤∞), which can be viewed as a natural generalization of the ℓ1−ℓ2 regularization (the group Lasso). The key condition is that the dimensionality pn can…
We consider the problem of estimating the topology of spatial interactions in a discrete state, discrete time spatio-temporal graphical model where the interactions affect the temporal evolution of each agent in a network. Among other models, the susceptible, infected, recovered (SIR) model for interaction events fal…
We analyze coresets for regularized regression problems and propose a modified lasso that yields smaller coresets.
problem Analyzing coresets for regularized regression problems.
method Examined coresets for ridge regression and proposed a modified lasso problem.
result No coreset for regularized regression can be smaller than the unregularized version when reqs. New method uses nuclear and ℓ1 penalties for matrix regression, improving brain disorder detection.
problem Modeling high-dimensional matrix predictors with binary responses.
method Convex optimization with ADMM for low-rank and sparse structures.
result Effective in identifying brain disorder-related connectivity patterns.
The paper improves count data regression models for overdispersed data.
problem Improving regression models for overdispersed count data.
method Double ℓ1-regularized negative binomial regressions. result Oracle inequalities and consistency for Lasso estimators of partial regression coefficients.
SAGA is a fast incremental gradient method on the finite sum problem and its effectiveness has been tested on a vast of applications. In this paper, we analyze SAGA on a class of non-strongly convex and non-convex statistical problem such as Lasso, group Lasso, Logistic regression with ℓ1 regularization, linear r…
Recently, Mahoney and Orecchia demonstrated that popular diffusion-based procedures to compute a quick \emph{approximation} to the first nontrivial eigenvector of a data graph Laplacian \emph{exactly} solve certain regularized Semi-Definite Programs (SDPs). In this paper, we extend that result by providing a statistica…
Study shows OWL-regularized regression, including OSCAR, is vulnerable to adversarial perturbations.
problem Vulnerability of sparse regression models with strongly correlated covariates to adversarial perturbations.
method Formulated adversarial attack as an optimization problem and analyzed OSCAR's robustness.
result Regression performance of grouping strongly correlated features can be severely degraded under adversarial settings.
In this paper, we consider stochastic dual coordinate (SDCA) {\em without} strongly convex assumption or convex assumption. We show that SDCA converges linearly under mild conditions termed restricted strong convexity. This covers a wide array of popular statistical models including Lasso, group Lasso, and logistic reg…
The optimization of the variance supplemented by a budget constraint and an asymmetric ℓ1 regularizer is carried out analytically by the replica method borrowed from the theory of disordered systems. The asymmetric regularizer allows us to penalize short and long positions differently, so the present treatment in…
State-of-the-art subspace clustering methods are based on expressing each data point as a linear combination of other data points while regularizing the matrix of coefficients with ℓ1, ℓ2 or nuclear norms. ℓ1 regularization is guaranteed to give a subspace-preserving affinity (i.e., there are no conne…
Study sparse function recovery from indirect noisy observations using ℓ1-regularization.
problem Recovering sparse functions from indirect, noisy observations.
method Proposes an ℓ1-regularized empirical risk minimizer and analyzes its statistical properties. result Established almost-sure consistency and derived high-probability convergence rates in prediction and ℓ1 norms. We propose a novel, efficient approach for distributed sparse learning in high-dimensions, where observations are randomly partitioned across machines. Computationally, at each round our method only requires the master machine to solve a shifted ell_1 regularized M-estimation problem, and other workers to compute the g…
We consider the problem of learning overcomplete dictionaries in the context of sparse coding, where each sample selects a sparse subset of dictionary elements. Our main result is a strategy to approximately recover the unknown dictionary using an efficient algorithm. Our algorithm is a clustering-style procedure, wher…
A new screening rule improves lasso solving speed.
problem Efficiently solving lasso problems with high correlation.
method Uses second-order information from the Hessian to screen predictors.
result Outperforms alternatives on simulated and real data.
Trans-Ising combines auxiliary datasets to estimate high-dimensional Ising models.
problem Limited target sample sizes and difficulty in using auxiliary binary datasets of unknown relevance.
method Trans-Ising uses a loss-based source screening rule and a two-stage estimation procedure.
result Trans-Ising achieves lower estimation errors than target-only estimation and naive data pooling.
We analyze learning curves of RF models with convex regularization and derive precise asymptotic expressions.
problem Understanding the learning curves of RF models with general convex regularization.
method Novel multi-level application of the convex Gaussian min max theorem (CGMT) to compute precise asymptotic expressions.
result Precise asymptotic expressions for learning curves of RF models with separable strongly convex regularization or ℓ1 regularization. Unified approach for estimating causal contrasts robust to model misspecification.
problem Estimating causal contrasts in non-parametric models with potential misspecification.
method A unifying approach using doubly-robust ℓ1 regularized estimation with cross-fitting and sample splitting. result Estimators are robust to misspecification of either nuisance function, ensuring consistent and normal asymptotic properties.
This paper improves neural network compression by using robust low-rank approximations.
problem Neural network compression sensitivity to outliers.
method Introduces robust low-rank approximations using ℓp norms (for p∈[1,2]) and provides efficient algorithms. result Achieves up to 28% compression with minimal accuracy loss compared to existing methods.
AEN-SAEs address feature starvation in sparse autoencoders by stabilizing the geometric alignment of sparse coding.
problem Feature starvation in sparse autoencoders, leading to unstable and misaligned representations.
method Adaptive Elastic Net SAEs (AEN-SAEs) combine ℓ2 and ℓ1 terms to stabilize the sparse coding map and control feature interactions. result AEN-SAEs mitigate feature starvation without heuristic resampling, maintaining competitive reconstruction abilities.
We study parameter estimation and asymptotic inference for sparse nonlinear regression. More specifically, we assume the data are given by y=f(x⊤β∗)+ε, where f is nonlinear. To recover β∗, we propose an ℓ1-regularized least-squares estimator. Unlike classical linear regression, the correspondin…
The paper examines how to protect LASSO-based feature selection from adversarial attacks.
problem Adversarial attacks on LASSO-based feature selection.
method Formulated as a bi-level optimization problem, reformulated LASSO with linear inequality constraints, solved using interior-point method, and modified using projected gradient descent.
result Demonstrated the effectiveness of the proposed method in protecting LASSO-based feature selection from adversarial attacks.
Study supports recovery of PDEs from noisy data using a specific regularization method.
problem Support recovery of PDEs from a single noisy trajectory.
method Applying ℓ1-regularized Pseudo-Least Squares model to a given data set.
result Support of ℓ1-c coefficients asymptotically converges to the true signed-support of the PDE.
Study examines stability of image-reconstruction algorithms using variational regularization.
problem Stability and robustness of image-reconstruction algorithms in medical imaging.
method Review and novel stability results for ℓp-regularized linear inverse problems, focusing on p∈(1,∞). result Guarantees Lipschitz continuity for small p and Hölder continuity for larger p in Lp(Ω) function spaces. The paper analyzes how gradient descent implicitly regularizes solutions in overparameterized neural networks, revealing depth-dependent regularization effects.
problem Understanding implicit regularization in overparameterized linear neural networks for regression problems.
method Analyzing the approximation error between gradient flow limit points and ℓ1-minimization solutions, deriving tight upper and lower bounds. result The approximation error decreases linearly for D≥3 and at a slower rate for D=2, linked to null space property constants. We introduce a recursive adaptive group lasso algorithm for real-time penalized least squares prediction that produces a time sequence of optimal sparse predictor coefficient vectors. At each time index the proposed algorithm computes an exact update of the optimal ℓ1,∞-penalized recursive least squares (R…
FasTR efficiently solves sparse and unit-rank tensor regression problems.
problem Sparse and unit-rank tensor regression problems in tensor data analysis.
method FasTR decomposes tensor coefficients into component vectors and estimates each with ℓ1 regularized regression, solving in parallel. result FasTR computes better solutions faster than baseline models.