arXiv research

Survey on minimal penalty algorithms and slope heuristics.

New approach avoids excess empirical risk in domain generalization.

PGD algorithm converges to local minima in nonconvex matrix completion.

The use of machine-learning in neuroimaging offers new perspectives in early diagnosis and prognosis of brain diseases. Although such multivariate methods can capture complex relationships in the data, traditional approaches provide irregular (l2 penalty) or scattered (l1 penalty) predictive pattern with a very limited…

Proposes an alternative invariance penalty to address domain generalization issues.

We provide investment advice for an individual who wishes to minimize her lifetime poverty, with a penalty for bankruptcy or ruin. We measure poverty via a non-negative, non-increasing function of (running) wealth. Thus, the lower wealth falls and the longer wealth stays low, the greater the penalty. This paper general…

We tackle the problem of penalty selection of regularization on the basis of the minimum description length (MDL) principle. In particular, we consider that the design space of the penalty function is high-dimensional. In this situation, the luckiness-normalized-maximum-likelihood(LNML)-minimization approach is favorab…

Global minima found for multidimensional scaling with penalties.

In this paper we consider general rank minimization problems with rank appearing in either objective function or constraint. We first establish that a class of special rank minimization problems has closed-form solutions. Using this result, we then propose penalty decomposition methods for general rank minimization pro…

As surrogate functions of $L_0$-norm, many nonconvex penalty functions have been proposed to enhance the sparse vector recovery. It is easy to extend these nonconvex penalty functions on singular values of a matrix to enhance low-rank matrix recovery. However, different from convex optimization, solving the nonconvex l…

We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerate ellipsoid. Such problems have a wide range of applications in data science, where the objective is used for inducing spars…

Rejoinder on slope heuristics for model selection in regression.

In this paper we consider sparse approximation problems, that is, general $l_0$ minimization problems with the $l_0$-"norm" of a vector being a part of constraints or objective function. In particular, we first study the first-order optimality conditions for these problems. We then propose penalty decomposition (PD) me…

Paper designs a penalty for model order selection using information criteria.

PPO-B improves sampling efficiency by using a logarithmic barrier method.

This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression, spline smoothing or locally weighted regression, and the choice of a kernel in multiple kern…

This study proves local stability of SGP μ-WGAN and shows penalizing data or sample manifold is key.

Recently, there has been focus on penalized log-likelihood covariance estimation for sparse inverse covariance (precision) matrices. The penalty is responsible for inducing sparsity, and a very common choice is the convex $l_1$ norm. However, the best estimator performance is not always achieved with this penalty. The …

Algorithm minimizes loss and constraint violations in online convex optimization with smooth penalties.

In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…

New method improves signal reconstruction with nonconvex penalties and parameter control.

Paper proposes efficient algorithms for designing SLOPE penalty sequences.

Estimation in generalized linear models (GLM) is complicated by the presence of constraints. One can handle constraints by maximizing a penalized log-likelihood. Penalties such as the lasso are effective in high dimensions, but often lead to unwanted shrinkage. This paper explores instead penalizing the squared distanc…

Optimal reinsurance minimizes expected discounted penalty in a Cramer-Lundberg model.

A new spline method for manifold learning using Hessian-based curvature penalties.

This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of…

Proposes spred for solving $L_1$ penalty with SGD.

In this paper we study nonconvex penalization using Bernstein functions whose first-order derivatives are completely monotone. The Bernstein function can induce a class of nonconvex penalty functions for high-dimensional sparse estimation problems. We derive a thresholding function based on the Bernstein penalty and di…

The paper surveys methods to approximate non-negative matrices using lower-dimensional factors.

The paper develops a classification method using penalties on feature selection for high-dimensional data.

In this paper we consider $l_0$ regularized convex cone programming problems. In particular, we first propose an iterative hard thresholding (IHT) method and its variant for solving $l_0$ regularized box constrained convex programming. We show that the sequence generated by these methods converges to a local minimizer.…

New method balances performance and cost in identifying best arm.

Optimal subset selection for hypothesis testing with penalties.

Paper develops algorithms for sparse linear regression with generalized elastic net penalty.

We investigate properties of estimators obtained by minimization of U-processes with the Lasso penalty in high-dimensional settings. Our attention is focused on the ranking problem that is popular in machine learning. It is related to guessing the ordering between objects on the basis of their observed predictors. We p…

Paper introduces stability in model averaging and proposes a L2-penalty method.

Universal audio perturbations fool multiple classification models.

Random Fourier features model reconstructs wind fields from sparse measurements.

PAIR optimizes machine learning models to generalize better to out-of-distribution data.

The popular cubic smoothing spline estimate of a regression function arises as the minimizer of the penalized sum of squares $\sum_j(Y_j - μ(t_j))^2 + λ\int_a^b [μ"(t)]^2 dt$, where the data are $t_j,Y_j$, $j=1,..., n$. The minimization is taken over an infinite-dimensional function space, the space of all functions wi…

A new framework selects information sources to test hypotheses robustly, even with misclassifications.

Proposes a method to constrain singular values of convolutional kernels in neural networks.

Data-driven optimization improves mean-variance portfolios by penalizing norms.

Two sparsity-aware NSAF algorithms improve sparse system identification with lower complexity.

We consider the problem of binary classification where one can, for a particular cost, choose not to classify an observation. We present a simple proof for the oracle inequality for the excess risk of structural risk minimizers using a lasso type penalty.

Proposes a method to control model complexity in neural network optimization.

Significant attention has been given to minimizing a penalized least squares criterion for estimating sparse solutions to large linear systems of equations. The penalty is responsible for inducing sparsity and the natural choice is the so-called $l_0$ norm. In this paper we develop a Momentumized Iterative Shrinkage Th…

AutoTransfer improves subject transfer learning for biosignal datasets.