Improved penalty-based methods for bilevel optimization with reduced complexity.
problem Suboptimal complexity in solving bilevel optimization problems with large penalty terms.
method Novel penalty reformulation that decouples upper and lower-level variables, enabling larger step sizes and reduced iteration complexity.
result PBGD-Free algorithm that avoids inner loops for coupled constraint BLO problems, with reduced iteration complexity.
Efficient cross-validation for multi-penalty ridge regression.
problem Optimizing penalties in cross-validation for multi-penalty ridge regression.
method Developed a computationally efficient formula for the multi-penalty, sample-weighted hat-matrix.
result Significant speed-up in cross-validation for ridge regression models.
We consider high-dimensional binary classification by sparse logistic regression. We propose a model/feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the non-asymptotic bounds for the resulting misclassification excess risk. The bounds can be reduc…
Determining how to appropriately select the tuning parameter is essential in penalized likelihood methods for high-dimensional data analysis. We examine this problem in the setting of penalized likelihood methods for generalized linear models, where the dimensionality of covariates p is allowed to increase exponentiall…
The use of machine-learning in neuroimaging offers new perspectives in early diagnosis and prognosis of brain diseases. Although such multivariate methods can capture complex relationships in the data, traditional approaches provide irregular (l2 penalty) or scattered (l1 penalty) predictive pattern with a very limited…
SP-SPCA improves sparse PCA by adaptively adjusting variable penalties, enhancing interpretability and stability.
problem Poor interpretability and variable redundancy in PCA for high-dimensional data.
method Introduces a single equilibrium parameter to adaptively adjust variable penalties in the L2 regularization framework.
result Consistently outperforms standard sparse PCA methods in identifying sparse loading patterns and preserving cumulative variance.
New single-loop algorithm tackles weakly convex constraints in stochastic optimization.
problem Optimization with weakly convex constraints in machine learning.
method Single-loop penalty-based stochastic algorithm using hinge-based penalty.
result Achieves state-of-the-art complexity for finding approximate KKT solutions.
In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…
Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection. However, it is quite challenging to solve the nonconvex penalized SVMs due to their nondifferentiability, nonsmoothness and nonconve…
New findings show a balance between data fit and complexity in kernel hyperparameters.
problem Overcorrelation due to reparametrization of kernel hyperparameters.
method Reparametrization of kernel hyperparameters and analysis of marginal likelihood.
result Data fit term influences all other kernel hyperparameters, not just the complexity penalty.
Unified analysis of multi-task functional linear regression with manifold and composite penalties.
problem Estimating slope functions from functional data with multi-task learning.
method Penalized splines with manifold constraint and composite quadratic penalty.
result Unified convergence upper bound and phase transition behaviors for estimators.
A new method reduces bias in adaptive Lasso estimates.
problem Bias in adaptive Lasso estimates.
method Proximal gradient approach to learn penalty coefficients as decision variables.
result Reduces bias in estimates and encourages arbitrary sparsity structure.
Two new regularization methods improve neural network performance and complexity control.
problem Improving neural network performance and complexity control with correlated or high-dimensional features.
method Two regularization strategies: covariance-aware ridge and covariance-aware lasso.
result Improves predictive performance and complexity control over standard penalties.
In the multiple changepoint setting, various search methods have been proposed which involve optimising either a constrained or penalised cost function over possible numbers and locations of changepoints using dynamic programming. Such methods are typically computationally intensive. Recent work in the penalised optimi…
RiskNet predicts penalties in unreliable communication networks using GNNs.
problem Predicting penalties in networks with unreliable resources.
method Graph Neural Network (GNN) based approach trained on random graphs.
result Precisely models penalties across various network topologies.
New method solves complex bilevel optimization problems.
problem Challenging bilevel optimization problems in machine learning.
method Penalty-based bilevel gradient descent (PBGD) algorithm.
result PBGD algorithm converges for constrained bilevel problems.
We formulate a principle for classification with the knowledge of the marginal distribution over the data points (unlabeled data). The principle is cast in terms of Tikhonov style regularization where the regularization penalty articulates the way in which the marginal density should constrain otherwise unrestricted co…
Accelerated gradient method tackles nonconvex penalties in sparse learning.
problem Optimizing nonconvex penalties in sparse statistical learning.
method Generalized Nesterov's accelerated gradient method with hyperparameter optimization.
result Convergence can be made considerably faster with optimal hyperparameters.
Semidefinite programs (SDP) are important in learning and combinatorial optimization with numerous applications. In pursuit of low-rank solutions and low complexity algorithms, we consider the Burer--Monteiro factorization approach for solving SDPs. We show that all approximate local optima are global optima for the pe…
A method of simultaneously optimizing both the structure of neural networks and the connection weights in a single training loop can reduce the enormous computational cost of neural architecture search. We focus on the probabilistic model-based dynamic neural network structure optimization that considers the probabilit…
In this paper, we study the Nystr{ö}m type subsampling for large scale kernel methods to reduce the computational complexities of big data. We discuss the multi-penalty regularization scheme based on Nystr{ö}m type subsampling which is motivated from well-studied manifold regularization schemes. We develop a theoretica…
In sparse Bayesian learning (SBL), Gaussian scale mixtures (GSMs) have been used to model sparsity-inducing priors that realize a class of concave penalty functions for the regression task in real-valued signal models. Motivated by the relative scarcity of formal tools for SBL in complex-valued models, this paper propo…
GTMs model complex multivariate data with varying conditional independencies.
problem Modeling multivariate data with intricate marginals and complex dependency structures.
method Semiparametric approach using penalized splines and lasso regularization.
result GTMs accurately learn complex dependencies and identify conditional independencies.
New methods solve complex optimization problems without strong convexity assumptions.
problem Complex bilevel optimization problems with minimax lower-level structures.
method Penalty-based first-order methods for bilevel minimax optimization.
result Achieves ε-KKT point with improved oracle complexity. Zeroth-order (a.k.a, derivative-free) methods are a class of effective optimization methods for solving complex machine learning problems, where gradients of the objective functions are not available or computationally prohibitive. Recently, although many zeroth-order methods have been developed, these approaches still…
Paper proposes SMO for solving bilevel optimization problems efficiently.
problem Solving bilevel optimization problems with nonsmooth convex lower-level and nonconvex upper-level objectives.
method Sequential minimax optimization (SMO) method using modified augmented Lagrangian and penalty schemes.
result Improves operation complexity for finding ε-KKT solutions. Regularization methods are often employed in deep learning neural networks (DNNs) to prevent overfitting. For penalty based DNN regularization methods, convex penalties are typically considered because of their optimization guarantees. Recent theoretical work have shown that nonconvex penalties that satisfy certain reg…
Regularized least-squares approaches have been successfully applied to linear system identification. Recent approaches use quadratic penalty terms on the unknown impulse response defined by stable spline kernels, which control model space complexity by leveraging regularity and bounded-input bounded-output stability. T…
New algorithms reduce complexity for solving nonconvex optimization problems with stochastic objectives and constraints.
problem Solving nonconvex optimization problems with stochastic objectives and constraints.
method Single-loop quadratic penalty and augmented Lagrangian algorithms with variance reduction techniques.
result Achieved best-known complexity guarantees for solving nonconvex optimization problems with stochastic objectives and constraints.
Bayesian optimization tackles constrained high-dimensional problems with penalties and trust regions.
problem Constrained optimization in high-dimensional black-box settings with expensive evaluations and complex feasibility regions.
method Penalty formulation, surrogate model, trust region strategy, Expected Improvement acquisition function.
result The proposed Trust Region method identifies high-quality feasible solutions with fewer evaluations and maintains stable performance.
In this paper we consider l0 regularized convex cone programming problems. In particular, we first propose an iterative hard thresholding (IHT) method and its variant for solving l0 regularized box constrained convex programming. We show that the sequence generated by these methods converges to a local minimizer.…
WrapNet optimizes inference for low-resolution neural networks by using 8-bit additions.
problem Reducing multiplication complexity in low-resolution neural networks.
method Adapting neural networks to use low-resolution (8-bit) additions in accumulators, with a cyclic activation layer and overflow penalty regularizer.
result Achieves comparable classification accuracy to 32-bit counterparts using low-resolution additions.
One-bit measurements widely exist in the real world, and they can be used to recover sparse signals. This task is known as the problem of learning halfspaces in learning theory and one-bit compressive sensing (1bit-CS) in signal processing. In this paper, we propose novel algorithms based on both convex and nonconvex s…
Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions. However, generalization properties of GANs have not been well understood. In this paper, we analyze the generalization of GANs in practical settings. We show that discriminators trained on discr…
The paper studies robust risk measures with linear penalties under uncertain distributions.
problem Risk measurement under distributional uncertainty.
method Robust distortion risk measures with linear penalty function under distributional constraints.
result Explicit characterization of optimal quantile distribution and value function.
Whilst there are many approaches to detecting changes in mean for a univariate time-series, the problem of detecting multiple changes in slope has comparatively been ignored. Part of the reason for this is that detecting changes in slope is much more challenging. For example, simple binary segmentation procedures do no…
We propose two sparsity-aware normalized subband adaptive filter (NSAF) algorithms by using the gradient descent method to minimize a combination of the original NSAF cost function and the l1-norm penalty function on the filter coefficients. This l1-norm penalty exploits the sparsity of a system in the coefficients upd…
2-level SLOPE improves high-dimensional inference with fewer hyperparameters.
problem High-dimensional linear regression with complex penalty sequences.
method 2-level SLOPE with three hyperparameters, reducing penalty space.
result Sharp trade-off between TPP and FDP, computationally efficient.
New sparse penalty improves biclustering for gene expression data.
problem Identifying significant clusters in gene expression data.
method Prenet penalty applied to SSVD for biclustering.
result Mixed Prenet penalty effectively clusters non-overlapped data.
New approach avoids excess empirical risk in domain generalization.
problem Learning models that generalize to unseen distributions from diverse data sets.
method Minimizes penalty under constraint of optimal empirical risk, leveraging rate-distortion theory.
result Significant improvements in domain generalization performance across multiple methods.
We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlappin…
A popular heuristic for improved performance in Generative adversarial networks (GANs) is to use some form of gradient penalty on the discriminator. This gradient penalty was originally motivated by a Wasserstein distance formulation. However, the use of gradient penalty in other GAN formulations is not well motivated.…
Curvature penalties improve interpretability of KANs without sacrificing accuracy.
problem Pathologically high-curvature oscillations in KANs activations make them hard to interpret.
method Derived a curvature penalty and proved an upper bound on model curvature.
result KANs with curvature penalties achieve substantially smoother activations while maintaining accuracy.
Regularization plays a crucial role in supervised learning. Most existing methods enforce a global regularization in a structure agnostic manner. In this paper, we initiate a new direction and propose to enforce the structural simplicity of the classification boundary by regularizing over its topological complexity. In…
New method reduces bias in sparse Bayesian learning.
problem High sparsity in statistical models leads to significant bias.
method Variable-coefficient ℓ1 penalty with hyperpriors. result Reduces bias in sparse Bayesian learning.
We consider a one-period Kyle (1985) framework where the insider can be subject to a penalty if she trades. We establish existence and uniqueness of equilibrium for virtually any penalty function when noise is uniform. In equilibrium, the demand of the insider and the price functions are in general non-linear and remai…
New nonconvex penalty smooths at origin for deep learning.
problem Improving variable selection and bias in high-dimensional statistical learning.
method Developed a new nonconvex penalty function smooth at origin.
result Asymptotic bias of new penalty function vanishes exponentially fast.