Proposes an alternative invariance penalty to address domain generalization issues.
problem Addressing domain generalization problems by finding invariant representations.
method Revisits the Gramian matrix of the data representation to propose an alternative invariance penalty.
result The proposed approach guarantees recovery of an invariant representation under mild conditions.
PAIR optimizes machine learning models to generalize better to out-of-distribution data.
problem Optimization of machine learning models for out-of-distribution (OOD) generalization often leads to compromises that weaken robustness.
method Introduces a multi-objective optimization (MOO) perspective and a new optimization scheme called PAreto Invariant Risk Minimization (PAIR).
result PAIR improves robustness of OOD objectives by cooperatively optimizing with other objectives, yielding top OOD performances.
New algorithm uses conditionally invariant components to improve domain adaptation performance.
problem Improving domain adaptation performance when source and target data distributions differ.
method Conditionally invariant components (CICs) and importance-weighted conditional invariant penalty (IW-CIP) algorithm.
result New algorithm provides target risk guarantees and addresses label-flipping features.
Proposes a new framework for invariant quadratic P&L predictions in option books.
problem Inconsistent second-order P&L predictions across different factor parameterizations.
method Local, model-agnostic framework using covariant Hessian defined by an affine connection.
result Coordinate-invariant quadratic P&L predictions that match desk targets.
Regularization methods are often employed in deep learning neural networks (DNNs) to prevent overfitting. For penalty based DNN regularization methods, convex penalties are typically considered because of their optimization guarantees. Recent theoretical work have shown that nonconvex penalties that satisfy certain reg…
Gradient penalty improves GAN performance by inducing a large-margin classifier.
problem Improving GAN performance and addressing vanishing gradients.
method A unifying framework of expected margin maximization, showing gradient penalties induce large-margin classifiers.
result Gradient penalties reduce vanishing gradients and produce better generated outputs.
One-bit measurements widely exist in the real world, and they can be used to recover sparse signals. This task is known as the problem of learning halfspaces in learning theory and one-bit compressive sensing (1bit-CS) in signal processing. In this paper, we propose novel algorithms based on both convex and nonconvex s…
The paper studies robust risk measures with linear penalties under uncertain distributions.
problem Risk measurement under distributional uncertainty.
method Robust distortion risk measures with linear penalty function under distributional constraints.
result Explicit characterization of optimal quantile distribution and value function.
The use of machine-learning in neuroimaging offers new perspectives in early diagnosis and prognosis of brain diseases. Although such multivariate methods can capture complex relationships in the data, traditional approaches provide irregular (l2 penalty) or scattered (l1 penalty) predictive pattern with a very limited…
New sparse penalty improves biclustering for gene expression data.
problem Identifying significant clusters in gene expression data.
method Prenet penalty applied to SSVD for biclustering.
result Mixed Prenet penalty effectively clusters non-overlapped data.
New approach avoids excess empirical risk in domain generalization.
problem Learning models that generalize to unseen distributions from diverse data sets.
method Minimizes penalty under constraint of optimal empirical risk, leveraging rate-distortion theory.
result Significant improvements in domain generalization performance across multiple methods.
We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlappin…
Curvature penalties improve interpretability of KANs without sacrificing accuracy.
problem Pathologically high-curvature oscillations in KANs activations make them hard to interpret.
method Derived a curvature penalty and proved an upper bound on model curvature.
result KANs with curvature penalties achieve substantially smoother activations while maintaining accuracy.
New method reduces bias in sparse Bayesian learning.
problem High sparsity in statistical models leads to significant bias.
method Variable-coefficient ℓ1 penalty with hyperpriors. result Reduces bias in sparse Bayesian learning.
We consider a one-period Kyle (1985) framework where the insider can be subject to a penalty if she trades. We establish existence and uniqueness of equilibrium for virtually any penalty function when noise is uniform. In equilibrium, the demand of the insider and the price functions are in general non-linear and remai…
New nonconvex penalty smooths at origin for deep learning.
problem Improving variable selection and bias in high-dimensional statistical learning.
method Developed a new nonconvex penalty function smooth at origin.
result Asymptotic bias of new penalty function vanishes exponentially fast.
We study the problem of learning high dimensional regression models regularized by a structured-sparsity-inducing penalty that encodes prior structural information on either input or output sides. We consider two widely adopted types of such penalties as our motivating examples: 1) overlapping group lasso penalty, base…
Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution. Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint. In this study, we prove the local stability of optimizing the simple gradien…
Multi-group learners suffer a penalty in transductive learning.
problem The penalty on multi-group learners in transductive learning.
method Analyzing the relationship between the number of groups and the error rate.
result The penalty can increase linearly with the number of groups, up to the square-root of the sample size.
Global minima found for multidimensional scaling with penalties.
problem Finding global minima in multidimensional scaling.
method Combining stress loss function with a quadratic penalty term to find minimizers.
result Trajectory of minimizers leads to global minima.
Improved penalty-based methods for bilevel optimization with reduced complexity.
problem Suboptimal complexity in solving bilevel optimization problems with large penalty terms.
method Novel penalty reformulation that decouples upper and lower-level variables, enabling larger step sizes and reduced iteration complexity.
result PBGD-Free algorithm that avoids inner loops for coupled constraint BLO problems, with reduced iteration complexity.
Paper proposes efficient algorithms for designing SLOPE penalty sequences.
problem Designing SLOPE penalty sequences is computationally expensive.
method Developed two efficient algorithms: PGD and CD for Gaussian and general data matrices respectively.
result Demonstrated improved mean squared error performance of SLOPE with designed penalties.
A geometric theory explains loss functions for robust representation learning.
problem Treats robustness, domain adaptation, and sensor drift as separate literatures.
method Estimates covariance Sigma_task and uses it to pin Jacobian penalties.
result Proves optimality and necessity of range coverage for penalty matrices.
Insider trading is reduced when penalized, affecting expected penalties in a non-monotone way.
problem Reducing insider trading behavior when insiders face legal penalties.
method Characterized via a backward stochastic differential equation (BSDE) with a non-linear operator.
result The insider's expected penalties are non-monotone in the fee structure and determined by relative entropy.
Sparse reconstruction approaches using the re-weighted l1-penalty have been shown, both empirically and theoretically, to provide a significant improvement in recovering sparse signals in comparison to the l1-relaxation. However, numerical optimization of such penalties involves solving problems with l1-norms in the ob…
In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…
Improved online penalty selection for time series models.
problem Efficiently selecting penalty parameters for lasso in time series models.
method Enhanced autoregressive model with online penalty selection.
result Significantly improved computational performance and forecast accuracy.
In the multiple changepoint setting, various search methods have been proposed which involve optimising either a constrained or penalised cost function over possible numbers and locations of changepoints using dynamic programming. Such methods are typically computationally intensive. Recent work in the penalised optimi…
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural prop…
Bayesian priors and penalties are equivalent in variational inference.
problem Understanding the relationship between Bayesian priors and penalties in variational inference.
method Characterizing the regularizers that can arise in variational inference and providing a systematic way to compute the prior corresponding to a given penalty.
result Equivalence between Bayesian priors and penalties in variational inference.
Proposes using MLP for predicting optimal penalty in changepoint detection.
problem Predicting optimal penalty for changepoints in sequences.
method Uses a multilayer perceptron (MLP) with ReLU activation function to predict penalty.
result Improves accuracy and F1 score compared to existing models.
In this paper, we introduce a novel combined reward cum penalty loss function to handle the regression problem. The proposed combined reward cum penalty loss function penalizes the data points which lie outside the ε-tube of the regressor and also assigns reward for the data points which lie inside of the ε-tube of…
Recently, there has been focus on penalized log-likelihood covariance estimation for sparse inverse covariance (precision) matrices. The penalty is responsible for inducing sparsity, and a very common choice is the convex l1 norm. However, the best estimator performance is not always achieved with this penalty. The …
A new penalty-free method optimizes portfolios without quantum annealing penalties.
problem Optimizing portfolios with quantum annealing penalties.
method Removing the penalty term and using a classical feasibility projector.
result Significant reduction in chain-break fractions and post-processed regret.
Study ablated data augmentation techniques and their mathematical equivalence to penalties.
problem Lack of mathematical understanding of differences between ablated data augmentation techniques.
method Formal model of mean ablated data augmentation and inverted dropout for linear regression; empirical validation for deep networks.
result Ablated data augmentation and inverted dropout are mathematically equivalent to penalties in optimization.
Faster, better sparse model estimation for large datasets.
problem Sparse model estimation for large datasets with millions of samples and features.
method Coordinate descent, working sets, Anderson acceleration.
result Significantly faster and more efficient than state-of-the-art algorithms.
Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection. However, it is quite challenging to solve the nonconvex penalized SVMs due to their nondifferentiability, nonsmoothness and nonconve…
A fast method estimates group-adaptive elastic net penalties using co-data.
problem Computational inefficiency in estimating group-adaptive elastic net penalties.
method Derive low-dimensional representation of Taylor approximation for marginal likelihood and its derivative for group-adaptive ridge penalties; approximate elastic net marginal likelihood by ridge; transform ridge penalties to elastic net penalties.
result Significantly decreases computation time and outperforms other methods.
This work addresses the issue of large covariance matrix estimation in high-dimensional statistical analysis. Recently, improved iterative algorithms with positive-definite guarantee have been developed. However, these algorithms cannot be directly extended to use a nonconvex penalty for sparsity inducing. Generally, a…
Proximal policy optimization(PPO) has been proposed as a first-order optimization method for reinforcement learning. We should notice that an exterior penalty method is used in it. Often, the minimizers of the exterior penalty functions approach feasibility only in the limits as the penalty parameter grows increasingly…
Paper develops algorithms for sparse linear regression with generalized elastic net penalty.
problem Sparse linear regression with robust penalty for high-dimensional data.
method Iterative Reweighted Framework based on ADMM and PMM with SNN.
result Efficient algorithms provide superior performance in both simulated and real data.
We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerate ellipsoid. Such problems have a wide range of applications in data science, where the objective is used for inducing spars…
Generative model for set-valued data using permutation invariant flows.
problem Modeling set-valued data with conditional generative models.
method Conditional generative probabilistic model using continuous normalizing flows with permutation equivariant dynamics.
result Significantly outperforms non-permutation invariant baselines in log likelihood and domain-specific metrics.
Efficient cross-validation for multi-penalty ridge regression.
problem Optimizing penalties in cross-validation for multi-penalty ridge regression.
method Developed a computationally efficient formula for the multi-penalty, sample-weighted hat-matrix.
result Significant speed-up in cross-validation for ridge regression models.
Paper designs a penalty for model order selection using information criteria.
problem Selecting the correct model order from a set of candidate models.
method Designs a penalty for the generalized information criterion (GIC) to minimize underestimation.
result Optimal penalty minimizes underestimation while keeping overestimation below a specified level.
Optimal subset selection for hypothesis testing with penalties.
problem Optimal subset selection of information sources for hypothesis testing with misclassification penalties.
method Proposes a misclassification penalty framework and studies two variants of subset selection problems under centralized Bayesian learning.
result Proves the submodularity of the objective and constraints of the subset selection problems and establishes performance guarantees for greedy algorithms.
Proposes a new SPVM model for RVM with more flexible priors.
problem Improper priors on multiple penalty parameters in RVM lead to improper posteriors.
method Introduces a single penalty approach (SPRVM) and a semi-Bayesian fitting method.
result SPRVM allows for more flexible priors and has proven conditions for posterior propriety.
Unified analysis of multi-attribute graph learning with non-convex penalties.
problem Graph inference from multi-attribute data.
method Penalized log-likelihood objective function with ADMM and local linear approximation.
result Local consistency in support recovery and precision matrix estimation for non-convex penalties.