Curvature penalties improve interpretability of KANs without sacrificing accuracy.
problem Pathologically high-curvature oscillations in KANs activations make them hard to interpret.
method Derived a curvature penalty and proved an upper bound on model curvature.
result KANs with curvature penalties achieve substantially smoother activations while maintaining accuracy.
Gradient descent training of neural networks leads to solutions close to natural cubic splines.
problem Understanding the implicit bias of gradient descent in neural networks.
method Analysis of gradient descent training for wide neural networks, focusing on the curvature penalty and initialization schemes.
result The solutions of gradient descent training are polyharmonic splines for certain initialization schemes.
Improved penalty-based methods for bilevel optimization with reduced complexity.
problem Suboptimal complexity in solving bilevel optimization problems with large penalty terms.
method Novel penalty reformulation that decouples upper and lower-level variables, enabling larger step sizes and reduced iteration complexity.
result PBGD-Free algorithm that avoids inner loops for coupled constraint BLO problems, with reduced iteration complexity.
A new spline method for manifold learning using Hessian-based curvature penalties.
problem Learning manifolds with curvature penalties in high dimensions.
method Generalizes thin-plate splines to flat manifolds using Hessian matrices, minimizing square error with curvature constraints.
result Existence and uniqueness of the spline solution, expressed as Green's functions and Hessian approximations.
A new framework selects information sources to test hypotheses robustly, even with misclassifications.
problem Robust hypothesis testing with misclassification penalties.
method Introduces a misclassification penalty framework and an efficient greedy algorithm.
result Proposes a submodular surrogate metric for better selection.
Proposes a new network to improve nuclei segmentation in histopathology images.
problem Challenges in separating overlapped nuclei in histopathology images.
method Introduces a bending loss regularized network to minimize contour points with large curvatures.
result Outperforms six state-of-the-art approaches on five quantitative metrics.
Unified analysis of multi-task functional linear regression with manifold and composite penalties.
problem Estimating slope functions from functional data with multi-task learning.
method Penalized splines with manifold constraint and composite quadratic penalty.
result Unified convergence upper bound and phase transition behaviors for estimators.
Mirror flow in shallow neural networks shows similar implicit bias to gradient flow, with key differences in curvature penalties.
problem Analyzing implicit bias in shallow neural networks with mirror flow.
method Characterization through variational problems and scaled potentials.
result Mirror flow with scaled potentials induces a rich class of biases not captured by RKHS norms.
Regularization methods are often employed in deep learning neural networks (DNNs) to prevent overfitting. For penalty based DNN regularization methods, convex penalties are typically considered because of their optimization guarantees. Recent theoretical work have shown that nonconvex penalties that satisfy certain reg…
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
Gradient penalty improves GAN performance by inducing a large-margin classifier.
problem Improving GAN performance and addressing vanishing gradients.
method A unifying framework of expected margin maximization, showing gradient penalties induce large-margin classifiers.
result Gradient penalties reduce vanishing gradients and produce better generated outputs.
One-bit measurements widely exist in the real world, and they can be used to recover sparse signals. This task is known as the problem of learning halfspaces in learning theory and one-bit compressive sensing (1bit-CS) in signal processing. In this paper, we propose novel algorithms based on both convex and nonconvex s…
The paper studies robust risk measures with linear penalties under uncertain distributions.
problem Risk measurement under distributional uncertainty.
method Robust distortion risk measures with linear penalty function under distributional constraints.
result Explicit characterization of optimal quantile distribution and value function.
The use of machine-learning in neuroimaging offers new perspectives in early diagnosis and prognosis of brain diseases. Although such multivariate methods can capture complex relationships in the data, traditional approaches provide irregular (l2 penalty) or scattered (l1 penalty) predictive pattern with a very limited…
New sparse penalty improves biclustering for gene expression data.
problem Identifying significant clusters in gene expression data.
method Prenet penalty applied to SSVD for biclustering.
result Mixed Prenet penalty effectively clusters non-overlapped data.
New approach avoids excess empirical risk in domain generalization.
problem Learning models that generalize to unseen distributions from diverse data sets.
method Minimizes penalty under constraint of optimal empirical risk, leveraging rate-distortion theory.
result Significant improvements in domain generalization performance across multiple methods.
We study the problem of estimating high-dimensional regression models regularized by a structured sparsity-inducing penalty that encodes prior structural information on either the input or output variables. We consider two widely adopted types of penalties of this kind as motivating examples: (1) the general overlappin…
New method reduces bias in sparse Bayesian learning.
problem High sparsity in statistical models leads to significant bias.
method Variable-coefficient ℓ1 penalty with hyperpriors. result Reduces bias in sparse Bayesian learning.
We consider a one-period Kyle (1985) framework where the insider can be subject to a penalty if she trades. We establish existence and uniqueness of equilibrium for virtually any penalty function when noise is uniform. In equilibrium, the demand of the insider and the price functions are in general non-linear and remai…
New nonconvex penalty smooths at origin for deep learning.
problem Improving variable selection and bias in high-dimensional statistical learning.
method Developed a new nonconvex penalty function smooth at origin.
result Asymptotic bias of new penalty function vanishes exponentially fast.
We study the problem of learning high dimensional regression models regularized by a structured-sparsity-inducing penalty that encodes prior structural information on either input or output sides. We consider two widely adopted types of such penalties as our motivating examples: 1) overlapping group lasso penalty, base…
Wasserstein GAN(WGAN) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution. Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint. In this study, we prove the local stability of optimizing the simple gradien…
Multi-group learners suffer a penalty in transductive learning.
problem The penalty on multi-group learners in transductive learning.
method Analyzing the relationship between the number of groups and the error rate.
result The penalty can increase linearly with the number of groups, up to the square-root of the sample size.
Global minima found for multidimensional scaling with penalties.
problem Finding global minima in multidimensional scaling.
method Combining stress loss function with a quadratic penalty term to find minimizers.
result Trajectory of minimizers leads to global minima.
Proposes an alternative invariance penalty to address domain generalization issues.
problem Addressing domain generalization problems by finding invariant representations.
method Revisits the Gramian matrix of the data representation to propose an alternative invariance penalty.
result The proposed approach guarantees recovery of an invariant representation under mild conditions.
Paper proposes efficient algorithms for designing SLOPE penalty sequences.
problem Designing SLOPE penalty sequences is computationally expensive.
method Developed two efficient algorithms: PGD and CD for Gaussian and general data matrices respectively.
result Demonstrated improved mean squared error performance of SLOPE with designed penalties.
Insider trading is reduced when penalized, affecting expected penalties in a non-monotone way.
problem Reducing insider trading behavior when insiders face legal penalties.
method Characterized via a backward stochastic differential equation (BSDE) with a non-linear operator.
result The insider's expected penalties are non-monotone in the fee structure and determined by relative entropy.
Sparse reconstruction approaches using the re-weighted l1-penalty have been shown, both empirically and theoretically, to provide a significant improvement in recovering sparse signals in comparison to the l1-relaxation. However, numerical optimization of such penalties involves solving problems with l1-norms in the ob…
In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…
Improved online penalty selection for time series models.
problem Efficiently selecting penalty parameters for lasso in time series models.
method Enhanced autoregressive model with online penalty selection.
result Significantly improved computational performance and forecast accuracy.
In the multiple changepoint setting, various search methods have been proposed which involve optimising either a constrained or penalised cost function over possible numbers and locations of changepoints using dynamic programming. Such methods are typically computationally intensive. Recent work in the penalised optimi…
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural prop…
Bayesian priors and penalties are equivalent in variational inference.
problem Understanding the relationship between Bayesian priors and penalties in variational inference.
method Characterizing the regularizers that can arise in variational inference and providing a systematic way to compute the prior corresponding to a given penalty.
result Equivalence between Bayesian priors and penalties in variational inference.
Proposes using MLP for predicting optimal penalty in changepoint detection.
problem Predicting optimal penalty for changepoints in sequences.
method Uses a multilayer perceptron (MLP) with ReLU activation function to predict penalty.
result Improves accuracy and F1 score compared to existing models.
In this paper, we introduce a novel combined reward cum penalty loss function to handle the regression problem. The proposed combined reward cum penalty loss function penalizes the data points which lie outside the ε-tube of the regressor and also assigns reward for the data points which lie inside of the ε-tube of…
Recently, there has been focus on penalized log-likelihood covariance estimation for sparse inverse covariance (precision) matrices. The penalty is responsible for inducing sparsity, and a very common choice is the convex l1 norm. However, the best estimator performance is not always achieved with this penalty. The …
A new penalty-free method optimizes portfolios without quantum annealing penalties.
problem Optimizing portfolios with quantum annealing penalties.
method Removing the penalty term and using a classical feasibility projector.
result Significant reduction in chain-break fractions and post-processed regret.
Study ablated data augmentation techniques and their mathematical equivalence to penalties.
problem Lack of mathematical understanding of differences between ablated data augmentation techniques.
method Formal model of mean ablated data augmentation and inverted dropout for linear regression; empirical validation for deep networks.
result Ablated data augmentation and inverted dropout are mathematically equivalent to penalties in optimization.
Faster, better sparse model estimation for large datasets.
problem Sparse model estimation for large datasets with millions of samples and features.
method Coordinate descent, working sets, Anderson acceleration.
result Significantly faster and more efficient than state-of-the-art algorithms.
Support vector machines (SVMs) with sparsity-inducing nonconvex penalties have received considerable attentions for the characteristics of automatic classification and variable selection. However, it is quite challenging to solve the nonconvex penalized SVMs due to their nondifferentiability, nonsmoothness and nonconve…
A fast method estimates group-adaptive elastic net penalties using co-data.
problem Computational inefficiency in estimating group-adaptive elastic net penalties.
method Derive low-dimensional representation of Taylor approximation for marginal likelihood and its derivative for group-adaptive ridge penalties; approximate elastic net marginal likelihood by ridge; transform ridge penalties to elastic net penalties.
result Significantly decreases computation time and outperforms other methods.
This work addresses the issue of large covariance matrix estimation in high-dimensional statistical analysis. Recently, improved iterative algorithms with positive-definite guarantee have been developed. However, these algorithms cannot be directly extended to use a nonconvex penalty for sparsity inducing. Generally, a…
Proximal policy optimization(PPO) has been proposed as a first-order optimization method for reinforcement learning. We should notice that an exterior penalty method is used in it. Often, the minimizers of the exterior penalty functions approach feasibility only in the limits as the penalty parameter grows increasingly…
Paper develops algorithms for sparse linear regression with generalized elastic net penalty.
problem Sparse linear regression with robust penalty for high-dimensional data.
method Iterative Reweighted Framework based on ADMM and PMM with SNN.
result Efficient algorithms provide superior performance in both simulated and real data.
We consider a class of constrained optimization problems with a possibly nonconvex non-Lipschitz objective and a convex feasible set being the intersection of a polyhedron and a possibly degenerate ellipsoid. Such problems have a wide range of applications in data science, where the objective is used for inducing spars…
Efficient cross-validation for multi-penalty ridge regression.
problem Optimizing penalties in cross-validation for multi-penalty ridge regression.
method Developed a computationally efficient formula for the multi-penalty, sample-weighted hat-matrix.
result Significant speed-up in cross-validation for ridge regression models.
Optimal subset selection for hypothesis testing with penalties.
problem Optimal subset selection of information sources for hypothesis testing with misclassification penalties.
method Proposes a misclassification penalty framework and studies two variants of subset selection problems under centralized Bayesian learning.
result Proves the submodularity of the objective and constraints of the subset selection problems and establishes performance guarantees for greedy algorithms.
Proposes a new SPVM model for RVM with more flexible priors.
problem Improper priors on multiple penalty parameters in RVM lead to improper posteriors.
method Introduces a single penalty approach (SPRVM) and a semi-Bayesian fitting method.
result SPRVM allows for more flexible priors and has proven conditions for posterior propriety.