Improved Kriging model reduces prediction errors.
problem Improving prediction accuracy in Kriging models.
method Theta-regularized Kriging model with Lasso, Ridge, and Elastic-net penalties.
result The Theta-regularized Kriging model outperforms other penalized Kriging models in accuracy and stability.
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
The paper improves model robustness by regularizing posterior differences.
problem Improving model robustness in noisy input scenarios.
method Posterior differential regularization with f-divergence. result Regularizing with f-divergence improves model robustness. New measure shows various training techniques control model complexity.
problem Understanding how to control model complexity in deep learning.
method Developed geometric complexity measure and demonstrated its effectiveness.
result Many training techniques control geometric complexity, providing a unified framework.
Cross-regularization adapts model complexity during training.
problem Manual tuning of model complexity for overfitting prevention.
method Directly adapts regularization parameters through validation gradients during training.
result Organic emergence of architecture-specific regularization during training.
Spectral regularization simplifies sequence models by focusing on grammatical simplicity.
problem Sequence modeling challenges in learning tasks.
method Introduces spectral regularization based on Hankel matrices and trace norm, addressing bi-infinite matrices with an unbiased estimator.
result Demonstrates spectral regularization's potential benefits on Tomita grammars.
Establish C^{1,2} regularity of American value functions in Heston model
problem Regularity of American put options in Heston model
method PDE techniques
result C^{1,2} regularity in exercise domain and smooth-fit principle
Regularization methods can overregulate, suppressing causal features.
problem Mitigating shortcuts in models exploiting spurious correlations.
method Analysis of regularization methods and their effects on causal features.
result Regularization can overregulate, suppressing causal features.
The paper studies how regularization parameters affect sparsity in deep neural networks.
problem Reducing the complexity of deep neural networks by promoting sparsity.
method Derives ℓ1-norm sparsity-promoting models, characterizes sparsity levels, and develops algorithms for selecting optimal regularization parameters. result Developed algorithms to select regularization parameters for desired sparsity levels in neural networks.
Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum …
While the impact of variational inference (VI) on posterior inference in a fixed generative model is well-characterized, its role in regularizing a learned generative model when used in variational autoencoders (VAEs) is poorly understood. We study the regularizing effects of variational distributions on learning in ge…
Bayesian regularization improves policy performance in noisy MDPs.
problem Suboptimal policies from estimated model parameters.
method Bayesian regularization of MDP objective function with prior information.
result Regularized policies show better robustness against model noise.
A framework for analyzing regularizers to ensure trustworthy theory-driven model estimation.
problem Uncertain choice of regularizers can compromise the interpretability of deep grey-box models.
method Adapting neural net architecture and training objective to analyze regularizer behavior empirically.
result Empirical analysis of regularizers helps in making a justified choice for trustworthy theory-driven model estimation.
Novel regularization for Vision Transformers improves model generalization and sparsity.
problem Improving generalization and sparsity in Vision Transformers.
method Likelihood-guided variational Ising-based regularization.
result Improved generalization and sparsity in Vision Transformers.
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
A simple regularization technique speeds up training of Neural ODEs.
problem Training Neural ODEs is computationally expensive.
method Randomly sampling the end time of the ODE during training.
result Significantly decreases training time and improves performance.
In application domains such as healthcare, we want accurate predictive models that are also causally interpretable. In pursuit of such models, we propose a causal regularizer to steer predictive models towards causally-interpretable solutions and theoretically study its properties. In a large-scale analysis of Electron…
Dropout improves regularization in flexible models for rare features.
problem Understanding theoretical properties of dropout in generalized linear models.
method Theoretical analysis and application to adaptive smoothing with B-splines.
result Dropout prefers rare features in mean and dispersion parameters.
Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the d…
Local regularization fails in transductive learning for some multiclass problems.
problem Whether local regularization can learn all transductive multiclass problems.
method Provided a negative answer by exhibiting a specific multiclass problem.
result Local regularization cannot learn all transductive multiclass problems.
Inserts proximal mapping into deep networks for better regularization.
problem Effective regularization of deep learning models to handle adversarial perturbations and correlations between modalities.
method Proposes a new layer that directly produces regularized hidden layer outputs using proximal mapping.
result Outperforms state-of-the-art methods in robust temporal learning and multiview modeling.
Propagation-regularization improves GNN performance by infusing extra graph information.
problem The effectiveness of graph Laplacian regularization in GNNs is questioned and improved upon.
method Introducing Propagation-regularization (P-reg) to enhance GNN performance.
result P-reg boosts GNN performance on various tasks across multiple datasets.
Double descent risk in L2-regularized models explained and mitigated.
problem Risk of overparameterized models in machine learning.
method Analysis of L2-regularized models, two-layer neural networks, and CNNs.
result Double descent risk in L2-regularized models can be explained and mitigated by adjusting regularization strengths.
Study iterative regularization for linear models with convex bias, improving robust sparse recovery.
problem Improving robust sparse recovery with iterative regularization for linear models.
method Primal-dual gradient approach, analyzing convergence in presence of noise, combining regularization and optimization.
result Theoretical results show state-of-the-art performances with computational speed-ups.
New regularization scheme for FMs improves feature interaction selection.
problem Feature selection in FMs leads to loss of feature interactions.
method Proposes a new regularization scheme for FMs with upper bound of ℓ1 regularizer. result Improves feature interaction selection without restricting sparsity patterns.
DANR improves network regularization for spatio-temporal data.
problem Inadequate edge weights compromise network regularization.
method Discrepancy-aware network regularization (DANR) with ADMM.
result DANR achieves improved performance on various tasks.
A simple regularization method improves model generalization.
problem Overfitting due to lack of labeled data in machine learning models.
method Density-fixing regularization method based on class prior distribution.
result Improves model generalization performance by approximating class prior distribution.
Unified framework for accurate coresets in latent variable models and regularized regression.
problem Efficiently training models on large datasets.
method Unified framework for constructing accurate coresets for latent variable models and ℓp-regularized regression. result Unified framework reduces coreset size for latent variable models and ℓp-regularized regression. We propose a tree regularization framework, which enables many tree models to perform feature selection efficiently. The key idea of the regularization framework is to penalize selecting a new feature for splitting when its gain (e.g. information gain) is similar to the features used in previous splits. The regularizat…
Hidden regular variation is a sub-model of multivariate regular variation and facilitates accurate estimation of joint tail probabilities. We generalize the model of hidden regular variation to what we call hidden domain of attraction. We exhibit examples that illustrate the need for a more general model and discuss de…
Optimal regularization can prevent the double descent phenomenon in learning models.
problem The double descent phenomenon in learning models, where test performance is non-monotonic in sample size and model size.
method Theoretical and empirical study of optimal ℓ2 regularization for linear regression models and neural networks. result Optimally-tuned ℓ2 regularization achieves monotonic test performance for certain models and mitigates the double descent phenomenon for more general models. The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. Study compares dropout and l2 regularization in linear models.
problem Understanding the statistical behavior of dropout and l2 regularization in linear models.
method Derives non-asymptotic bounds for gradient descent iterates with dropout and compares them to l2 regularization.
result Indicates a more subtle relationship between dropout and l2 regularization, highlighting interactions between dynamics and randomness.
Study Lipschitz regularity for manifold-constrained ROF model on curved surfaces.
problem Lipschitz regularity for manifold-constrained ROF model on curved surfaces.
method Generalization of ROF model, existence and uniqueness of minimizers, regularity results on PDE system.
result Lipschitz regularity of minimizers without convexity requirements.
High-dimensional prediction is a challenging problem setting for traditional statistical models. Although regularization improves model performance in high dimensions, it does not sufficiently leverage knowledge on feature importances held by domain experts. As an alternative to standard regularization techniques, we p…
We analyze learning curves of RF models with convex regularization and derive precise asymptotic expressions.
problem Understanding the learning curves of RF models with general convex regularization.
method Novel multi-level application of the convex Gaussian min max theorem (CGMT) to compute precise asymptotic expressions.
result Precise asymptotic expressions for learning curves of RF models with separable strongly convex regularization or ℓ1 regularization. In this work we study input gradient regularization of deep neural networks, and demonstrate that such regularization leads to generalization proofs and improved adversarial robustness. The proof of generalization does not overcome the curse of dimensionality, but it is independent of the number of layers in the networ…
New method encodes function preferences into neural nets for better generalization.
problem Challenges in encoding explicit function preferences in neural network training.
method Function-space empirical Bayes (FSEB) regularization.
result FSEB leads to near-perfect semantic shift detection and improved generalization.
Proves an ε-regularity theorem for Ricci flows, leading to new singularity estimates.
problem Analyzing singularity models in Fano Kähler-Ricci flows.
method Proves ε-regularity theorem and uses it to derive new estimates.
result Establishes new estimates for singularity models of Fano Kähler-Ricci flows.
Study introduces a new method for multiple parameter regularization in polynomial functional regression.
problem Handling varying regularization parameters in polynomial functional regression.
method Developed a theoretically grounded algorithm for multiple parameter regularization and model aggregation.
result Promising results from evaluations on synthetic and real-world data.
Study L2 regularization in deep networks, uncovering performance relations and proposing a training schedule.
problem Understanding and optimizing L2 regularization in deep learning models. method Empirical observations and theoretical analysis of gradient flow dynamics in infinitely wide networks.
result Empirical relations between model performance, L2 coefficient, learning rate, and training steps; optimal regularization parameter prediction; improved training schedule. We study the regularity problem of the nonlinear sigma model with gravitino fields in higher dimensions. After setting up the geometric model, we derive the Euler--Lagrange equations and consider the regularity of weak solutions defined in suitable Sobolev spaces. We show that any weak solution is actually smooth under…
Method introduces topological regularization using information filtering networks.
problem Sparse probabilistic modeling and multicollinear regression.
method Topological regularization via information filtering network.
result Direct application to L0-norm regularized problems. Proposes a guaranteed regularization method for maximum likelihood estimation using gauge symmetry in Kullback-Leibler divergence.
problem Overfitting in maximum likelihood estimation.
method Introduces a regularization approach based on gauge symmetry in Kullback-Leibler divergence.
result The method provides a theoretically guaranteed optimal model without frequent hyperparameter tuning.
This work compares regularization and constrained inference for label constraints in machine learning.
problem Improving model performance with label constraints in machine learning.
method Comparison of regularization and constrained inference strategies.
result Constrained inference reduces population risk by correcting model violations, while regularization narrows the generalization gap but introduces bias.
A new method for community detection in networks is presented.
problem Community detection in network analysis.
method Mixed regularized spectral clustering (Mixed-RSC) based on the regularized Laplacian matrix.
result The method is asymptotically consistent under mild conditions.
This paper studies least-square regression penalized with partly smooth convex regularizers. This class of functions is very large and versatile allowing to promote solutions conforming to some notion of low-complexity. Indeed, they force solutions of variational problems to belong to a low-dimensional manifold (the so…
Exploiting the appropriate inductive bias based on the knowledge of data is essential for achieving good performance in statistical machine learning. In practice, however, the domain knowledge of interest often provides information on the relationship of data attributes only distantly, which hinders direct utilization …