Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
New research shows CPE only occurs when Bayesian posterior underfits.
problem Model misspecification leading to CPE under perfect model specification.
method Theoretical analysis of Bayesian posterior and underfitting.
result No CPE if there is no underfitting of the Bayesian posterior.
Multi-expert L2D underfits more severely, requiring new methods.
problem Underfitting in multi-expert L2D settings.
method PiCCE (Pick the Confident and Correct Expert), a surrogate-based method.
result PiCCE effectively reduces multi-expert L2D to a single-expert-like problem, resolving underfitting.
Bayesian deep learning avoids underfitting by projecting onto null space of generalized Gauss-Newton matrix.
problem Bayesian deep learning often underfits, leading to less accurate predictions than point estimates.
method Proposes a matrix-free algorithm to project onto the null space of the generalized Gauss-Newton matrix, ensuring Bayesian predictions do not underfit.
result The method scales to large models, including vision transformers with 28 million parameters, and avoids underfitting.
OUI tool detects optimal Weight Decay for DNNs without validation data.
problem Optimal Weight Decay hyperparameter selection for DNNs.
method Overfitting-Underfitting Indicator (OUI) tool.
result OUI correlates with improved generalization and validation scores.
New regularization method corrects over-shrinkage in small data regression.
problem Over-shrinkage in small data regression leading to underfitting.
method Negative-capable ridge family that permits negative regularization.
result Negative regularization acts as controlled anti-shrinkage, increasing effective complexity.
A common data mining task on networks is community detection, which seeks an unsupervised decomposition of a network into structural groups based on statistical regularities in the network's connectivity. Although many methods exist, the No Free Lunch theorem for community detection implies that each makes some kind of…
Bayesian model selection can be misled by ELBO under certain conditions.
problem Misleading model selection when using ELBO for Bayesian inference.
method Analysis of ELBO-based hyperparameter learning in a simple regression model.
result Bayesian model selection can prefer overfit models when ELBO is used, contrary to evidence.
Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.
problem Uncertainty quantification in neural networks is underfitting or computationally demanding.
method Gaussian Process Activation function (GAPA) for neuron-level uncertainty, with two methods: GAPA-Free and GAPA-Variational.
result GAPA-Variational outperforms Laplace approximation on most datasets in uncertainty quantification metrics.
Proposes a new approach to regression learning that addresses overfitting and underfitting.
problem Regression learning issues, including overfitting and underfitting.
method Introduces epsilon-Confidence Approximately Correct (epsilon CoAC) framework using Kullback Leibler divergence.
result Demonstrates improved learnability and accuracy compared to cross-validation.
Research shows bias in machine learning can be due to algorithmic flaws, not just data.
problem Underestimation bias in machine learning algorithms.
method Initial research to understand factors contributing to bias in classification algorithms.
result Regularization methods to address overfitting can also accentuate bias.
We study parameter estimation in Nonlinear Factor Analysis (NFA) where the generative model is parameterized by a deep neural network. Recent work has focused on learning such models using inference (or recognition) networks; we identify a crucial problem when modeling large, sparse, high-dimensional datasets -- underf…
GWRBoost improves GWR for better spatial relationship quantification.
problem Underfitting in GWR for complex data and lack of explainable quantification.
method Geographically weighted gradient boosting model using localized additive model and gradient boosting optimization.
result Significant improvement in RMSE and AICc compared to classic GWR.
This paper explains double descent using VC theory.
problem Understanding the generalization of overfitting neural networks.
method VC-theoretical analysis of double descent.
result Double descent can be explained by classical VC-generalization bounds.
We analyze MDL for binary classification, quantifying overfitting and underfitting.
problem Understanding the trade-off between underfitting and overfitting in MDL for binary classification.
method Complete characterization of the regularization curve for MDL, extending previous work to all λ. result Precise quantitative description of the worst case limiting error as a function of λ and noise level. Knowledge distillation improves model accuracy by mimicking teacher model probabilities.
problem Improving model accuracy through model compression.
method Casting knowledge distillation as a semiparametric inference problem, deriving new guarantees, and developing enhancements.
result Enhancements improve student performance by mitigating teacher overfitting and underfitting.
This study analyzes VAEs using ID and II, revealing a transition in behaviour and distinct training phases.
problem Understanding the hidden representations and training phases of VAEs.
method Analysis using Intrinsic Dimension (ID) and Information Imbalance (II).
result VAEs exhibit a transition in behaviour and distinct training phases when the bottleneck size exceeds the Intrinsic Dimension of the data.
ResNet models overfit benignly on Cifar10 but not on ImageNet due to label noise.
problem Understanding why benign overfitting fails in real-world classification tasks with label noise.
method Theoretical analysis of benign overfitting under a mild overparameterization setup.
result Benign overfitting can fail in the presence of label noise, unlike in heavy overparameterization settings.
This paper improves Bayesian neural nets by using local linearization.
problem Underfitting in Bayesian neural networks.
method Local linearization of Bayesian neural networks to create a generalized linear model (GLM) for predictions.
result The GLM predictive resolves common underfitting problems of the Laplace approximation.
Proposes a new learning method for RBMs that combines strengths of forward and reverse KLD.
problem Underfitting and mode-collapse issues in RBM learning.
method Ratio divergence learning using target energy.
result Significantly outperforms other learning methods in energy function fitting, mode-covering, and stability.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
We investigate under and overfitting in Generative Adversarial Networks (GANs), using discriminators unseen by the generator to measure generalization. We find that the model capacity of the discriminator has a significant effect on the generator's model quality, and that the generator's poor performance coincides with…
Causal forests use honesty to reduce overfitting, but it can also reduce accuracy, especially with large datasets.
problem Causal forests' honesty can reduce accuracy of individual treatment effects.
method Using honest estimation to divide data into two samples, one for subgroup definition and another for effect estimation.
result Honest estimation can reduce accuracy by requiring 27% more data to match performance of non-honest models.
Wide BNNs with odd activations fail to approximate data under mean-field inference.
problem Theoretical limitations of mean-field variational inference in wide, deep Bayesian neural networks.
method Analysis of mean-field variational inference in fully-connected BNNs with odd activation functions and Gaussian likelihood.
result The optimal mean-field variational posterior predictive distribution converges to the prior predictive distribution as network width increases.
Improves classifier performance in multi-stage processes with adversarial autoencoders and multi-task learning.
problem Challenges in training classifiers due to varying sample sizes and information content across stages.
method Combines adversarial autoencoders, multi-task learning, and semi-supervised learning to address underfitting and overfitting.
result Our approach outperforms state-of-the-art methods across different domains.
Self-distillation improves model performance but can lead to underfitting.
problem Understanding why self-distillation improves model performance and its limitations.
method Theoretical analysis of self-distillation in Hilbert space with ℓ2 regularization. result Self-distillation modifies regularization by limiting the number of basis functions, potentially leading to underfitting.
Batchboost stabilizes training by mixing and pairing samples, improving accuracy.
problem Stabilizing training in machine learning, especially avoiding overfitting and underfitting.
method Batchboost pipeline with three stages: pairing, mixing, and feeding. Mixing uses mixup technique.
result Batchboost achieves 0.5-3% better accuracy than mixup on CIFAR-10 & Fashion-MNIST.
Double descent observed in tree-based models for genomic prediction.
problem Understanding the generalization behavior of tree-based models in machine learning.
method Systematic variation of model complexity in a genomic prediction task using whole-genome sequencing data.
result Double descent emerges only when complexity is scaled jointly across learner capacity and ensemble size.
In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fun…
GANs trained on artificial datasets avoid dataset biases, revealing generative model weaknesses.
problem GANs trained on real datasets often underfit or overfit, making analysis difficult.
method Trained GANs on artificial datasets with infinite samples and simple distributions.
result GANs fail to learn optimal parameters, suggesting limitations in generative models.
Supervised (linear) embedding models like Wsabie and PSI have proven successful at ranking, recommendation and annotation tasks. However, despite being scalable to large datasets they do not take full advantage of the extra data due to their linear nature, and typically underfit. We propose a new class of models which …
We propose Learned Accept/Reject Sampling (LARS), a method for constructing richer priors using rejection sampling with a learned acceptance function. This work is motivated by recent analyses of the VAE objective, which pointed out that commonly used simple priors can lead to underfitting. As the distribution induced …
Support Vector Data Description (SVDD) is a machine-learning technique used for single class classification and outlier detection. SVDD formulation with kernel function provides a flexible boundary around data. The value of kernel function parameters affects the nature of the data boundary. For example, it is observed …
This chapter provides a self-contained introduction to the use of Bayesian inference to extract large-scale modular structures from network data, based on the stochastic blockmodel (SBM), as well as its degree-corrected and overlapping generalizations. We focus on nonparametric formulations that allow their inference i…
We analyze bias-variance of margin losses.
problem Understanding model overfitting/underfitting.
method Bias-variance decomposition for strictly convex margin losses.
result Expected risk decomposes into central model risk and data variation.
Statistical shape models enhance machine learning algorithms providing prior information about deformation. A Point Distribution Model (PDM) is a popular landmark-based statistical shape model for segmentation. It requires choosing a model order, which determines how much of the variation seen in the training data is a…
This work considers the problem of binary classification: given training data x1,…,xn from a certain population, together with associated labels y1,…,yn∈{0,1}, determine the best label for an element x not among the training data. More specifically, this work considers a variant o…
The paper analyzes and mitigates biases in scalable Gaussian Process methods.
problem Modeling biases in scalable Gaussian Process methods.
method Randomized truncation estimators to eliminate bias in exchange for increased variance.
result Randomized truncation estimators meaningfully outperform biased counterparts with minimal additional computation.
We place an Indian Buffet process (IBP) prior over the structure of a Bayesian Neural Network (BNN), thus allowing the complexity of the BNN to increase and decrease automatically. We further extend this model such that the prior on the structure of each hidden layer is shared globally across all layers, using a Hierar…
In few-shot learning, typically, the loss function which is applied at test time is the one we are ultimately interested in minimising, such as the mean-squared-error loss for a regression problem. However, given that we have few samples at test time, we argue that the loss function that we are interested in minimising…
Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data ef…
Support vector data description (SVDD) is a popular technique for detecting anomalies. The SVDD classifier partitions the whole space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD class…
CATVI improves variational inference for Bayesian nonparametric models by reducing divergence and improving prediction accuracy.
problem Limitations of current variational inference methods in characterizing latent correlations and inferring true posterior dimensions.
method CATVI integrates conditional and adaptive truncation into variational inference, maximizing nonparametric evidence lower bound and using Monte Carlo integration.
result CATVI outperforms traditional methods in Bayesian nonparametric topic models, reducing perplexity and improving topic-word clustering.
New priors improve Bayesian neural networks without cooling.
problem Bayesian neural networks underfit on clean datasets.
method Introduce DirClip and confidence priors to replace cooling.
result DirClip and confidence priors outperform cold posterior.
Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression. We here examine whether inclusion of the rate also acts…
Several structure learning algorithms have been proposed towards discovering causal or Bayesian Network (BN) graphs. The validity of these algorithms tends to be evaluated by assessing the relationship between the learnt and the ground truth graph. However, there is no agreed scoring metric to determine this relationsh…
Proposes MGCE for improved classification performance.
problem Optimizing between robustness and optimization difficulty in classification.
method Minimax formulation of GCE leading to convex optimization over margins.
result MGCE achieves strong accuracy and better calibration, especially in noisy labels.
The paper addresses the invariance issue in Bayesian neural networks using linearized Laplace approximation.
problem Bayesian neural networks fail to maintain invariance under reparameterization, leading to different posterior densities for identical functions.
method Developed a geometric view of reparameterizations and a Riemannian diffusion process to extend reparameterization invariance to neural network predictive.
result Empirically improved posterior fit through approximate posterior sampling.