arXiv research

Framework for understanding overfitting and underfitting using information theory.

New research shows CPE only occurs when Bayesian posterior underfits.

Multi-expert L2D underfits more severely, requiring new methods.

Bayesian deep learning avoids underfitting by projecting onto null space of generalized Gauss-Newton matrix.

OUI tool detects optimal Weight Decay for DNNs without validation data.

New regularization method corrects over-shrinkage in small data regression.

A common data mining task on networks is community detection, which seeks an unsupervised decomposition of a network into structural groups based on statistical regularities in the network's connectivity. Although many methods exist, the No Free Lunch theorem for community detection implies that each makes some kind of…

Bayesian model selection can be misled by ELBO under certain conditions.

Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.

Proposes a new approach to regression learning that addresses overfitting and underfitting.

Research shows bias in machine learning can be due to algorithmic flaws, not just data.

We study parameter estimation in Nonlinear Factor Analysis (NFA) where the generative model is parameterized by a deep neural network. Recent work has focused on learning such models using inference (or recognition) networks; we identify a crucial problem when modeling large, sparse, high-dimensional datasets -- underf…

GWRBoost improves GWR for better spatial relationship quantification.

This paper explains double descent using VC theory.

We analyze MDL for binary classification, quantifying overfitting and underfitting.

Knowledge distillation improves model accuracy by mimicking teacher model probabilities.

This study analyzes VAEs using ID and II, revealing a transition in behaviour and distinct training phases.

ResNet models overfit benignly on Cifar10 but not on ImageNet due to label noise.

This paper improves Bayesian neural nets by using local linearization.

Proposes a new learning method for RBMs that combines strengths of forward and reverse KLD.

New attention mechanism improves meta-transfer learning in dynamic tasks.

Overfitting & underfitting and stable training are an important challenges in machine learning. Current approaches for these issues are mixup, SamplePairing and BC learning. In our work, we state the hypothesis that mixing many images together can be more effective than just two. Batchboost pipeline has three stages: (…

We investigate under and overfitting in Generative Adversarial Networks (GANs), using discriminators unseen by the generator to measure generalization. We find that the model capacity of the discriminator has a significant effect on the generator's model quality, and that the generator's poor performance coincides with…

Causal forests use honesty to reduce overfitting, but it can also reduce accuracy, especially with large datasets.

Wide BNNs with odd activations fail to approximate data under mean-field inference.

Improves classifier performance in multi-stage processes with adversarial autoencoders and multi-task learning.

Double descent observed in tree-based models for genomic prediction.

In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fun…

GANs trained on artificial datasets avoid dataset biases, revealing generative model weaknesses.

Supervised (linear) embedding models like Wsabie and PSI have proven successful at ranking, recommendation and annotation tasks. However, despite being scalable to large datasets they do not take full advantage of the extra data due to their linear nature, and typically underfit. We propose a new class of models which …

We propose Learned Accept/Reject Sampling (LARS), a method for constructing richer priors using rejection sampling with a learned acceptance function. This work is motivated by recent analyses of the VAE objective, which pointed out that commonly used simple priors can lead to underfitting. As the distribution induced …

Support Vector Data Description (SVDD) is a machine-learning technique used for single class classification and outlier detection. SVDD formulation with kernel function provides a flexible boundary around data. The value of kernel function parameters affects the nature of the data boundary. For example, it is observed …

Current variational inference methods for hierarchical Bayesian nonparametric models can neither characterize the correlation structure among latent variables due to the mean-field setting, nor infer the true posterior dimension because of the universal truncation. To overcome these limitations, we propose the conditio…

This chapter provides a self-contained introduction to the use of Bayesian inference to extract large-scale modular structures from network data, based on the stochastic blockmodel (SBM), as well as its degree-corrected and overlapping generalizations. We focus on nonparametric formulations that allow their inference i…

We analyze bias-variance of margin losses.

Statistical shape models enhance machine learning algorithms providing prior information about deformation. A Point Distribution Model (PDM) is a popular landmark-based statistical shape model for segmentation. It requires choosing a model order, which determines how much of the variation seen in the training data is a…

This work considers the problem of binary classification: given training data $x_1, \dots, x_n$ from a certain population, together with associated labels $y_1,\dots, y_n \in \left\{0,1 \right\}$, determine the best label for an element $x$ not among the training data. More specifically, this work considers a variant o…

The paper analyzes and mitigates biases in scalable Gaussian Process methods.

We place an Indian Buffet process (IBP) prior over the structure of a Bayesian Neural Network (BNN), thus allowing the complexity of the BNN to increase and decrease automatically. We further extend this model such that the prior on the structure of each hidden layer is shared globally across all layers, using a Hierar…

In few-shot learning, typically, the loss function which is applied at test time is the one we are ultimately interested in minimising, such as the mean-squared-error loss for a regression problem. However, given that we have few samples at test time, we argue that the loss function that we are interested in minimising…

Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data ef…

Support vector data description (SVDD) is a popular technique for detecting anomalies. The SVDD classifier partitions the whole space into an inlier region, which consists of the region near the training data, and an outlier region, which consists of points away from the training data. The computation of the SVDD class…

New priors improve Bayesian neural networks without cooling.

Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression. We here examine whether inclusion of the rate also acts…

Several structure learning algorithms have been proposed towards discovering causal or Bayesian Network (BN) graphs. The validity of these algorithms tends to be evaluated by assessing the relationship between the learnt and the ground truth graph. However, there is no agreed scoring metric to determine this relationsh…

Proposes MGCE for improved classification performance.

The paper addresses the invariance issue in Bayesian neural networks using linearized Laplace approximation.

New method selects optimal bandwidth for price return density estimation, impacting efficient market hypothesis evaluation.