This paper addresses overfitting in dimension reduction methods by calibrating hyperparameters considering noise.
problem Overfitting in dimension reduction methods, especially t-SNE and UMAP, when data contains noise.
method Present a framework to calibrate hyperparameters in the presence of noise for t-SNE and UMAP.
result Recommended hyperparameter values for t-SNE and UMAP are too small and overfit the noise.
MMbeddings reduces categorical embeddings by treating them as latent effects, significantly decreasing parameters and mitigating overfitting.
problem Large cardinalities in categorical embeddings lead to high parameter counts and overfitting.
method MMbeddings treats embeddings as latent random effects in a variational autoencoder framework, reducing parameter count and mitigating overfitting.
result MMbeddings consistently outperforms traditional embeddings across various tasks, demonstrating its potential in machine learning applications.
Physics analogies explain machine learning overfitting control.
problem Understanding and controlling overfitting in machine learning.
method Analogies from physics and biology applied to algorithmic stability and GAN models.
result Physics formulas provide insights into reducing overfitting in machine learning.
The paper tackles manifold overfitting in deep generative models.
problem Manifold overfitting occurs when generative models learn the manifold itself instead of the distribution on it.
method The authors propose a two-step procedure: dimensionality reduction followed by maximum-likelihood density estimation.
result The two-step procedure avoids manifold overfitting and enables density estimation on learned manifolds.
Proposes a simple solution to Gini importance bias in random forests.
problem Gini importance measure in random forests is biased and unreliable.
method Computes loss reduction on out-of-bag samples instead of in-bag.
result Solves the misleading/untrustworthy Gini importance issue.
Meta-learning with network pruning reduces overfitting and improves few-shot learning.
problem Overfitting in meta-learning models with over-parameterized neural networks.
method Network pruning to control capacity and explicitly reduce generalization gap.
result Uniform concentration analysis shows the benefit of network capacity constraint.
Pruning neural networks can improve test accuracy even with significant parameter reduction.
problem The tradeoff between generalization and stability in neural network pruning.
method Analysis of pruning behavior over training, focusing on instability and its relation to generalization.
result Pruning's benefit to generalization increases with its instability.
From only positive (P) and unlabeled (U) data, a binary classifier could be trained with PU learning, in which the state of the art is unbiased PU learning. However, if its model is very flexible, empirical risks on training data will go negative, and we will suffer from serious overfitting. In this paper, we propose a…
Overfitting frequently occurs in deep learning. In this paper, we propose a novel regularization method called Drop-Activation to reduce overfitting and improve generalization. The key idea is to drop nonlinear activation functions by setting them to be identity functions randomly during training time. During testing, …
Optimized GPRNN reduces model complexity and overfitting, improving performance.
problem Overfitting in neural networks and high model complexity.
method Gaussian Process Regression - Neural Network hybrid with optimized redundant coordinates.
result Optimized GPRNN achieves lower test set error with fewer terms/neurons.
BAE uses boosting to improve autoencoder ensembles for robust outlier detection.
problem Overfitting in autoencoders limits their effectiveness in unsupervised outlier detection.
method Boosting-based Autoencoder Ensemble (BAE) trains autoencoders sequentially with weighted sampling to reduce outliers and inject diversity.
result BAE outperforms state-of-the-art approaches in various outlier detection conditions.
DIMCO learns discrete codes maximizing mutual info with labels, reducing overfitting and improving efficiency.
problem Learning compact discrete representations for data.
method DIMCO learns k-way d-dimensional codes maximizing mutual information between codes and labels, with regularization to enforce code independence.
result DIMCO reduces overfitting and improves efficiency in few-shot classification.
A deep learning method for probabilistic weather forecasting.
problem Probabilistic forecasting of weather.
method Two chained machine-learning steps: dimension reduction and density estimation using normalizing flows.
result The method produces accurate conditional forecast distributions for weather.
A new method solves complex hydroelectricity planning problems.
problem Solving multistage stochastic linear programming for hydrothermal dispatch planning.
method Regularized Linear Decision Rules (AdaLASSO) to reduce overfitting and improve out-of-sample performance.
result Significant reductions in non-zero coefficients and improved spot-price profiles.
Graphon pooling preserves spectral properties in GNNs, reducing overfitting.
problem Unclear pooling and sampling strategies in GNNs that alter graph structure.
method Modeling graph layers as elements of a sequence converging to a graphon.
result Graphon pooling GNNs reduce overfitting and improve performance.
Assisted by the availability of data and high performance computing, deep learning techniques have achieved breakthroughs and surpassed human performance empirically in difficult tasks, including object recognition, speech recognition, and natural language processing. As they are being used in critical applications, un…
Machine learning reduces variance in online experiment results.
problem Reducing variance in randomized controlled trials.
method Machine learning regression-adjusted treatment effect estimator (MLRATE).
result MLRATE reduces estimator variance by over 70% in A/A tests.
This paper introduces early exits in neural networks for faster inference.
problem Reducing inference time and preventing overfitting in neural networks.
method Designing and training multi-output neural networks with early exits.
result Significant reductions in inference time and improved robustness.
Two-layer CNNs can overfit without harm under certain conditions.
problem Understanding when and how overfitting occurs in neural networks.
method Theoretical analysis of a two-layer CNN trained by gradient descent.
result A sharp phase transition between benign and harmful overfitting based on signal-to-noise ratio.
New algorithms reduce overfitting in multiclass classification.
problem Excessive reuse of test datasets in machine learning leads to overfitting, especially in multiclass classification.
method Developed computationally efficient algorithms to reduce overfitting bias in multi-class classification.
result Achieved overfitting bias of Θ(√(k/(mn)), k/n), matching known upper bounds.
Attention models can overfit without harming test performance.
problem Understanding benign overfitting in single-head attention models.
method Analyzing conditions for benign overfitting in a single-head softmax attention model.
result A single-head attention model can overfit without harming test performance under certain conditions.
The paper tackles backtest overfitting in cryptocurrency trading using deep reinforcement learning.
problem Backtest overfitting in deep reinforcement learning for cryptocurrency trading.
method Formulated hypothesis test for overfitting detection, trained agents, estimated overfitting probability, and rejected overfitted agents.
result Less overfitted deep reinforcement learning agents outperformed more overfitted agents and market benchmarks.
The paper proposes a generic learning method for complex structures.
problem Learning algorithms for complex structures are inefficient and require significant effort.
method Mapping any complex structure onto a generic form (serialization) and applying sequence-based density estimators.
result The method is competitive or better than specialized algorithms for given structures and provides protection from overfitting.
ResNet models overfit benignly on Cifar10 but not on ImageNet due to label noise.
problem Understanding why benign overfitting fails in real-world classification tasks with label noise.
method Theoretical analysis of benign overfitting under a mild overparameterization setup.
result Benign overfitting can fail in the presence of label noise, unlike in heavy overparameterization settings.
Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When R…
We prove overfitting in minimal and random NNs, tempering the effect.
problem Overfitting in minimal and random neural networks.
method Analyzing binary weight fitting to noisy data, proving overfitting is tempered.
result The overfitting of minimal and random neural networks is tempered.
Detects overfitting in models trained on test sets.
problem Challenges in verifying overfitting without independent test sets.
method Uses adversarial examples and unbiased error estimates to test for overfitting.
result Correctly identifies overfitting to the training set but not to the test set.
The paper examines how spike strengths and alignments affect overfitting in linear regression models.
problem The impact of spike strengths and alignments on overfitting in linear regression models.
method Characterization of generalization error through exact expressions and analysis of spike strengths, aspect ratio, and target alignment.
result Increasing spike strength can lead to catastrophic overfitting before benign overfitting, especially in well-specified aligned problems.
New insights into why neural networks can overfit without interpolating data.
problem Understanding why neural networks can overfit without interpolating data in fixed dimensions.
method Analyzing the smoothness of estimators and their derivatives.
result Benign overfitting is possible with estimators that have large enough derivatives, not just in high dimensions but also in fixed dimensions.
Efficient FPGA dropout algorithm reduces memory usage.
problem Overfitting in deep neural networks.
method Hardware-oriented dropout algorithm for FPGA implementation.
result Significant resource reduction in FPGA implementation.
In statistics and machine learning, approximation of an intractable integration is often achieved by using the unbiased Monte Carlo estimator, but the variances of the estimation are generally high in many applications. Control variates approaches are well-known to reduce the variance of the estimation. These control v…
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
New study finds many neural networks are not benignly overfitting.
problem Understanding the behavior of overfitting in neural networks.
method Exploring kernel ridge regression and deep neural networks to identify overfitting behaviors.
result Many interpolating methods, including neural networks, exhibit tempered overfitting rather than benign or catastrophic.
The paper explores how neural networks avoid overfitting by learning from flat minima.
problem How to prevent overfitting in neural networks by learning from flat minima.
method Study of one- and two-layer neural network models, derivation of algorithms focusing on wide flat regions.
result Wide flat minima coexist with narrower minima and critical points, and are associated with good minimizers.
Two-layer ReLU networks can overfit without harm, study finds.
problem Understanding when and how two-layer ReLU networks can overfit without harming generalization.
method Established algorithm-dependent risk bounds for two-layer ReLU convolutional neural networks with label-flipping noise.
result Gradient descent-trained ReLU networks can achieve near-zero training loss and Bayes optimal test risk.
Study on GANs under and overfitting using discriminator unseen data.
problem Understanding and mitigating under and overfitting in GANs.
method Using discriminators unseen by the generator to measure generalization.
result Model capacity of the discriminator affects generator quality; large discriminator capacities do not prevent overfitting.
Overfitting occurs when RL agents correlate rewards with spurious observation features.
problem Overfitting in reinforcement learning due to correlation with spurious observation features.
method Developed a framework to analyze and design synthetic benchmarks from modified observation spaces.
result Agents can overfit to different observation spaces even if the MDP dynamics are fixed.
New findings on how neural networks generalize with varying dimensions.
problem Understanding how neural networks generalize with different dimensions and noise levels.
method Study of 2-layer ReLU NNs in a classification setting, proving transitions in overfitting types.
result The type of overfitting transitions from tempered to benign as input dimension increases.
Study on benign overfitting in leaky ReLUs with moderate input dimensions.
problem Understanding when overfitting is beneficial in neural networks.
method Two-layer leaky ReLU networks trained with hinge loss, considering signal-to-noise ratio.
result Characterization of conditions for benign overfitting based on signal-to-noise ratio.
Two new algorithms reduce feature space while preserving non-linear relationships.
problem High-dimensional data and overfitting issues.
method Bias-variance analysis for non-linear transformations and generalized linear models.
result Competitive performance on regression and classification tasks.
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
Adversarial training can lead to overfitting without compromising robustness.
problem Explaining benign overfitting in adversarially robust linear classification.
method Theoretical analysis and numerical experiments on adversarial training.
result Adversarially trained linear classifiers can achieve near-optimal risks despite overfitting noisy data.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
New insights into when benign overfitting occurs in linear and classification tasks.
problem Understanding when benign overfitting happens in linear and classification models.
method Analysis of a generic data model and comparison of predictors (minimum-norm interpolating and max-margin).
result The minimum-norm interpolating predictor is biased towards an inconsistent solution, preventing benign overfitting in linear regression.
Empirical study shows GANs overfit and drop modes when training is deterministic.
problem Understanding overfitting and mode drop in GAN training.
method Empirical analysis of GAN training with and without stochasticity.
result GANs overfit and drop modes when training is deterministic.
Paper explains adversarial training's robust overfitting through a minimax game perspective.
problem Adversarial training suffers from robust overfitting after learning rate decay.
method Viewing adversarial training as a dynamic minimax game, analyzing how LR decay breaks balance and leads to overfitting.
result ReBalanced Adversarial Training (ReBAT) alleviates robust overfitting without sacrificing robustness.
Tests for overfitting in machine learning models.
problem Overfitting in high complexity models.
method Hypothesis test using concentration bounds.
result Valid test for identifying overfitting.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.