New quantum kernels avoid overfitting by combining local and global components.
problem Exponential concentration in quantum kernels leads to overfitting.
method Local-global quantum kernels combining small subsystem and full-system measurements.
result Demonstrated benign overfitting in local-global quantum kernels.
TPBS models improve robustness to overfitting with localized Dirichlet energy regularization.
problem Global Dirichlet energy-based regularization fails for TPBS models due to perfect interpolation.
method Propose local Dirichlet energy regularization and two inference estimators.
result TPBS models outperform neural networks in overfitting regimes and maintain competitive performance otherwise.
Efficient regularization mitigates catastrophic overfitting in single-step adversarial training.
problem Catastrophic overfitting in single-step adversarial training.
method ELLE regularization term to enforce local linearity of the loss function.
result Our regularization term effectively mitigates catastrophic overfitting without the drawbacks of previous methods.
DCTN uses tensor networks for image classification, achieving state-of-the-art results.
problem Improving image classification accuracy with deep neural networks.
method Developed a novel deep convolutional tensor network (DCTN) based on Entangled plaquette states (EPS).
result DCTN achieves state-of-the-art results on MNIST and FashionMNIST but overfits on CIFAR10.
Gen-LRA attacks synthetic data leakage without model knowledge.
problem Auditing synthetic data privacy leakage.
method Generative Likelihood Ratio Attack (Gen-LRA).
result Gen-LRA outperforms other attacks across metrics.
Graph kernels based on the 1-dimensional Weisfeiler-Leman algorithm and corresponding neural architectures recently emerged as powerful tools for (supervised) learning with graphs. However, due to the purely local nature of the algorithms, they might miss essential patterns in the given data and can only handle binar…
Locally sparse neural networks improve interpretability for biomedical tabular data.
problem Overfitting and lack of interpretability in neural networks for tabular biomedical data.
method Locally sparse neural network with a gating network to select relevant features.
result The method outperforms state-of-the-art models in synthetic and real-world biomedical datasets.
FedCONST adapts update magnitudes to enhance feature generalization in FL.
problem Heterogeneous client data in FL leads to overfitting and distorted transferable features.
method FedCONST uses linear convex constraints to stabilize training and preserve generalization.
result FedCONST enhances feature transferability and robustness, achieving state-of-the-art performance.
DropCluster clusters features in convolutional layers to prevent overfitting.
problem Dropout's effectiveness in convolutional layers is limited due to feature spatial correlation.
method DropCluster clusters features in convolutional layers and randomly drops clusters during training.
result DropCluster controls overfitting better than other approaches on various datasets.
LocalDrop uses local Rademacher complexity for neural network regularization.
problem Overfitting in deep neural networks.
method Developed a new regularization function based on local Rademacher complexity.
result Demonstrated effectiveness of LocalDrop through extensive experiments.
New method prevents adversarial training failure in deep networks.
problem Adversarial training failure in deep networks.
method GradAlign method to prevent catastrophic overfitting.
result GradAlign prevents adversarial training failure in deep networks.
Paper introduces a new optimization method for imbalanced datasets.
problem Overfitting in imbalanced datasets, especially in financial fraud detection.
method Fractional Weyl Integral optimization algorithm.
result Significantly improved performance in financial fraud detection (40% improvement in PR-AUC).
Certified training improves robustness against adversarial attacks.
problem Certified training's gap with empirical robustness limits its practical utility.
method Combining adversarial attacks with network over-approximations.
result Certified training can prevent catastrophic overfitting and bridge the gap to multi-step baselines.
Develops a new fuzzy model using QPs and ewl2 regularization to improve local region behavior.
problem Inability of constant and linear functions to accurately describe local regions in fuzzy models.
method Applied Fuzzy C-Means for structure identification, used QPs as consequents, introduced ewl2 regularization.
result Improved model's ability to describe local regions without overfitting.
Proposes Coherent Gradients to explain and reduce overfitting in neural networks.
problem Why neural networks generalize well despite fitting random data.
method Hypothesis about gradient dynamics and a modification to gradient descent.
result Supports hypothesis with heuristic arguments and perturbative experiments.
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
Similarity/Distance measures play a key role in many machine learning, pattern recognition, and data mining algorithms, which leads to the emergence of metric learning field. Many metric learning algorithms learn a global distance function from data that satisfy the constraints of the problem. However, in many real-wor…
Two-layer CNNs can overfit without harm under certain conditions.
problem Understanding when and how overfitting occurs in neural networks.
method Theoretical analysis of a two-layer CNN trained by gradient descent.
result A sharp phase transition between benign and harmful overfitting based on signal-to-noise ratio.
Attention models can overfit without harming test performance.
problem Understanding benign overfitting in single-head attention models.
method Analyzing conditions for benign overfitting in a single-head softmax attention model.
result A single-head attention model can overfit without harming test performance under certain conditions.
The paper tackles backtest overfitting in cryptocurrency trading using deep reinforcement learning.
problem Backtest overfitting in deep reinforcement learning for cryptocurrency trading.
method Formulated hypothesis test for overfitting detection, trained agents, estimated overfitting probability, and rejected overfitted agents.
result Less overfitted deep reinforcement learning agents outperformed more overfitted agents and market benchmarks.
New theory shows large learning rates prevent overfitting in neural networks.
problem Generalization of two-layer ReLU neural networks in noisy regression problems.
method Gradient descent with constant learning rate converges to stable minima.
result Gradient descent with large learning rates finds smooth, sparse fits.
ResNet models overfit benignly on Cifar10 but not on ImageNet due to label noise.
problem Understanding why benign overfitting fails in real-world classification tasks with label noise.
method Theoretical analysis of benign overfitting under a mild overparameterization setup.
result Benign overfitting can fail in the presence of label noise, unlike in heavy overparameterization settings.
We prove overfitting in minimal and random NNs, tempering the effect.
problem Overfitting in minimal and random neural networks.
method Analyzing binary weight fitting to noisy data, proving overfitting is tempered.
result The overfitting of minimal and random neural networks is tempered.
The paper examines how spike strengths and alignments affect overfitting in linear regression models.
problem The impact of spike strengths and alignments on overfitting in linear regression models.
method Characterization of generalization error through exact expressions and analysis of spike strengths, aspect ratio, and target alignment.
result Increasing spike strength can lead to catastrophic overfitting before benign overfitting, especially in well-specified aligned problems.
New insights into why neural networks can overfit without interpolating data.
problem Understanding why neural networks can overfit without interpolating data in fixed dimensions.
method Analyzing the smoothness of estimators and their derivatives.
result Benign overfitting is possible with estimators that have large enough derivatives, not just in high dimensions but also in fixed dimensions.
Framework for understanding overfitting and underfitting using information theory.
problem Understanding and preventing overfitting and underfitting in machine learning.
method Information-theoretic framework measuring algorithm capacity and dataset information transfer.
result Upper-bounding algorithm capacity and establishing its relationship to machine learning quantities.
Physics analogies explain machine learning overfitting control.
problem Understanding and controlling overfitting in machine learning.
method Analogies from physics and biology applied to algorithmic stability and GAN models.
result Physics formulas provide insights into reducing overfitting in machine learning.
A primary concern of excessive reuse of test datasets in machine learning is that it can lead to overfitting. Multiclass classification was recently shown to be more resistant to overfitting than binary classification. In an open problem of COLT 2019, Feldman, Frostig, and Hardt ask to characterize the dependence of th…
New study finds many neural networks are not benignly overfitting.
problem Understanding the behavior of overfitting in neural networks.
method Exploring kernel ridge regression and deep neural networks to identify overfitting behaviors.
result Many interpolating methods, including neural networks, exhibit tempered overfitting rather than benign or catastrophic.
SGD generalizes well in high dimensions without regularization.
problem Generalization of overparameterized models in high dimensions.
method Stochastic Gradient Descent (SGD) for convex and locally convex loss functions.
result Generalization error is independent of the ambient dimension p under certain conditions. Two-layer ReLU networks can overfit without harm, study finds.
problem Understanding when and how two-layer ReLU networks can overfit without harming generalization.
method Established algorithm-dependent risk bounds for two-layer ReLU convolutional neural networks with label-flipping noise.
result Gradient descent-trained ReLU networks can achieve near-zero training loss and Bayes optimal test risk.
Study on GANs under and overfitting using discriminator unseen data.
problem Understanding and mitigating under and overfitting in GANs.
method Using discriminators unseen by the generator to measure generalization.
result Model capacity of the discriminator affects generator quality; large discriminator capacities do not prevent overfitting.
Overfitting occurs when RL agents correlate rewards with spurious observation features.
problem Overfitting in reinforcement learning due to correlation with spurious observation features.
method Developed a framework to analyze and design synthetic benchmarks from modified observation spaces.
result Agents can overfit to different observation spaces even if the MDP dynamics are fixed.
New findings on how neural networks generalize with varying dimensions.
problem Understanding how neural networks generalize with different dimensions and noise levels.
method Study of 2-layer ReLU NNs in a classification setting, proving transitions in overfitting types.
result The type of overfitting transitions from tempered to benign as input dimension increases.
Study on benign overfitting in leaky ReLUs with moderate input dimensions.
problem Understanding when overfitting is beneficial in neural networks.
method Two-layer leaky ReLU networks trained with hinge loss, considering signal-to-noise ratio.
result Characterization of conditions for benign overfitting based on signal-to-noise ratio.
The repeated community-wide reuse of test sets in popular benchmark problems raises doubts about the credibility of reported test-error rates. Verifying whether a learned model is overfitted to a test set is challenging as independent test sets drawn from the same data distribution are usually unavailable, while other …
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
CFFL framework improves fairness in FL without sacrificing accuracy.
problem Overfitting and lack of collaborative fairness in Federated Learning.
method CFFL framework uses reputation to ensure participants converge to different models.
result CFFL achieves high fairness, comparable accuracy, and better performance than Standalone and Distributed frameworks.
Adversarial training can lead to overfitting without compromising robustness.
problem Explaining benign overfitting in adversarially robust linear classification.
method Theoretical analysis and numerical experiments on adversarial training.
result Adversarially trained linear classifiers can achieve near-optimal risks despite overfitting noisy data.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
New insights into when benign overfitting occurs in linear and classification tasks.
problem Understanding when benign overfitting happens in linear and classification models.
method Analysis of a generic data model and comparison of predictors (minimum-norm interpolating and max-margin).
result The minimum-norm interpolating predictor is biased towards an inconsistent solution, preventing benign overfitting in linear regression.
Empirical study shows GANs overfit and drop modes when training is deterministic.
problem Understanding overfitting and mode drop in GAN training.
method Empirical analysis of GAN training with and without stochasticity.
result GANs overfit and drop modes when training is deterministic.
Paper explains adversarial training's robust overfitting through a minimax game perspective.
problem Adversarial training suffers from robust overfitting after learning rate decay.
method Viewing adversarial training as a dynamic minimax game, analyzing how LR decay breaks balance and leads to overfitting.
result ReBalanced Adversarial Training (ReBAT) alleviates robust overfitting without sacrificing robustness.
Tests for overfitting in machine learning models.
problem Overfitting in high complexity models.
method Hypothesis test using concentration bounds.
result Valid test for identifying overfitting.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.
New bounds for KRR condition number reveal overfitting phenomena.
problem Characterizing overfitting in KRR with varying kernel spectral decay.
method Derived new bounds for kernel matrices, enhanced test error bounds, and identified feature independence role.
result Identified tempered and catastrophic overfitting phenomena.
New method uses randomized sparse neural networks to solve time-dependent PDEs more accurately and efficiently.
problem Numerical challenges in training neural networks sequentially in time to solve time-dependent PDEs.
method Introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step.
result Up to two orders of magnitude more accurate and two orders of magnitude faster than dense update schemes.
A new exploration method for bandit and reinforcement learning.
problem Uncertainty quantification in exploration.
method Residual Overfit Method of Exploration (ROME).
result ROME drives exploration towards actions with overfitting.