New method finds optimal training stop point with noisy labeled data.
problem Finding optimal training stop point with noisy labeled data.
method Analyzed training accuracy rate changes for different noise ratios to identify a training stop region. Developed a heuristic algorithm based on a small-learning assumption.
result Identified optimal training stop point at or close to maximum obtainable test accuracy.
Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overfitting to the noisy labels. In this paper, we claim that such overfitting can be avoided by "early stopping" training a deep neural network before the noisy labels are severely memorized. Then, we re…
DE-QT detects optimal Q-learning stopping points.
problem Information loss in Q-learning during prolonged training.
method Introducing DE-QT to detect entropy changes in Q-tables.
result DE-QT identifies the best stopping point for Q-learning.
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing est…
SAM minimizes loss sharpness, improving adversarial transferability.
problem Improving adversarial transferability of deep neural networks.
method Evaluating surrogate models trained with seven minimizers, focusing on loss sharpness and flat neighborhoods.
result SAM minimizes loss sharpness, leading to better adversarial transferability.
Annotation of training data is the major bottleneck in the creation of text classification systems. Active learning is a commonly used technique to reduce the amount of training data one needs to label. A crucial aspect of active learning is determining when to stop labeling data. Three potential sources for informing …
We introduce techniques for exploring the functionality of a neural network and extracting simple, human-readable approximations to its performance. By performing gradient ascent on the input space of the network, we are able to produce large populations of artificial events which strongly excite a given classifier. By…
In iterative supervised learning algorithms it is common to reach a point in the search where no further induction seems to be possible with the available data. If the search is continued beyond this point, the risk of overfitting increases significantly. Following the recent developments in inductive semantic stochast…
A new method for early stopping in neural networks without validation sets.
problem Determining when to stop training neural networks to avoid overfitting.
method Channel-wise DeepNNK (CW-DeepNNK) using non-negative kernel regression and polytope interpolation.
result The proposed early stopping criterion based on CW-DeepNNK performs better than standard validation-based methods.
Continuous-time optimal stopping solved with deep reinforcement learning
problem Optimal stopping problems in continuous time
method CARLOS (Continuous-time Adaptive Reinforcement Learning for Optimal Stopping)
result Higher prices than existing Bermudan solvers, approaching American upper bound
The study reveals optimal early stopping behaviors in deep learning models.
problem Understanding optimal early stopping in deep learning models.
method Theoretical analysis of linear models and experimental validation.
result Two distinct behaviors of optimal early stopping time depending on model dimension relative to dataset features.
This paper improves neural network predictions with early stopping using conformal calibration.
problem Lack of precise statistical guarantees for neural networks trained with early stopping.
method Conformalized early stopping that combines early stopping with conformal calibration.
result Models provide both accuracy and precise inferences without additional data splits.
We consider the optimal double stopping time problem defined for each stopping time S by $v(S)=\esssup\{E[ψ(τ_1, τ_2) | \F_S], τ_1, τ_2 \geq S \}$. Following the optimal one stopping time problem, we study the existence of optimal stopping times and give a method to compute them. The key point is the construction of …
The study examines how shallow neural nets converge to training samples or manifold points during diffusion.
problem Understanding when and how shallow neural nets converge to training samples or manifold points during diffusion.
method Analysis of shallow ReLU neural network denoisers trained with minimal ℓ2 norm, comparing score flow and diffusion flow. result Probability flow converges to training points, sums of training points, or manifold points, depending on the diffusion time scheduler.
In this paper we introduce and solve a class of optimal stopping problems of recursive type. In particular, the stopping payoff depends directly on the value function of the problem itself. In a multi-dimensional Markovian setting we show that the problem is well posed, in the sense that the value is indeed the unique …
Early stopping in meta-learning improved by analyzing neural activation patterns.
problem Early stopping in few-shot learning is challenging due to distributional shifts between meta-validation and meta-test sets.
method Activation-Based Early-stopping (ABE) analyzes hidden layer activations from unlabelled support examples to detect when target generalization diverges from source data.
result Simple activation statistics can effectively estimate target generalization, improving few-shot transfer learning across various algorithms and datasets.
An unconventional approach for optimal stopping under model ambiguity is introduced. Besides ambiguity itself, we take into account how ambiguity-averse an agent is. This inclusion of ambiguity attitude, via an α-maxmin nonlinear expectation, renders the stopping problem time-inconsistent. We look for subgame perfect…
Mixup improves feature learning by mixing common and rare features.
problem Improving generalization in deep learning models.
method Mixup, a data augmentation technique, is applied to feature learning. Theoretical and experimental studies are conducted to understand its benefits.
result Mixup effectively learns rare features from common ones, leading to better generalization.
We use martingale and stochastic analysis techniques to study a continuous-time optimal stopping problem, in which the decision maker uses a dynamic convex risk measure to evaluate future rewards. We also find a saddle point for an equivalent zero-sum game of control and stopping, between an agent (the "stopper") who c…
GD-trained shallow ReLU nets learn Lipschitz functions with noise.
problem Learning Lipschitz functions with additive noise in overparameterized neural networks.
method Gradient Descent (GD) with early stopping, focusing on the Neural Tangent Kernel (NTK).
result Early-stopped GD achieves minimax optimal rates for learning Lipschitz functions.
Learning in the presence of label noise is a challenging yet important task: it is crucial to design models that are robust in the presence of mislabeled datasets. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of c…
A new stopping criterion for active learning based on deterministic generalization bounds.
problem Determining the optimal stopping point for active learning when data acquisition is costly.
method The proposed stopping criterion is based on the difference in expected generalization errors and hypothesis testing, derived from PAC-Bayesian theory.
result The proposed stopping criterion effectively stops active learning by combining an upper bound with a statistical test.
Early stopping improves generalization in overparameterized diffusion models.
problem Understanding and optimizing generalization in overparameterized diffusion models.
method Revisiting diffusion models, showing generalization occurs before memorization, and developing a phase diagram.
result Generalization time scales with dataset size, supporting early-stopping criteria.
This paper extends stock trading results to include stop-loss orders.
problem Generalizing stock trading results with stop-loss orders.
method Geometric Brownian motion model, affine feedback controller, closed-form expression for cumulative distribution function.
result Affine feedback controller with stop-loss order generalizes results without stop-loss orders.
A deep model learns to stop early based on variational stopping policy.
problem Varying optimal depth for different inputs in deep architectures.
method A steerable architecture learns a feed-forward deep model and a variational stopping policy together.
result The learned deep model and stopping policy improve diverse tasks.
GD with early stopping trains shallow neural nets for nonparametric regression robustly.
problem Learning Lipschitz regression functions with noisy labels.
method Overparameterized shallow neural networks trained by GD with early stopping.
result Optimal rates of convergence for nonparametric regression.
Adversarial training overfits, harming robustness; early stopping fixes this.
problem Adversarial robustness in deep learning overfits to training data.
method Empirical study of adversarially trained deep networks.
result Overfitting to training data harms robust performance in adversarial training.
When a feed-forward neural network (FNN) is trained for source ranging in an ocean waveguide, it is difficult evaluating the range accuracy of the FNN on unlabeled test data. A fitting-based early stopping (FEAST) method is introduced to evaluate the range error of the FNN on test data where the distance of source is u…
Gradient descent trains neural networks to match kernel regression's sharp generalization rate.
problem Training over-parameterized neural networks for nonparametric regression.
method Gradient descent with early stopping on over-parameterized two-layer neural networks.
result Trained neural networks achieve sharp generalization rate of O(εn2). We propose a new approach to solve optimal stopping problems via simulation. Working within the backward dynamic programming/Snell envelope framework, we augment the methodology of Longstaff-Schwartz that focuses on approximating the stopping strategy. Namely, we introduce adaptive generation of the stochastic grids an…
This work bounds the run-time of nonconvex optimization with early stopping.
problem Bounding the expected run-time of nonconvex optimization with early stopping.
method Derives conditions for well-defined early stopping based on validation function norms and bounds the expected number of iterations and gradient evaluations.
result Guarantees the validity of early stopping and provides bounds on the expected run-time for various optimization algorithms.
Bayesian optimization stops when a solution is within ε of the optimum with high probability.
problem Stopping Bayesian optimization prematurely based on a probabilistic criterion.
method Introducing a (ε,δ)-criterion for stopping Bayesian optimization. result Bayesian optimization satisfies the (ε,δ)-criterion under mild assumptions. We first study an optimal stopping problem in which a player (an agent) uses a discrete stopping time in order to stop optimally a payoff process whose risk is evaluated by a (non-linear) g-expectation. We then consider a non-zero-sum game on discrete stopping times with two agents who aim at minimizing their respect…
The paper examines bounds for stop-loss payoffs using transformed random variables.
problem Bounding stop-loss payoffs for a difference of two random variables.
method Analyzes crossing points of cdfs of original and transformed random variables.
result Unique pairwise crossing points for mortality-linked securities under symmetric copulas.
Early stopping of iterative algorithms is a widely-used form of regularization in statistics, commonly used in conjunction with boosting and related gradient-type algorithms. Although consistency results have been established in some settings, such estimators are less well-understood than their analogues based on penal…
The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.
problem Understanding the effect of early stopping on linear regression models.
method Characterization of gradient descent dynamics and analysis of excess risk.
result Early stopped solution is equivalent to minimum norm solution for a generalized ridge regularized problem.
A new method uses deep learning for optimal stopping problems.
problem Solving optimal stopping problems in financial mathematics.
method Deep primal-dual BSDE framework with a novel loss function.
result The method provides a true upper bound for the optimal value.
The paper analyzes and proposes a new stopping criterion for recursive Bayesian classification.
problem Limitations of conventional stopping criteria in recursive Bayesian classification.
method Geometric interpretation of state posterior progression and analysis of conventional criteria.
result Proposes a new stopping criterion to overcome limitations of conventional methods.
Early stopping is a well known approach to reduce the time complexity for performing training and model selection of large scale learning machines. On the other hand, memory/space (rather than time) complexity is the main constraint in many applications, and randomized subsampling techniques have been proposed to tackl…
Gradient-flow optimization is reinterpreted as a statistical inference problem.
problem Optimizing training duration and assessing model performance in deep learning.
method Develops a statistical framework for gradient-flow training, treating it as a random-effects model.
result Establishes asymptotic optimality for prediction and reduces reliance on validation splits.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
New approach improves adversarial robustness without sacrificing natural generalization.
problem Balancing adversarial robustness and natural generalization in machine learning.
method Friendly adversarial training (FAT) using early-stopped PGD to find least adversarial data.
result Early-stopped PGD achieves adversarial robustness without compromising natural generalization.
Adaptive stopping in MCMC using classifier-based dynamics
problem Sampling from complex, unnormalized probability densities
method Training state-dependent neural classifiers
result Significant reduction in average trajectory lengths
Paper proposes a method for early stopping in regression using reproducing kernels.
problem Early stopping for iterative learning algorithms in nonparametric regression.
method Data-driven rule based on minimum discrepancy principle, validated by fixed-point analysis of localized Rademacher complexities.
result The proposed rule is minimax-optimal and performs comparably to cross-validation.
Early stopping improves neural networks' performance on binary classification tasks.
problem Improving shallow ReLU networks' performance on binary classification tasks.
method Gradient descent with early stopping on binary classification data.
result Gradient descent with early stopping achieves population risk arbitrarily close to optimal.
We pursue an early stopping technique that helps Gaussian Restricted Boltzmann Machines (GRBMs) to gain good natural image representations in terms of overcompleteness and data fitting. GRBMs are widely considered as an unsuitable model for natural images because they gain non-overcomplete representations which include…
This paper introduces a more efficient method for estimating level sets with a stopping criterion.
problem Efficiently estimating regions where a function exceeds a threshold without exhaustive evaluations.
method Acquisition strategy with a stopping criterion for ε-accurate level set estimation. result The method satisfies ε-accuracy with a confidence level of 1−δ and guarantees on lower bounds of performance metrics. We consider the problem of stopping a diffusion process with a payoff functional that renders the problem time-inconsistent. We study stopping decisions of naive agents who reoptimize continuously in time, as well as equilibrium strategies of sophisticated agents who anticipate but lack control over their future selves…