In unsupervised learning, collecting more data is not always a costly process unlike the training. For example, it is not hard to enlarge the 40GB WebText used for training GPT-2 by modifying its sampling methodology considering how many webpages there are in the Internet. On the other hand, given that training on this…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “one epoch training”
Few-shot learning aims to train efficient predictive models with a few examples. The lack of training data leads to poor models that perform high-variance or low-confidence predictions. In this paper, we propose to meta-learn the ensemble of epoch-wise empirical Bayes models (E3BM) to achieve robust predictions. "Epoch…
This paper tackles overfitting in CTR models by introducing Multi-Epoch learning with Data Augmentation.
problem Overfitting of the embedding layer in CTR models during multi-epoch training.
method Introduces Multi-Epoch learning with Data Augmentation (MEDA) framework to reduce overfitting and enhance performance.
result MEDA minimizes overfitting and achieves data augmentation through varied embedding spaces, improving performance without overfitting.
The paper analyzes how repeating epochs affects data scaling in linear regression.
problem Understanding how to scale data for multi-epoch training in linear regression.
method Theoretical analysis of effective reuse rate (E(K, N)) under strong convexity or Zipf-distributed data.
result The effective reuse rate E(K, N) plateaus at a problem-dependent value that grows with N, indicating diminishing marginal gains.
We propose Unbalanced GANs, which pre-trains the generator of the generative adversarial network (GAN) using variational autoencoder (VAE). We guarantee the stable training of the generator by preventing the faster convergence of the discriminator at early epochs. Furthermore, we balance between the generator and the d…
One epoch training yields certifiably robust models.
problem Vulnerability of machine learning models to adversarial attacks.
method Deterministic certification approach based on regularized loss.
result Certifiable robust models achieved in one epoch.
A method for faster neural architecture search using low-fidelity training.
problem Time-consuming evaluations in neural architecture search.
method Bayesian multi-fidelity method with knowledge distillation.
result Training for a few epochs with knowledge distillation leads to better architecture selection.
The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizerscs.LG
This paper identifies and analyzes the Epochal Sawtooth Phenomenon in training loss curves.
problem Training loss oscillations in adaptive gradient-based optimizers.
method Empirical analysis of Adam and other optimizers, focusing on parameters, batch size, data shuffling, and sample replacement.
result The Epochal Sawtooth Phenomenon (ESP) is a recurring pattern in training loss curves, arising from adaptive learning rate adjustments and data shuffling.
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…
Adaptive regularization prevents overfitting in large-scale sparse feature models.
problem Overfitting in models with large-scale sparse categorical features.
method Adaptive regularization of embedding layers' norm budget.
result Improves model performance within a single epoch and prevents multi-epoch performance degradation.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
Poly-view contrastive learning improves image representation learning.
problem Learning representations from multiple related views in image data.
method Developed new representation learning objectives for poly-view tasks using information maximization and sufficient statistics.
result Poly-view contrastive models trained for fewer epochs and with smaller batch sizes outperform models trained for more epochs and with larger batch sizes.
Deep learning is rapidly becoming a go-to tool for many artificial intelligence problems due to its ability to outperform other approaches and even humans at many problems. Despite its popularity we are still unable to accurately predict the time it will take to train a deep learning network to solve a given problem. T…
InstanceFlow visualizes classifier confusion over training epochs.
problem Limited model interpretability through aggregate performance measures.
method Dual-view visualization tool showing instance-level learning behavior.
result Allows temporal analysis of training process and instance-level performance.
Averaging recent model checkpoints speeds up training time.
problem Training large vision or language models is time-consuming.
method Average the weights of the k latest checkpoints.
result Speeds up training by dozens of epochs, saving up to 68 GPU hours.
While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that there exists a convex loss function for which the stability gap fo…
RAE improves image representation learning with simplified design choices.
problem Improving image representation learning using pretrained vision encoders.
method Generalized RAE formulation, complementary working mechanisms of RAE and REPA, and free CFG guidance.
result RAEv2 achieves state-of-the-art results with 10x faster convergence and less training time.
Automatic sleep staging has been often treated as a simple classification problem that aims at determining the label of individual target polysomnography (PSG) epochs one at a time. In this work, we tackle the task as a sequence-to-sequence classification problem that receives a sequence of multiple epochs as input and…
Paper proposes faster certified robust training methods with short warmup.
problem Certified robust training methods require long warmup schedules, making training costly.
method Proposes three improvements: new weight initialization, BN, and regularization.
result Achieves 65.03% verified error on CIFAR-10 and 82.36% on TinyImageNet with short warmup.
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
problem Understanding how Adam's implicit bias is affected by mini-batch size and momentum parameters.
method Theoretical framework to analyze mini-batch noise's impact on Adam's memory and bias.
result The magnitude of anti-regularization by memory depends on batch size and momentum hyperparameters.
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
Deep learning models can overfit noisy data without losing generalization.
problem Understanding the generalization of deep learning models in noisy data.
method Empirical investigation of epoch-wise double descent in fully connected neural networks trained on CIFAR-10 with 30% label noise.
result The model achieves strong re-generalization on test data after overfitting noisy training data, corresponding to a 'benign overfitting' state.
An Empirical Analysis of Proximal Policy Optimization with Kronecker-factored Natural Gradientscs.AI
In this technical report, we consider an approach that combines the PPO objective and K-FAC natural gradient optimization, for which we call PPOKFAC. We perform a range of empirical analysis on various aspects of the algorithm, such as sample complexity, training speed, and sensitivity to batch size and training epochs…
New insights into training machine learning models with momentum.
problem Lack of theoretical understanding on the generalization error of momentum-based methods.
method Analyzed modified momentum-based update rule (SGDEM) for smooth Lipschitz loss functions.
result SGDEM admits an upper-bound on the generalization error for smooth Lipschitz loss functions.
High fidelity behavior prediction of intelligent agents is critical in many applications. However, the prediction model trained on the training set may not generalize to the testing set due to domain shift and time variance. The challenge motivates the adoption of online adaptation algorithms to update prediction model…
Ordering examples in modular arithmetic training affects model performance.
problem Training efficiency and model performance in modular arithmetic.
method Varying example ordering while keeping other conditions constant.
result Adversarially structured ordering suppresses learning, while fixed-ordering strategies achieve high accuracy.
Label noise in adversarial training leads to robust overfitting, explained and mitigated.
problem Label noise in adversarial training causes robust overfitting.
method Proposed a method to automatically calibrate labels.
result Consistent performance improvements across various models and datasets.
In this paper we analyze, evaluate, and improve the performance of training generalized linear models on modern CPUs. We start with a state-of-the-art asynchronous parallel training algorithm, identify system-level performance bottlenecks, and apply optimizations that improve data parallelism, cache line locality, and …
Deep learning dynamics and NTK evolution studied through diverse measures.
problem Understanding the training dynamics of deep neural networks and their loss landscapes.
method Phenomenological analysis of training dynamics in multiple architectures and datasets.
result Training dynamics exhibit a chaotic initial transient followed by a stable phase, with the NTK evolving to match full network performance.
A deep learning model, named IITNet, is proposed to learn intra- and inter-epoch temporal contexts from raw single-channel EEG for automatic sleep scoring. To classify the sleep stage from half-minute EEG, called an epoch, sleep experts investigate sleep-related events and consider the transition rules between the foun…
Epoch-GDA achieves optimal convergence rate for SCSC min-max problems.
problem Solving stochastic min-max problems with strong convexity and strong concavity.
method Epoch-wise stochastic gradient descent ascent method (Epoch-GDA) without additional assumptions.
result Achieves the optimal rate of for the duality gap of general SCSC min-max problems.
How can we make machine learning provably robust against adversarial examples in a scalable way? Since certified defense methods, which ensure -robust, consume huge resources, they can only achieve small degree of robustness in practice. Lipschitz margin training (LMT) is a scalable certified defense, but it can als…
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
One of the biggest bottlenecks in a machine learning workflow is waiting for models to train. Depending on the available computing resources, it can take days to weeks to train a neural network on a large dataset with many classes such as ImageNet. For researchers experimenting with new algorithmic approaches, this is …
This research improves financial market predictions using LSTM networks.
problem Accurate real-time forecasting of financial time series.
method Sequentially trained many-to-one LSTMs with adaptive training epochs.
result Our approach maintains superior accuracy as predictions are made further in the future.
While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few …
This paper presents by simulation how approximate multipliers can be utilized to enhance the training performance of convolutional neural networks (CNNs). Approximate multipliers have significantly better performance in terms of speed, power, and area compared to exact multipliers. However, approximate multipliers have…
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…
Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the tas…
Large-scale distributed training of deep neural networks suffer from the generalization gap caused by the increase in the effective mini-batch size. Previous approaches try to solve this problem by varying the learning rate and batch size over epochs and layers, or some ad hoc modification of the batch normalization. W…
New method reduces deep learning training costs by approximating vector-jacobian products.
problem Efficiently training deep neural networks with reduced computational and memory costs.
method Randomized, unbiased approximations of vector-jacobian products during backpropagation.
result Validated potential for reducing deep learning training costs through unbiased estimates.
A new method selects clean samples to train DNNs with noisy labels.
problem Training deep neural networks with noisy labeled data.
method Adaptive k-set selection to choose clean samples at each epoch.
result The method guarantees performance with a theoretical bound on regret.
IDS improves RLHF by smoothing reward data, enhancing model performance.
problem Reward model performance degrades and overoptimization hinders true objective.
method Iterative Data Smoothing (IDS) updates model and data labels during each epoch.
result IDS outperforms traditional methods in RLHF.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcs.LG
Analyzes the generalization and training errors of the random feature model over time.
problem Understanding the temporal behavior of generalization and training errors in deep learning.
method Uses Cauchy complex integral representations and random matrix methods based on linear pencils.
result Analytical solution of the full time-evolution path of generalization and training errors.
Deep networks can interpolate noisy data without losing generalization.
problem Characterizing the relationship between interpolation and generalization in overparameterized deep networks.
method Analyzing the loss landscape of neural network functions over volumes around training data points, varying model parameters and training epochs.
result Loss sharpness in the input space follows a double descent, with large models predicting noisy targets over larger volumes around training data points.
Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes can improve the system scalability by reducing the communication-to-computation ratio, it may hurt the generalization ability of the models…
Traditionally, an artificial neural network (ANN) is trained slowly by a gradient descent algorithm such as the backpropagation algorithm since a large number of hyperparameters of the ANN need to be fine-tuned with many training epochs. To highly speed up training, we created a novel shallow 4-layer ANN called "Pairwi…