Unbalanced GANs stabilize GAN training by pre-training the generator with VAE.
problem Stable training of GANs to avoid mode collapses and improve image quality.
method Pre-train GAN generator with VAE, balance generator and discriminator training, prevent discriminator's early convergence.
result Unbalanced GANs reduce mode collapses and outperform ordinary GANs in stability, convergence, and image quality.
The paper uses geometry to understand how neural networks learn.
problem Understanding the learning capability of neural networks.
method Statistical and differential geometric analysis of neural networks performing simple regression.
result Neural networks with higher generalization capability have a slower convergence rate.
Paper optimizes RL algorithm selection for better performance.
problem Choosing the best RL algorithm for each episode.
method Epochal Stochastic Bandit Algorithm Selection (ESBAS) and its adaptation SSBAS.
result ESBAS and SSBAS outperform individual RL algorithms in various tasks.
We show that parametric models trained by a stochastic gradient method (SGM) with few iterations have vanishing generalization error. We prove our results by arguing that SGM is algorithmically stable in the sense of Bousquet and Elisseeff. Our analysis only employs elementary tools from convex and continuous optimizat…
New method improves generalization of SGD with momentum.
problem Lack of theoretical understanding of generalization error in momentum-based SGD.
method Introduced SGD with early momentum (SGDEM) and analyzed its generalization properties.
result SGDEM can train machine learning models with a guarantee for generalization.
Proposes a new GAN framework using adversarial dropout to improve sample diversity and stability.
problem Mode collapse in GANs.
method Adversarial dropout in a dynamic ensemble of discriminators.
result Promotes sample diversity and stabilizes training.
Paper proposes E3BM for robust few-shot learning with few examples.
problem Few-shot learning with limited data leads to poor model performance.
method Meta-learn ensemble of epoch-wise empirical Bayes models (E3BM).
result Top performance achieved using epoch-dependent transductive hyperprior learner.
IITNet learns sleep stages from raw EEG using sub-epoch features and temporal contexts.
problem Automatic sleep scoring from raw single-channel EEG.
method IITNet uses a residual neural network to extract sub-epoch features and bidirectional LSTM to capture intra- and inter-epoch temporal contexts.
result IITNet achieves comparable performance to state-of-the-art methods, especially with longer sequence lengths.
New insights into training machine learning models with momentum.
problem Lack of theoretical understanding on the generalization error of momentum-based methods.
method Analyzed modified momentum-based update rule (SGDEM) for smooth Lipschitz loss functions.
result SGDEM admits an upper-bound on the generalization error for smooth Lipschitz loss functions.
Training for just one epoch improves model performance without overfitting.
problem Training large unsupervised models is costly and time-consuming.
method Train models for only one epoch, adjust model size and iterations.
result Performance significantly improved with one epoch training, especially for large models.
Epoch-GDA achieves optimal convergence rate for SCSC min-max problems.
problem Solving stochastic min-max problems with strong convexity and strong concavity.
method Epoch-wise stochastic gradient descent ascent method (Epoch-GDA) without additional assumptions.
result Achieves the optimal rate of O(1/T) for the duality gap of general SCSC min-max problems. This paper tackles overfitting in CTR models by introducing Multi-Epoch learning with Data Augmentation.
problem Overfitting of the embedding layer in CTR models during multi-epoch training.
method Introduces Multi-Epoch learning with Data Augmentation (MEDA) framework to reduce overfitting and enhance performance.
result MEDA minimizes overfitting and achieves data augmentation through varied embedding spaces, improving performance without overfitting.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
The paper analyzes how repeating epochs affects data scaling in linear regression.
problem Understanding how to scale data for multi-epoch training in linear regression.
method Theoretical analysis of effective reuse rate (E(K, N)) under strong convexity or Zipf-distributed data.
result The effective reuse rate E(K, N) plateaus at a problem-dependent value that grows with N, indicating diminishing marginal gains.
One epoch training yields certifiably robust models.
problem Vulnerability of machine learning models to adversarial attacks.
method Deterministic certification approach based on regularized loss.
result Certifiable robust models achieved in one epoch.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
Adaptive regularization prevents overfitting in large-scale sparse feature models.
problem Overfitting in models with large-scale sparse categorical features.
method Adaptive regularization of embedding layers' norm budget.
result Improves model performance within a single epoch and prevents multi-epoch performance degradation.
Efficiently trains robust models with faster and more stable interval bounds.
problem Training verifiably robust models against adversarial attacks.
method Integrates interval arithmetic and an additional cost function term to keep hidden layer bounds small.
result Comparable or better results achieved with fewer training iterations and more stability.
Historical economic growth in Asia (excluding Japan) is analysed. It is shown that Unified Growth Theory is contradicted by the data, which were used (but not analysed) during the formulation of this theory. Unified Growth Theory does not explain the mechanism of economic growth. It explains the mechanism of Malthusian…
The paper detects special epochs in oil price data using wavelet analysis and multi-fractional modeling.
problem Detecting regime shifts in oil price data.
method Wavelet-based decomposition, multi-fractional modeling, and joint estimation of Hurst exponent and volatility.
result Special epochs emerge as a result of regime shifts and switching in oil price data.
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…
SeqSleepNet tackles automatic sleep staging as a sequence-to-sequence problem.
problem Automatic sleep staging as a sequence-to-sequence classification problem.
method End-to-end hierarchical recurrent neural network (SeqSleepNet) with filterbank and attention-based recurrent layers.
result SeqSleepNet achieves high accuracy (87.1% overall accuracy, 83.3% macro F1-score, 0.815 Cohen's kappa) on a publicly available dataset.
Topic models have proven to be a useful tool for discovering latent structures in document collections. However, most document collections often come as temporal streams and thus several aspects of the latent structure such as the number of topics, the topics' distribution and popularity are time-evolving. Several mode…
A method for faster neural architecture search using low-fidelity training.
problem Time-consuming evaluations in neural architecture search.
method Bayesian multi-fidelity method with knowledge distillation.
result Training for a few epochs with knowledge distillation leads to better architecture selection.
A new learning scheme improves model efficiency and performance.
problem Characterizing correlation between batch-level and global data distributions.
method Epoch-evolving Gaussian Process Guided Learning (GPGL) scheme with context labels and triangle consistency loss.
result Significantly outperforms existing models on mainstream datasets.
This paper identifies and analyzes the Epochal Sawtooth Phenomenon in training loss curves.
problem Training loss oscillations in adaptive gradient-based optimizers.
method Empirical analysis of Adam and other optimizers, focusing on β parameters, batch size, data shuffling, and sample replacement. result The Epochal Sawtooth Phenomenon (ESP) is a recurring pattern in training loss curves, arising from adaptive learning rate adjustments and data shuffling.
Random shuffling outperforms SGD after a few epochs in optimization problems.
problem Proving RandomShuffle converges faster than SGD after a finite number of epochs.
method Introduced RandomShuffle and proved its convergence rate under strong convexity and smoothness conditions.
result RandomShuffle converges faster than SGD after a reasonable number of epochs, with a rate of O(1/T^2 + n^3/T^3).
Poly-view contrastive learning improves image representation learning.
problem Learning representations from multiple related views in image data.
method Developed new representation learning objectives for poly-view tasks using information maximization and sufficient statistics.
result Poly-view contrastive models trained for fewer epochs and with smaller batch sizes outperform models trained for more epochs and with larger batch sizes.
Optimal SGD rates achieved with shuffling, covering non-convex and convex cases.
problem Optimizing finite-sum optimization problems with shuffling strategies.
method RandomShuffle and SingleShuffle algorithms for SGD, analyzing convergence rates.
result Minimax optimal convergence rates established, generalizing to non-convex costs.
Framework uses CNNs for joint sleep stage classification and prediction.
problem Diagnosing and treating sleep disorders requires accurate sleep stage identification.
method Joint classification-and-prediction CNN framework based on a novel architecture.
result Framework achieves 82.3% and 83.6% accuracy on two public datasets.
New RL algorithm for POMDPs using spectral methods.
problem Learning POMDPs where interactions change future observations.
method Epoch-based spectral decomposition for parameter learning, followed by optimal policy optimization.
result Order-optimal regret bound with efficient scaling.
Random matrix theory improves financial market analysis by smoothing out noise.
problem Choosing an appropriate epoch for computing empirical cross-correlation matrices in financial markets.
method Power mapping to apply non-linear distortion to short epoch correlation matrices, controlling noise and removing degeneracy of zero eigenvalues.
result Interesting properties of eigenvalue spectra are found in simulated and empirical return matrices.
RAE improves image representation learning with simplified design choices.
problem Improving image representation learning using pretrained vision encoders.
method Generalized RAE formulation, complementary working mechanisms of RAE and REPA, and free CFG guidance.
result RAEv2 achieves state-of-the-art results with 10x faster convergence and less training time.
New active learning framework for multiclass classification beyond realizability assumption.
problem Active learning in non-realizable settings with convex model classes.
method Surrogate risk minimization, epoch-based fitting, aggregation of models.
result Achieves label and sample complexity comparable to prior work in non-realizable settings.
Quantized Epoch-SGD reduces communication in distributed learning.
problem High communication cost in SGD-based distributed learning.
method Quantizes parameters with variance reduction for efficient distributed learning.
result QESGD achieves better performance with less communication compared to other methods.
Within a statistical learning setting, we propose and study an iterative regularization algorithm for least squares defined by an incremental gradient method. In particular, we show that, if all other parameters are fixed a priori, the number of passes over the data (epochs) acts as a regularization parameter, and prov…
It is hypothesized that price charts can be empirically decomposed into two components as random and non random. The non random component, which can be treated as approximately regular behavior of the prices (trend) in an epoch, is a geometric line. Thus, the random component fluctuates around the non random component …
Deep learning models can overfit noisy data without losing generalization.
problem Understanding the generalization of deep learning models in noisy data.
method Empirical investigation of epoch-wise double descent in fully connected neural networks trained on CIFAR-10 with 30% label noise.
result The model achieves strong re-generalization on test data after overfitting noisy training data, corresponding to a 'benign overfitting' state.
InstanceFlow visualizes classifier confusion over training epochs.
problem Limited model interpretability through aggregate performance measures.
method Dual-view visualization tool showing instance-level learning behavior.
result Allows temporal analysis of training process and instance-level performance.
Adam optimizer's bias is influenced by mini-batch size and momentum hyperparameters.
problem Understanding how Adam's implicit bias is affected by mini-batch size and momentum parameters.
method Theoretical framework to analyze mini-batch noise's impact on Adam's memory and bias.
result The magnitude of anti-regularization by memory depends on batch size and momentum hyperparameters.
Study finds common poetic themes across languages over time.
problem Understanding thematic evolution in different poetic traditions.
method Applied Latent Dirichlet Allocation (LDA) to poetry corpora of four languages.
result Identified common themes and their temporal trends across poetic traditions.
Paper proposes an online adaptation algorithm for improving model performance.
problem Improving model fidelity in real-time for domain shift and time variance.
method Extended Kalman Filter with Exponential Moving Average and Dynamic Multi-Epoch strategy.
result Proposed algorithm outperforms existing methods in experiments.
This study examines how hidden layers affect CNN performance on handwritten digit recognition.
problem Impact of hidden layers on CNN performance in handwritten digit recognition.
method Applied CNN with varying hidden layers on MNIST dataset, trained with stochastic gradient and backpropagation, tested with feedforward.
result Variations in accuracies for different hidden layers and epochs.
New approach predicts deep learning training times more accurately.
problem Inaccurate prediction of deep learning training times.
method Trains a deep learning network to predict execution time for parts of a network.
result Can predict execution times for unseen scenarios and hardware choices.
ADASS selects adaptive subsets for SGD training acceleration.
problem Fixed sample size in SGD limits training efficiency.
method ADASS selects adaptive subsets based on Lipschitz constants.
result ADASS achieves comparable accuracy with full training set.
New algorithm reduces regret from sqrt(T) to polylog(T) in stochastic contextual linear bandits.
problem Achieving logarithmic regret in stochastic contextual linear bandits.
method Low Regret Stochastic Contextual Bandits ( exttt{LR-SCB}) algorithm, exploiting stochastic contexts and parameter estimation.
result Logarithmic regret (polylog(T)) achieved, improving over sqrt(T) lower bound.
The paper uses LSMC to price capped American options with time-dependent caps.
problem Pricing American options with time-capped features.
method Least Squares Monte Carlo (LSMC) method.
result The LSMC method converges to the true price as discretization step and number of trajectories approach limits.
Efficiently searches for optimal neural architecture and hyperparameters.
problem Separate tuning of architecture and hyperparameters leads to suboptimal results.
method Combines Bayesian optimization and Hyperband for joint search.
result Joint search yields better results with fewer epochs.