This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
Paper proposes E3BM for robust few-shot learning with few examples.
problem Few-shot learning with limited data leads to poor model performance.
method Meta-learn ensemble of epoch-wise empirical Bayes models (E3BM).
result Top performance achieved using epoch-dependent transductive hyperprior learner.
Deep learning models can overfit noisy data without losing generalization.
problem Understanding the generalization of deep learning models in noisy data.
method Empirical investigation of epoch-wise double descent in fully connected neural networks trained on CIFAR-10 with 30% label noise.
result The model achieves strong re-generalization on test data after overfitting noisy training data, corresponding to a 'benign overfitting' state.
Label noise in adversarial training leads to robust overfitting, explained and mitigated.
problem Label noise in adversarial training causes robust overfitting.
method Proposed a method to automatically calibrate labels.
result Consistent performance improvements across various models and datasets.
Improves early stopping in deep networks by adjusting stepsizes.
problem Epoch-wise double descent in deep networks.
method Analytical and empirical study of bias-variance tradeoffs in different network layers.
result Eliminating epoch-wise double descent through adjusting stepsizes of different layers improves early stopping performance.
Analyzes the generalization and training errors of the random feature model over time.
problem Understanding the temporal behavior of generalization and training errors in deep learning.
method Uses Cauchy complex integral representations and random matrix methods based on linear pencils.
result Analytical solution of the full time-evolution path of generalization and training errors.
New insights into double descent phenomenon in neural networks.
problem Understanding the double descent behavior in deep learning models.
method Linear teacher-student setup and tools from statistical physics.
result Distinct features are learned at different scales, leading to epoch-wise double descent.
Unified analysis of generalization curves in large models using gradient flow.
problem Analyzing generalization error curves in simple learning models.
method Gradient flow in the Gaussian covariate model, using random matrix theory.
result Unified understanding of multiple descent structures in learning curves.
Epoch-GDA achieves optimal convergence rate for SCSC min-max problems.
problem Solving stochastic min-max problems with strong convexity and strong concavity.
method Epoch-wise stochastic gradient descent ascent method (Epoch-GDA) without additional assumptions.
result Achieves the optimal rate of O ( 1 / T ) O(1/T) O ( 1/ T ) for the duality gap of general SCSC min-max problems. Double descent found in DRL, improving generalization with model capacity.
problem Generalization in over-parameterized DRL models.
method Actor-Critic framework, Policy Entropy metric.
result Policy Entropy significantly reduces as model capacity increases, indicating improved generalization.
Study shows double and triple descent in unsupervised autoencoders, improving performance in various tasks.
problem Exploring the phenomenon of double descent in unsupervised learning.
method Analytical demonstration and extensive experiments on synthetic and real datasets.
result Over-parameterized unsupervised autoencoders exhibit double and triple descent, enhancing performance in downstream tasks.
Deep neural net improves sleep stage classification from raw EEG.
problem Improving accuracy of sleep stage classification from raw EEG data.
method 34-layer deep residual ConvNet architecture trained on raw single channel EEG data.
result Proposed network outperforms state-of-the-art methods in sleep staging accuracy.
Deep networks can interpolate noisy data without losing generalization.
problem Characterizing the relationship between interpolation and generalization in overparameterized deep networks.
method Analyzing the loss landscape of neural network functions over volumes around training data points, varying model parameters and training epochs.
result Loss sharpness in the input space follows a double descent, with large models predicting noisy targets over larger volumes around training data points.
Automatic sleep staging has been often treated as a simple classification problem that aims at determining the label of individual target polysomnography (PSG) epochs one at a time. In this work, we tackle the task as a sequence-to-sequence classification problem that receives a sequence of multiple epochs as input and…
Proposes a new method for GNNs that avoids iterative node state convergence.
problem Iterative computation of node states in GNNs is inefficient and requires many epochs.
method Constrained optimization in the Lagrangian framework to learn transition function and node states simultaneously.
result The proposed method compares favorably with existing models on various benchmarks.
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogeneous environment, while asynchronous algorithms using a parameter server suffer from 1) communication bottleneck at parameter servers when wo…
LGD breaks the chicken-and-egg loop in adaptive SGD by using LSH sampling.
problem Challenging per-iteration cost of adaptive gradient sampling.
method Locality Sensitive Hashing (LSH) sampled Stochastic Gradient Descent (LGD).
result Superior and faster gradient estimation with similar per-iteration cost.
TOFU-POV tackles partially observed linear bandits, achieving sublinear regret with low-dimensional action vectors.
problem Stochastic linear bandits with partially observed actions in settings like recommendation and healthcare.
method TOFU-POV estimates latent action subspace, imputes missing actions, and runs OFUL in low-dimensional coordinates.
result TOFU-POV achieves T \sqrt{T} T regret scaling with intrinsic subspace dimension, improving upon natural baselines. This paper tackles computational bottlenecks in federated learning on mobile devices.
problem Computationally heterogeneous mobile devices hinder federated learning efficiency.
method Proposes efficient algorithms to schedule mobile devices based on data heterogeneity.
result Achieves up to 100x speedup and 7% accuracy gain in federated learning.