Neural network accuracy improves with denser training samples.
problem Improving neural network accuracy on unseen test samples.
method Bounding empirical training error smoothed across activation regions and using it to discard high-risk test samples.
result Discarding high-risk test samples based on error bounds improves prediction accuracy by up to 20%.
Dynamic Influence Tracker measures changing sample importance during model training.
problem Static influence measurements during training overlook how sample importance varies over time.
method Dynamic Influence Tracker (DIT) captures time-varying sample influence across arbitrary time windows.
result DIT reveals distinct learning phases with shifting priorities and detects corrupted samples more efficiently.
The study proves sampling-based GNNs can approximate training on full graphs with small subgraphs.
problem Training Graph Neural Networks (GNNs) on large graphs is computationally expensive.
method Theoretical framework using graph local limits to prove approximation of GNN training on small samples.
result Parameters learned from sampling-based GNNs on small subgraphs are close to those on full graphs.
The paper analyzes a neural network two-sample test using kernel analysis.
problem Determining if two datasets come from the same distribution.
method Time-analysis on a neural tangent kernel (NTK) two-sample test, extending to realistic neural network dynamics.
result Training times needed to detect deviations are well-separated in null and alternative hypothesis scenarios.
Improves sample quality of generative models using energy-based methods.
problem Low sample quality in generative models.
method Constructs an energy function on latent space, trains an energy-based model, and generates improved samples.
result Significant improvement in sample quality with minimal computational overhead.
LIIT uses large deviations to train neural networks faster.
problem Training neural networks requires large amounts of data, leading to inefficiencies.
method LIIT uses large deviations principle to generate and update training samples.
result LIIT achieves similar classification performance with smaller, more representative samples.
Bayesian Attention Networks compress data by focusing on key training samples.
problem Lossless data compression for efficiency.
method Bayesian Attention Networks with attention factors and latent space.
result Efficient prediction using a few correlated training samples.
Paper shows pre-training and transfer learning reduce sample complexity for neural networks.
problem Training high-dimensional supervised learning with limited labeled data.
method Study of single-layer neural networks via online stochastic gradient descent, considering concept shift.
result Pre-training and transfer learning reduce sample complexity by polynomial factors under general assumptions.
Adversarial training is by far the most successful strategy for improving robustness of neural networks to adversarial attacks. Despite its success as a defense mechanism, adversarial training fails to generalize well to unperturbed test set. We hypothesize that this poor generalization is a consequence of adversarial …
Adaptive sampling improves deep learning efficiency by focusing on informative samples.
problem Expensive annotations and loss of sample informativeness in deep learning.
method Adaptive latent space sampling based on hardness-aware strategy.
result The proposed framework outperforms random sampling on various datasets.
Adaptive teacher improves sample efficiency and mode coverage in sampling tasks.
problem Efficient exploration and mode coverage in sampling tasks with amortized inference.
method An adaptive training distribution (teacher) guides the training of the primary sampler (student).
result Improves sample efficiency and mode coverage across various tasks.
The paper optimizes training samples for image denoising across different noise levels.
problem Training a denoiser for all noise levels with uniform sample distribution.
method Derives a dual ascent algorithm for optimal sampling distribution.
result The algorithm converges to an optimal sampling distribution for deep neural networks.
Improves GAN sample quality by refining the training dataset.
problem GANs generate unrealistic samples outside the data manifold.
method Instance selection to improve sample quality and reduce training time.
result Significantly reduces training time and improves sample fidelity.
In this work, we investigate a novel training procedure to learn a generative model as the transition operator of a Markov chain, such that, when applied repeatedly on an unstructured random noise sample, it will denoise it into a sample that matches the target distribution from the training set. The novel training pro…
Estimating the relative importance of each sample in a training set has important practical and theoretical value, such as in importance sampling or curriculum learning. This kind of focus on individual samples invokes the concept of sample-wise learnability: How easy is it to correctly learn each sample (cf. PAC learn…
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…
Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, whic…
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
problem Challenges in assessing variance reduction during DNN training using importance sampling.
method Proposes a method for estimating variance reduction using minibatches sampled under importance sampling.
result Demonstrates consistent reduction in variance, improved training efficiency, and enhanced model accuracy.
ES improves training efficiency by dynamically selecting data samples.
problem Efficiently selecting informative data samples for faster learning.
method Evolved Sampling (ES) dynamically selects data samples based on loss dynamics and differences.
result ES achieves significant training acceleration without compromising model performance.
We study the problem of semi-supervised singing voice separation, in which the training data contains a set of samples of mixed music (singing and instrumental) and an unmatched set of instrumental music. Our solution employs a single mapping function g, which, applied to a mixed sample, recovers the underlying instrum…
A new method for generating samples without training, using smoothed score matching.
problem Generating samples efficiently and without training.
method Moment-matched score-smoothed overdamped Langevin dynamics (MM-SOLD).
result The method enables fast, robust, training-free sampling with competitive sample fidelity and diversity.
We correct for sampling bias in training models to improve real-world performance.
problem Sampling bias causes discrepancies between lab and real-world model performance.
method Bayesian risk minimization and derived bias-corrected loss functions.
result Our approach integrates seamlessly into current learning paradigms and improves model performance.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Dynamic sample pruning speeds up spatio-temporal forecasting models.
problem Training deep learning models on large, redundant datasets is computationally expensive.
method Dynamic sample pruning based on real-time learning state.
result Significant acceleration of training speed with improved performance.
Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by pr…
New method improves sampling from noisy energy models.
problem Training and sampling challenges in Energy-Based Models.
method Pseudo-Gibbs sampling with moment matching.
result Effective sampling from clean model using noisy DSM-trained model.
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learnin…
A training-free method for conditional sampling using flow matching.
problem Weight degeneracy in high-dimensional importance sampling.
method Sequential Monte Carlo with resampling and stochastic flow.
result Significantly outperforms existing methods on MNIST and CIFAR-10.
Paper proposes a new method for sampling from complex distributions.
problem Sampling from unnormalised density functions in complex distributions.
method Combines amortised and particle-based methods with reinforcement learning.
result Improves sampling from complex distributions compared to existing methods.
The study examines how shallow neural nets converge to training samples or manifold points during diffusion.
problem Understanding when and how shallow neural nets converge to training samples or manifold points during diffusion.
method Analysis of shallow ReLU neural network denoisers trained with minimal ℓ 2 \ell^2 ℓ 2 norm, comparing score flow and diffusion flow. result Probability flow converges to training points, sums of training points, or manifold points, depending on the diffusion time scheduler.
New study shows neural networks need many samples for training.
problem How much data is needed to train a ReLU feed-forward neural network?
method Theoretical and empirical analysis of ReLU feed-forward neural networks.
result Generalization error scales at 1 / n 1/\sqrt{n} 1/ n in sample size n n n . Modern applications and progress in deep learning research have created renewed interest for generative models of text and of images. However, even today it is unclear what objective functions one should use to train and evaluate these models. In this paper we present two contributions. Firstly, we present a critique o…
Unified framework for adaptive connection sampling in GNNs improves performance and robustness.
problem Over-smoothing and over-fitting in deep GNNs.
method Adaptive connection sampling trained jointly with GNN model parameters.
result Adaptive connection sampling mathematically equivalent to Bayesian GNNs approximation.
Improves generalization with few samples using a new regularization method.
problem Training deep neural networks with limited data leads to overfitting.
method Sample-based regularization (SBR) to improve generalization without relying on source model knowledge.
result SBR outperformed existing methods in various configurations.
New method generates novel samples from closed-form diffusion models.
problem Closed-form SGMs memorize training data and cannot generate novel samples.
method Explicitly smooth closed-form score, use nearest-neighbor estimator.
result Efficient method generates novel samples without training.
IMM generates high-quality samples in few steps with stable training.
problem Slow inference and instability in generating high-quality samples using diffusion models and Flow Matching.
method Inductive Moment Matching (IMM) is a new generative model for one- or few-step sampling with a single-stage training procedure.
result IMM achieves state-of-the-art 2-step FID of 1.98 on CIFAR-10 for a model trained from scratch.
SIFT reduces training time by selecting samples with approximate losses.
problem Reducing training time by selecting samples with large approximate losses.
method Developed SIFT which uses early exiting to obtain approximate losses with intermediate layer representations for sample selection.
result SIFT achieves significant gains in training time and number of backpropagation steps without optimized implementation.
Paper adapts diffusion sampler training for faster convergence and better sampling.
problem Training limitations in diffusion samplers.
method Decouples generation and destruction variances, learns both as unconstrained Gaussians.
result Training both processes leads to faster convergence and improved sampling quality.
The paper tackles sampling biases by ensuring minority groups are adequately represented in training data.
problem Sampling biases in training data lead to algorithmic biases in machine learning systems.
method The paper presents adaptive sampling methods to determine if it's possible to assemble a representative dataset from given data sources.
result The methods presented can determine with high confidence if a representative dataset can be assembled from given data sources.
The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-step-ahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of th…
SapAugment learns adaptive augmentation policies for better model training.
problem Fixed data augmentation methods often apply the same augmentation to all samples, ignoring sample difficulty.
method SapAugment adapts augmentation parameters based on training loss, learning a sample-adaptive policy.
result SapAugment achieves up to 21% relative reduction in word error rate on LibriSpeech dataset.
Improved sample complexity for training diffusion models.
problem How many samples are needed to train an accurate diffusion model?
method Analyzing the sample complexity of training diffusion models using neural networks.
result Exponential improvement in the dependence on Wasserstein error and depth, along with improved dependencies on other parameters.
Deriving and applying Proximal Policy Optimization to GFlowNets for efficient training of discrete sampling policies
problem Training stochastic policies to sample from structured discrete probability distributions
method Deriving policy gradient algorithms for GFlowNets and applying Proximal Policy Optimization
result Improved convergence speed and data efficiency compared to standard GFlowNet training objectives
DRSSS method reduces model training costs by eliminating unsafe samples.
problem Reducing storage and training costs for customized models across different environments.
method Combines DR optimization and SSS for covariate shift.
result Models trained on reduced dataset perform similarly to those on full dataset.
Neighborhood sampling affects graph neural network training outcomes.
problem Understanding the impact of neighborhood sampling on graph neural network training.
method Theoretical analysis using neural tangent kernels and Gaussian processes.
result Posterior covariance differs for different neighborhood sampling approaches, indicating no dominant approach.
A new method trains and samples from energy-based models using diffusion recovery likelihood.
problem Training and sampling high-dimensional datasets with energy-based models is challenging.
method Trains EBMs with a diffusion recovery likelihood method, maximizing conditional probabilities of data at different noise levels.
result Generates high-fidelity images with low FID and inception scores, and accurately estimates normalized data density.
Paper extends nonparametric regression bounds for dependent β \beta β -mixing samples.
problem Analyzing error in nonparametric regression with dependent data.
method Extends uniform deviation inequalities from independent to dependent β \beta β -mixing samples. result Derives generalization bounds for nonparametric regression with dependent data.
This paper introduces a neural sampler for scalable sampling from complex distributions.
problem Efficiently sampling from high-dimensional un-normalized distributions.
method Neural implicit sampler trained with KL and Fisher divergence methods.
result The neural sampler generates large batches of samples with low computational costs.