Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
Un-trained neural networks outperform trained methods in MRI reconstruction.
problem Accelerated MRI reconstruction with minimal training data.
method Variation of Deep Decoder without training data.
result Un-trained approach significantly outperforms other methods in reconstruction accuracy.
We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the k-th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory …
Meta-learning performance depends on train-validation split type.
problem Understanding the importance of train-validation split in meta-learning.
method Theoretical and experimental study comparing train-val and train-train methods.
result Train-train method can achieve strictly better excess loss in realizable cases.
New method reveals how training data influence diffusion model outputs.
problem Difficulty in assessing training data impact on diffusion model outputs.
method Use of ensembles trained on carefully engineered splits of training data to identify influential training examples.
result Demonstrated the viability of ensembles as generative models and validity of assessing influence.
Paper shows adversarial training can be fooled by new type of noise.
problem Adversarial training can be fooled by new types of noise.
method Designing ADVIN, a new type of inducing noise.
result ADVIN can degrade adversarial training robustness by 99.9%.
New MIP methods improve training of integer-valued neural networks.
problem Training integer-valued neural networks with limited data and resources.
method Formulated new MIP models to optimize training efficiency and handle more data.
result Significantly outperforms previous state-of-the-art methods in accuracy, training time, and data usage.
Free adversarial training reduces the generalization gap compared to vanilla method.
problem Improving generalization in adversarial training.
method Analysis of algorithmic stability in free adversarial training.
result Free adversarial training shows a lower generalization gap.
Making neural networks robust against adversarial inputs has resulted in an arms race between new defenses and attacks. The most promising defenses, adversarially robust training and verifiably robust training, have limitations that restrict their practical applications. The adversarially robust training only makes the…
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
Loss-guided training accelerates node embedding methods on graphs.
problem Training efficiency in graph learning methods with implicit positive examples.
method Dynamic adjustment of training distribution based on loss values.
result Significant acceleration in training and computation over static methods.
Paper explores fast adversarial training to improve robustness with less computation.
problem Efficiently defending against adversarial examples.
method Integrates simple self-attacks for faster training, focusing on overfitting recovery.
result Shows superior robust accuracy with reduced training time compared to strong adversarial training.
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
New method reconstructs significant parts of training data from neural networks.
problem Understanding and reconstructing training data from neural networks.
method Proposes a novel reconstruction scheme based on recent theoretical results about neural network training.
result Shows that a significant fraction of training data can be reconstructed from neural network parameters.
The paper studies the asymptotic behavior of adversarial training under ℓ∞-perturbation.
problem Theoretical guarantees for sparsity-recovery in adversarial training.
method Investigation of the asymptotic distribution of the adversarial training estimator in generalized linear models.
result The asymptotic distribution of the adversarial training estimator under ℓ∞-perturbation could have a positive probability mass at 0 when the true parameter is 0. This paper introduces 'General Cyclical Training' for neural networks.
problem Improving training efficiency and performance of neural networks.
method Cyclical training phases with varying hyperparameters, batch sizes, loss functions, and data augmentation.
result Cyclical weight decay, softmax temperature, and gradient clipping enhance model accuracy.
Speeds up training and inference by pruning entire channels before training.
problem Training and inference speed in deep neural networks.
method Structured pruning applied before training, focusing on removing entire channels and hidden units.
result 2x speedup in training and 3x speedup in inference.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
Theoretical study shows adversarial training improves robustness in deep learning models.
problem Ensuring robustness in pre-trained deep learning models.
method Theoretical analysis of adversarial training and feature purification in two-layer neural networks.
result Adversarial training leads to feature purification, making models more robust to attacks.
Training dynamics reveal neural network complexity.
problem Understanding the complexity of neural networks during training.
method Relating the Lipschitz constant and training behavior of neural networks.
result Steady training and bounded complexity in far regions of input space.
LIIT uses large deviations to train neural networks faster.
problem Training neural networks requires large amounts of data, leading to inefficiencies.
method LIIT uses large deviations principle to generate and update training samples.
result LIIT achieves similar classification performance with smaller, more representative samples.
Empirical study shows GANs overfit and drop modes when training is deterministic.
problem Understanding overfitting and mode drop in GAN training.
method Empirical analysis of GAN training with and without stochasticity.
result GANs overfit and drop modes when training is deterministic.
Joint training improves model accuracy by selectively using privileged information.
problem Two-stage training can lead to model failure with noisy privileged information.
method Joint training of two models to use privileged information selectively.
result Joint training outperforms two-stage baselines on synthetic and real-world tasks.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Adversarial training, in which a network is trained on adversarial examples, is one of the few defenses against adversarial attacks that withstands strong attacks. Unfortunately, the high cost of generating strong adversarial examples makes standard adversarial training impractical on large-scale problems like ImageNet…
New method improves adversarial training efficiency and robustness.
problem High computational costs and lack of stability in adversarial training.
method Backward smoothing for randomized smoothing of random initialization.
result Our method achieves similar model robustness as state-of-the-art methods but with significantly less training time.
Recent work on discriminative segmental models has shown that they can achieve competitive speech recognition performance, using features based on deep neural frame classifiers. However, segmental models can be more challenging to train than standard frame-based approaches. While some segmental models have been success…
Improved GAN training speed and quality with FastGAN.
problem Slower convergence and less expressive models in GAN training.
method Adversarial training with FastGAN algorithm.
result Better generation quality with less training time.
Predict training time without training using a linearized model.
problem Predicting the training time of a deep network without actual training.
method Approximate training dynamics with a linearized model and solve a low-dimensional SDE in function space.
result Predict training time within 20% error margin and 30-45x reduction in cost.
We present an unsupervised training approach for a neural network-based mask estimator in an acoustic beamforming application. The network is trained to maximize a likelihood criterion derived from a spatial mixture model of the observations. It is trained from scratch without requiring any parallel data consisting of …
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
GP-TS optimizes TLM pre-training hyperparameters efficiently.
problem Resource inefficiency in TLM pre-training.
method Bayesian optimization with Thompson sampling and Gaussian process.
result GP-TS achieves lower MLM loss in fewer epochs.
He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. …
Since the creation of Generative Adversarial Networks (GANs), much work has been done to improve their training stability, their generated image quality, their range of application but nearly none of them explored their self-training potential. Self-training has been used before the advent of deep learning in order to …
Adversarial training effectiveness varies widely due to inconsistent training settings.
problem Variability in adversarial training effectiveness due to inconsistent training settings.
method Comprehensive evaluation of 10+ adversarial training methods and their hyperparameters.
result Basic training settings like weight decay can significantly impact adversarial robustness.
New algorithm efficiently trains machine learning models to atomic forces data.
problem Efficiently training machine learning models to large amounts of force data.
method Developed an efficient algorithm for training machine learning models to all available force data.
result Training to all available force data is only a few times more expensive than training to energies alone.
Paper proposes a new method to measure model sensitivity using final model only.
problem Understanding model behavior using only the final trained model.
method Reframe TDA as measuring sensitivity, propose further training as gold standard, unify gradient-based methods.
result Gradient-based methods approximate further training but vary in quality.
Adversarial training leads to clean data generalization with significant robust overfitting gap.
problem Significant robust generalization gap in adversarial training.
method Two theoretical views: representation complexity and training dynamics.
result ReLU nets with O(ND) extra parameters can achieve CGRO. Paper tackles model vulnerabilities by reconstructing training data.
problem Reconstructing training data from model parameters poses a security risk.
method Developed a mathematical framework and score matching method for both Bayesian and non-Bayesian models.
result First score matching framework for reconstructing data in Bayesian models.
This paper uses SDEs to analyze GANs training and long-run behavior.
problem Understanding the training process and long-run behavior of GANs.
method Established SDE approximations for GANs training and analyzed long-run behavior via invariant measures.
result The long-run behavior of GANs training can be studied via the invariant measures of its SDE approximations.
A novel solve-training framework is proposed to train neural network in representing low dimensional solution maps of physical models. Solve-training framework uses the neural network as the ansatz of the solution map and train the network variationally via loss functions from the underlying physical models. Solve-trai…
A new training method improves stability and generalization of DeepONets.
problem Training deep operator networks (DeepONets) is challenging due to nonconvex and nonlinear nature.
method Two-step training method: first train trunk network, then branch network. Introduced Gram-Schmidt orthonormalization.
result Generalization error estimate and numerical examples demonstrating effectiveness.
Adversarial training improves linear regression solutions, revealing sparsity and abrupt interpolation.
problem Adversarial attacks on linear regression models.
method Formulated as a convex problem, adversarial training is used to find robust solutions that are sparse and interpolate data.
result Adversarial training with small disturbances gives the solution with the minimum-norm that interpolates the training data, revealing abrupt transition into interpolation.
In supervised machine learning for author name disambiguation, negative training data are often dominantly larger than positive training data. This paper examines how the ratios of negative to positive training data can affect the performance of machine learning algorithms to disambiguate author names in bibliographic …
This work reduces computation cost for on-device CNN training.
problem High computation cost during on-device CNN training.
method Self-supervised instance filtering and error map pruning.
result Substantial computation saving without significant accuracy loss.
A new scaling law predicts optimal batch size for training models.
problem Finding the optimal batch size for training models efficiently.
method Proposed a three-term scaling law that considers model size, training data, training steps, and batch size.
result The three-term law accurately recovers the optimal batch size and can be robustly fit with fewer training runs.
Survey on reducing deep neural network training complexity.
problem Efficient training of large deep neural networks.
method Pruning and freezing parts of the network during training.
result Dimensionality reduction improves training efficiency.