MixUp explained as a special adversarial training technique.
problem Improving model robustness against adversarial attacks.
method Introducing directional adversarial training (DAT) and proving MixUp's equivalence to a specific subclass of DAT.
result The family of Untied MixUp schemes is equivalent to the entire class of DAT schemes and can improve upon MixUp.
Paper compares training and sampling schemes for character-level RNNs.
problem Optimizing training and sampling schemes for character-level RNNs.
method Tested various training and sampling schemes on different datasets and architectures.
result The choice of training and sampling scheme affects stability, performance, and implementation effort.
Unified framework for supervised classification with diverse training data.
problem Handling different types of training data for supervised classification.
method Generalized robust risk minimization (GRRM) with probabilistic transformations.
result GRRM can handle various training data types and new supervision schemes.
Study different masking schemes for a universal marginaliser.
problem Understand how well a neural approximator learns conditional distributions.
method Compare networks trained with various masking schemes.
result Neural approximators perform differently based on the masking scheme.
Neural Galerkin schemes use active learning to solve high-dimensional equations.
problem Inaccurate function approximations in high dimensions with limited training data.
method Neural Galerkin schemes based on deep learning with active learning for high-dimensional PDEs.
result Active data collection improves the numerical solution of high-dimensional equations.
A novel beam training scheme optimizes multi-hop THz communications with up to 75% performance gain.
problem Optimizing beam training for multi-hop THz communications with high data rates and low time overhead.
method Developed a reinforcement learning-based hierarchical beam training scheme with dynamic training levels.
result The proposed scheme achieves up to 75% performance gain in spectral efficiency compared to conventional methods.
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
Improves DNN robustness to adversarial attacks.
problem Adversarial attacks degrade DNN robustness.
method Spectral normalization of weight matrices.
result Spectral normalization enhances DNN generalization.
Paper derives CLT for Bayesian neural networks trained with variational inference.
problem Analyzing the fluctuation behavior of Bayesian neural networks trained with different variational inference schemes.
method Rigorous derivation of CLT for three variational inference schemes: idealized, Bayes-by-Backprop, and Minimal VI.
result Minimal VI scheme has larger variances but is more computationally efficient.
Paper tackles model collapse in recursive generative models using a weighted training scheme.
problem Model collapse in recursive generative models trained on synthetic data.
method Iteratively trains models on real and synthetic data, evaluates weighted training schemes.
result Optimal weighting scheme for synthetic data follows a unified expression, revealing a trade-off with model performance.
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
New method uses randomized sparse neural networks to solve time-dependent PDEs more accurately and efficiently.
problem Numerical challenges in training neural networks sequentially in time to solve time-dependent PDEs.
method Introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step.
result Up to two orders of magnitude more accurate and two orders of magnitude faster than dense update schemes.
Proposes SHADE, a new regularization scheme for deep learning.
problem Improving classification performance in deep learning.
method SHADE uses information theory to define a prior based on conditional entropy, decoupling representation learning from data fitting.
result Empirically validated improvements over standard regularization schemes.
Proposes SHADE, a new regularization scheme for deep learning.
problem Improving classification performances in deep learning.
method SHADE uses information theory to define a prior based on conditional entropy, decoupling representation learning from data fitting.
result Empirically validated improvements over common regularization schemes.
Paper proposes using auto-encoders to efficiently purify adversarial perturbations.
problem Vulnerability of machine learning models to adversarial examples.
method Iterative adversarial training on an auto-encoder to purify perturbations.
result The auto-encoder trained model outperforms other methods in protecting models against white-box attacks.
EigenNoise provides a competitive word vector initialization scheme without pre-training data.
problem Improving word vector initialization without pre-training data.
method EigenNoise uses a dense, independent co-occurrence model to initialize word vectors.
result EigenNoise can approach GloVe performance without pre-training data.
Efficient quantization scheme for neural networks using integer arithmetic.
problem Efficient inference on mobile devices with limited computational resources.
method Quantization and co-design training procedure for integer-only inference.
result Improves accuracy-latency tradeoff on various models and hardware.
This study benchmarks data augmentation schemes to improve CNN performance.
problem Lack of training data for deep learning models.
method Various geometric and photometric data augmentation schemes evaluated on a CNN.
result Cropping in geometric augmentation significantly improves CNN task performance.
New method learns population dynamics from snapshots using JKO scheme and inverse optimization.
problem Recovering underlying process governing particle evolution from discrete time samples.
method Combines JKO scheme with inverse optimization techniques for end-to-end adversarial training.
result Improved performance over prior JKO-based methods with theoretical guarantees.
Deep learning schemes improve turbulence simulation accuracy.
problem Improving DL schemes for accurate turbulence simulation.
method Training DL schemes on turbulence simulations to correct for inaccuracies.
result Dynamic NN schemes improve large-scale turbulence geometry.
A blind scheme combines multiple classifiers without knowing their training labels.
problem Combining multiple classifiers to achieve high performance.
method Moment matching method using tensor and matrix factorization.
result Proposed blind scheme outperforms known methods on synthetic and real datasets.
New method learns various data manipulation schemes for model training.
problem Improving model training with data manipulation.
method Adapts RL reward learning algorithm for data manipulation learning.
result Significant improvement in classification performance.
Proposes a new deep neural network training scheme combining different loss functions.
problem Improving generalization ability of deep neural networks.
method Integrates multiple loss functions in a nonlinear manner.
result The new objective function enhances optimization and generalization.
Self-training improves GANs for semi-supervised learning.
problem Training GANs with limited labeled data.
method Combining self-training with GANs' infinite data generation.
result Self-training improves GANs' performance in semi-supervised learning.
ContinuousNet generalizes ResNets to continuous dynamical systems.
problem ResNets fail to be meaningful dynamical integrators.
method Embedding continuous dynamical systems into higher-order numerical integration schemes (Runge Kutta).
result ContinuousNet exhibits invariance to discrete time step sizes and numerical integration schemes.
Traditionally, when generative models of data are developed via deep architectures, greedy layer-wise pre-training is employed. In a well-trained model, the lower layer of the architecture models the data distribution conditional upon the hidden variables, while the higher layers model the hidden distribution prior. Bu…
LEGW improves large-batch training for both CNNs and RNNs.
problem Efficient large-batch training for deep neural networks.
method Linear-epoch gradual-warmup (LEGW) for large-batch training.
result Improved large-batch training for both CNNs and RNNs with Sqrt Scaling scheme.
Improved performance of factorized neural layers through spectral initialization and Frobenius decay.
problem Improving the performance of factorized neural layers in various deep learning contexts.
method Spectral initialization and Frobenius decay for initialization and regularization.
result Spectral initialization and Frobenius decay lead to improved performance across multiple deep learning settings.
PILAE learns DNNs without gradient descent, achieving better performance.
problem Training deep feedforward neural networks efficiently and accurately.
method PILAE uses a pseudoinverse learning algorithm for autoencoder building blocks of MLP DNNs.
result PILAE achieves better performance on tradeoff between training efficiency and accuracy.
Paper develops a generative model using Wasserstein-2 loss.
problem Creating realistic data samples from limited data.
method Uses a distribution-dependent ODE with a gradient flow for W2 loss.
result The method converges to the true data distribution exponentially.
Band-limited training reduces resource usage without sacrificing accuracy.
problem Resource constraints in training Convolutional Neural Networks (CNNs).
method Artificially constraining the frequency spectra of convolutional filters during training.
result CNNs can leverage lower-frequency components effectively, reducing resource usage.
New loss function and training scheme improve binary hash codes for better similarity search.
problem Improving binary hash codes for better similarity search tasks.
method Log likelihood loss on Hamming distance target, novel training scheme, multi-indexing.
result Significant improvements in MAP (84%) and query cost reduction for ImageNet and SIFT 1M.
Proposes dynamic channel pruning during neural network training.
problem Pruning neural networks during training to reduce computational cost and improve efficiency.
method Dynamic channel propagation to update channel utility values and selectively prune channels.
result Our scheme trains and prunes neural networks simultaneously, achieving superior performance.
Extracting a curriculum from a teacher network improves distillation efficiency.
problem Efficiently training a small network using a large teacher network's output.
method Random projection of teacher network's hidden representations to progressively train the student network.
result Extracted curriculum significantly outperforms one-shot distillation and achieves similar performance to progressive distillation.
The paper provides bounds for regression schemes using nonstationary training samples.
problem Developing confidence intervals for nonparametric regression with nonstationary data.
method The approach involves Rademacher and Vapnik-Chervonenkis theories to analyze the cost and optimality of regression schemes.
result The paper establishes nonasymptotic bounds for regression schemes and optimality in L2-distance. Adaptive neural network improves MIMO detection on real-world channels.
problem Challenges in symbol detection for Massive MIMO.
method MMNet, a deep learning MIMO detection scheme that uses iterative soft-thresholding and temporal/spectral correlation.
result Significantly outperforms existing approaches on realistic channels with lower computational complexity.
New learning scheme outperforms ERM in individual data settings.
problem Learning from individual data samples.
method Information-theoretic approach using self-information loss.
result pNML scheme outperforms ERM in specific test challenges.
The paper analyzes Bayesian neural networks trained with VI, proving a law of large numbers for different schemes.
problem Training Bayesian neural networks with variational inference.
method Analyzes three training schemes: exact estimation, Bayes by Backprop, and Minimal VI.
result All training schemes converge to the same mean-field limit.
ALPS improves neural network robustness and generalization.
problem Challenges in designing effective regularization schemes for adversarial robustness.
method Adversarial Labelling of Perturbed Samples (ALPS) using synthetic samples and min-max formulation.
result ALPS achieves state-of-the-art regularization performance and adversarial robustness.
Selective joint fine-tuning improves deep learning with limited data.
problem Insufficient labeled training data for deep learning tasks.
method Joint fine-tuning of shared convolutional layers between source and target tasks using selected training images.
result Improves classification accuracy by 2% - 10% on multiple visual classification tasks.
The paper analyzes how modern machine learning models can achieve zero training error and robust generalization.
problem Understanding why modern machine learning models achieve strong generalization despite achieving zero training error.
method The paper analyzes local interpolating schemes including geometric simplicial interpolation and singularly weighted k-nearest neighbor methods.
result The nearest neighbor schemes exhibit optimal rates under standard statistical assumptions and provide insights into adversarial examples.
A deep clustering model learns to separate audio sources without supervision.
problem Training deep clustering models requires supervision, limiting their applicability.
method Proposes an unsupervised spatial clustering approach to train a deep clustering system.
result The deep clustering model achieves similar performance to a multi-channel teacher without supervision.
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
New sampling scheme improves ML accuracy in physics simulations.
problem Improving accuracy of ML models in physics simulations.
method Taylor-based data sampling scheme for DNNs.
result Reduces error in DNN solutions of ODE systems.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
A new numerical scheme approximates nonlinear filtering densities for noisy and partial measurements.
problem Approximating nonlinear filtering densities for noisy and partial measurements.
method Deep splitting scheme applied to the Fokker--Planck equation followed by Bayes' formula.
result Convergence rate established for the numerical scheme under parabolic Hörmander condition.
New method reduces communication in deep learning training.
problem Communication overhead in distributed deep learning training.
method Random-block sparsification to reduce gradients communicated.
result Performance close to standard SGD with reduced communication.
AUTOVC converts voices without parallel data, achieving state-of-the-art results.
problem Non-parallel many-to-many voice conversion and zero-shot voice conversion.
method Only an autoencoder with a carefully designed bottleneck is used, training on a self-reconstruction loss.
result AUTOVC achieves state-of-the-art results in many-to-many voice conversion with non-parallel data and performs zero-shot voice conversion.