Proposes model-based robust deep learning to handle natural variation in data.
problem Deep learning's fragility to natural variation in data.
method Develops model-based robust training algorithms using deep generative models to learn natural variation.
result Deep neural networks trained with model-based algorithms outperform standard and norm-bounded robust algorithms.
New method reduces adversarial training time by up to 2.5x with comparable accuracy.
problem High computational complexity of adversarial training for deep neural networks.
method Replacing initial adversarial training with natural training.
result Significant reduction in training time (up to 2.5x) without loss in accuracy.
Improved natural gradient boosting with leaf number clipping for faster and better performance.
problem Slower training speed and poor performance on large datasets for natural gradient boosting.
method Leaf number clipping regularization to optimize hyperparameters and improve performance.
result Significant improvement in performance and up to 4.85x speed up on various datasets.
New approach improves adversarial robustness without sacrificing natural generalization.
problem Balancing adversarial robustness and natural generalization in machine learning.
method Friendly adversarial training (FAT) using early-stopped PGD to find least adversarial data.
result Early-stopped PGD achieves adversarial robustness without compromising natural generalization.
AOPU stabilizes NN training by approximating natural gradient, improving stability and convergence.
problem Stability and interpretability in online NN training for industrial soft sensors.
method AOPU truncates gradient backpropagation, optimizing trackable parameters, and approximating natural gradient.
result AOPU achieves stable convergence and superior performance on chemical process datasets.
NRWS improves training of SBNs and HMs using natural gradient.
problem Training Sigmoid Belief Networks and Helmholtz Machines efficiently.
method Exploits block-diagonal structure of Fisher Information Matrices to use natural gradient.
result NRWS and NBiHM achieve better log-likelihood and faster convergence.
Language models learn from training data and can leak private information.
problem Language models lack context understanding and can expose private data.
method Discussing the limitations of current privacy protection methods for language models.
result Existing privacy protection methods are insufficient for language models.
Paper trains models to resist spurious patterns by having humans revise documents.
problem Machine learning models often rely on spurious patterns, leading to poor performance.
method Humans revise documents to retain coherence and avoid unnecessary changes, creating a combined dataset.
result Models trained on combined datasets perform better than those trained on original or manipulated data alone.
The paper proposes a method to train time-varying generative models using natural gradients.
problem Training time-varying generative models efficiently and accurately.
method Projecting generative model parameters onto an exponential family manifold and optimizing using natural gradient descent.
result The proposed method efficiently approximates the natural gradient and can be applied to various exponential family models.
Natural gradient descent is a robust optimization method for machine learning.
problem Training poorly parameterized networks efficiently.
method Optimization algorithms with natural transformation properties.
result Optimization algorithms with natural transformation properties are more efficient for poorly parameterized networks.
We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outp…
We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions. Two desirable characteristics of an attribution-based explanation are: (1) $\texti…
SONet stabilizes ODE networks for robustness without adversarial training.
problem Improving adversarial robustness of neural networks without sacrificing natural accuracy.
method SONet uses skew-symmetric ODE blocks and DOPRI5 solver for robustness.
result SONet achieves comparable robustness to adversarial defense methods without trade-off.
Recent work on adversarial attack and defense suggests that PGD is a universal l∞ first-order attack, and PGD adversarial training can significantly improve network robustness against a wide range of first-order l∞-bounded attacks, represented as the state-of-the-art defense method. However, an obvious …
RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weigh…
We consider the problem of making machine translation more robust to character-level variation at the source side, such as typos. Existing methods achieve greater coverage by applying subword models such as byte-pair encoding (BPE) and character-level encoders, but these methods are highly sensitive to spelling mistake…
Proposes an alternative method to train RBMs with binary synapses using Bayesian learning rule.
problem Training RBMs with binary synapses is challenging due to discrete nature of synapses.
method Proposes an alternative optimization method using the Bayesian learning rule, updating natural parameters instead of expectation parameters.
result No additional clipping is needed as natural parameters take values in the entire real domain.
Untrained neural networks can recover natural images from few measurements.
problem Recovering natural images from a small number of measurements.
method Gradient descent on un-trained convolutional neural networks.
result Untrained neural networks can approximate reconstruct signals/images from a near minimal number of random measurements.
This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural g…
Stochastic defense improves natural classifiers against adversarial attacks.
problem Vulnerability of deep networks to adversarial attacks.
method Long-run MCMC sampling with Energy-Based Model for adversarial purification.
result Balancing memoryless and metastable behavior leads to effective purification and robust classification.
Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This pa…
Today's state-of-the-art image classifiers fail to correctly classify carefully manipulated adversarial images. In this work, we develop a new, localized adversarial attack that generates adversarial examples by imperceptibly altering the backgrounds of normal images. We first use this attack to highlight the unnecessa…
Paper improves natural language understanding with less data using a new training method.
problem Limited data hinders performance of small models in natural language tasks.
method Generation-Distillation: uses large finetuned models to generate new training data and distill knowledge into smaller models.
result Achieves comparable performance to BERT with 300x fewer parameters and outperforms prior distillation methods.
Automated learning rate tuning for neural networks across datasets and models.
problem Finding optimal learning rates for neural network training, especially in adversarial settings.
method A parameterless, adaptive learning rate trajectory algorithm.
result Consistently achieves top-level accuracy in natural and adversarial training.
Dropout training is shown to be optimal for adversarial covariate corruption.
problem Adversarial corruption of covariates in machine learning models.
method Minimax game analysis with dropout noise as the adversarial strategy.
result Dropout training minimizes expected loss for multiplicative perturbations of covariates.
Improves natural accuracy of deep learning models by combining robust predictions and features.
problem Maintaining natural accuracy while resisting adversarial attacks.
method Ensemble methods combining robust and standard models.
result Optimized natural accuracy through ensemble of robust models.
In this work, we introduce a two-step framework for generative modeling of temporal data. Specifically, the generative adversarial networks (GANs) setting is employed to generate synthetic scenes of moving objects. To do so, we propose a two-step training scheme within which: a generator of static frames is trained fir…
Study improves breast lesion segmentation with limited in vivo data using simulated and natural images.
problem Challenges in automatic breast lesion segmentation due to limited annotated data.
method Pre-training a segmentation network on simulated and natural images, followed by fine-tuning with limited in vivo data.
result Fine-tuning improves dice score by 21% with as little as 19 in vivo images.
Modified training direction reduces generalization error in neural networks.
problem Reducing generalization error in neural networks.
method Theoretical analysis of modified natural gradient descent in function space.
result Modifying training direction in function space reduces total generalization error.
New approach shows backdoor attacks are indistinguishable from natural data features.
problem Defending against backdoor attacks in machine learning models.
method Developed a new primitive for detecting backdoor attacks based on the assumption that they correspond to the strongest feature in the training data.
result Backdoor attacks are indistinguishable from natural data features, making traditional detection methods ineffective.
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also …
We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multi-core machines. In order to be as hardware-agnostic as possible, we needed a way to use multiple machines without …
Associating image regions with text queries has been recently explored as a new way to bridge visual and linguistic representations. A few pioneering approaches have been proposed based on recurrent neural language models trained generatively (e.g., generating captions), but achieving somewhat limited localization accu…
Empirical adversarial risk minimization (EARM) is a widely used mathematical framework to robustly train deep neural nets (DNNs) that are resistant to adversarial attacks. However, both natural and robust accuracies, in classifying clean and adversarial images, respectively, of the trained robust models are far from sa…
Our work proves convergence to low robust training loss for polynomial width ReLU networks.
problem Understanding why adversarial training leads to low robust training loss in over-parameterized neural nets.
method Extending convergence theory for standard supervised training to adversarial training, using tools from online learning and showing ReLU networks can approximate the step function.
result Convergence to low robust training loss for polynomial width ReLU networks under natural assumptions.
The paper explores how neural networks generalize differently from natural and medical images.
problem Discrepancies in generalization error between natural and medical images.
method Established and empirically validated a generalization scaling law with respect to intrinsic dataset properties.
result Higher intrinsic 'label sharpness' of medical images leads to higher adversarial vulnerability.
New method reduces bias in NLI models using ensemble adversarial training.
problem Spurious correlations between hypotheses and entailment classes in NLI datasets.
method Adversarial training with an ensemble of classifiers to reduce bias in sentence representations.
result Ensemble adversarial training produces more robust NLI models, outperforming previous methods.
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
problem Worse generalization performance with large mini-batch sizes in deep learning.
method SP-NGD, a natural gradient descent approach for large-scale deep learning.
result SP-NGD achieves similar generalization performance to first-order methods with accelerated convergence and negligible overhead.
The study enhances financial rule matching using NLP without datasets.
problem Performing semantic matching between financial rules and policies.
method Outperforming pre-trained models with NLP techniques using free resources.
result Improved semantic matching between financial rules and policies.
Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition t…
Analysis of model updates reveals sensitive data leaks.
problem Information leakage during model updates.
method Differential analysis of language model snapshots.
result New metrics (differential score, differential rank) reveal sensitive data leaks.
We present a theoretical analysis of Gaussian-binary restricted Boltzmann machines (GRBMs) from the perspective of density models. The key aspect of this analysis is to show that GRBMs can be formulated as a constrained mixture of Gaussians, which gives a much better insight into the model's capabilities and limitation…
New analysis shows diverse classes in pre-training boost NLP performance.
problem Improving sample efficiency in downstream NLP tasks.
method Proved that diverse classes in pre-training lead to better performance, using a large last linear layer singular value.
result Transfer learning excess risk improves with large ildeν and $O\left(\frac{1}{ ildeν \sqrt{n}}
ight)$ rate. LOGAN optimizes GAN training by improving adversarial dynamics.
problem Training GANs is challenging due to delicate adversarial dynamics and potential divergence.
method Integrates natural gradient-based latent optimisation into CS-GAN.
result Significant improvement in GAN training performance, achieving state-of-the-art results.
Language models can answer questions without external knowledge.
problem How much knowledge can be stored in language models?
method Fine-tuning pre-trained language models to answer questions without external context.
result Fine-tuned models perform competitively with open-domain systems.
Gaudy images help train deep neural networks with less data.
problem Training deep neural networks with limited real data from visual cortex neurons.
method Used high-contrast binarized natural images (gaudy images) to train DNNs.
result Reduced training data needed for accurate DNN predictions of visual cortex neuron responses.
Improving the accuracy and robustness of deep neural nets (DNNs) and adapting them to small training data are primary tasks in deep learning research. In this paper, we replace the output activation function of DNNs, typically the data-agnostic softmax function, with a graph Laplacian-based high dimensional interpolati…