arXiv research

Paper tackles catastrophic overfitting in single-step adversarial training.

IMM generates high-quality samples in few steps with stable training.

A new method reduces adversarial training time without overfitting.

New method defends against both single-step and iterative adversarial examples.

Proposes QDF to improve multi-step time-series forecasting.

Unified model improves sampling speed and quality.

Efficient regularization mitigates catastrophic overfitting in single-step adversarial training.

NCT simplifies one-step generator adaptation to new controls.

Proposes a neural network for learning step-size policies for L-BFGS optimization.

A new scaling law predicts optimal batch size for training models.

This paper establishes a theoretical foundation for consistency training in diffusion models.

The performance of deep (reinforcement) learning systems crucially depends on the choice of hyperparameters. Their tuning is notoriously expensive, typically requiring an iterative training process to run for numerous steps to convergence. Traditional tuning algorithms only consider the final performance of hyperparame…

SMT trains generative models by estimating mixture scores, outperforming existing methods.

In recent years, deep neural networks have demonstrated outstanding performance in many machine learning tasks. However, researchers have discovered that these state-of-the-art models are vulnerable to adversarial examples: legitimate examples added by small perturbations which are unnoticeable to human eyes. Adversari…

Paper presents faster, robust adversarial training methods.

Adversarial examples are malicious inputs designed to fool machine learning models. They often transfer from one model to another, allowing attackers to mount black box attacks without knowledge of the target model's parameters. Adversarial training is the process of explicitly training a model on adversarial examples,…

We propose Rademacher complexity bounds for multiclass classifiers trained with a two-step semi-supervised model. In the first step, the algorithm partitions the partially labeled data and then identifies dense clusters containing $κ$ predominant classes using the labeled training examples such that the proportion of t…

The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-step-ahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of th…

New method improves adversarial training efficiency and robustness.

Step decay schedules improve convergence in non-convex optimization.

Certified training improves robustness against adversarial attacks.

In this work, we introduce a two-step framework for generative modeling of temporal data. Specifically, the generative adversarial networks (GANs) setting is employed to generate synthetic scenes of moving objects. To do so, we propose a two-step training scheme within which: a generator of static frames is trained fir…

Downscaled models outperform larger ones on GLUE tasks.

W-Flow generates images in one step, faster and better than multi-step methods.

A hybrid training method reduces SNN training time and complexity.

One of the main challenges of deep learning methods is the choice of an appropriate training strategy. In particular, additional steps, such as unsupervised pre-training, have been shown to greatly improve the performances of deep structures. In this article, we propose an extra training step, called post-training, whi…

A higher-order Runge-Kutta optimizer performs poorly compared to Adam when evaluated fairly.

Neural networks can overfit perfectly to noisy data and then grok near-optimal generalization.

SPGD improves adversarial training efficiency and accuracy.

GradSkip reduces local training steps for better communication efficiency.

Fastens diffusion model sampling by distillation.

A new training method improves stability and generalization of DeepONets.

Injecting adversarial examples during training, known as adversarial training, can improve robustness against one-step attacks, but not for unknown iterative attacks. To address this challenge, we first show iteratively generated adversarial images easily transfer between networks trained with the same strategy. Inspir…

In this paper, we propose a two-step training procedure for source separation via a deep neural network. In the first step we learn a transform (and it's inverse) to a latent space where masking-based separation performance using oracles is optimal. For the second step, we train a separation module that operates on the…

Large SGD step sizes lead to sparse feature learning in neural networks.

Transformers learn multi-step reasoning through gradient descent.

New techniques improve distributed training with compressed gradients.

Training a neural network with the gradient descent algorithm gives rise to a discrete-time nonlinear dynamical system. Consequently, behaviors that are typically observed in these systems emerge during training, such as convergence to an orbit but not to a fixed point or dependence of convergence on the initialization…

New insights into how large learning rates affect transformer training dynamics.

Single-step samplers generate high-quality samples efficiently.

Deep unfolding is a promising deep-learning technique in which an iterative algorithm is unrolled to a deep network architecture with trainable parameters. In the case of gradient descent algorithms, as a result of the training process, one often observes the acceleration of the convergence speed with learned non-const…

Proposes a new adaptive learning rate for SGD.

Neural ODEs' performance varies with numerical method, requiring adaptive step size control.

Local Gradient Descent with local steps converges to the centralized model in the interpolation regime.

This paper presents a simulator-assisted training method (SimVAE) for variational autoencoders (VAE) that leads to a disentangled and interpretable latent space. Training SimVAE is a two-step process in which first a deep generator network(decoder) is trained to approximate the simulator. During this step, the simulato…

Adversarial examples are perturbed inputs designed to fool machine learning models. Adversarial training injects such examples into training data to increase robustness. To scale this technique to large datasets, perturbations are crafted using fast single-step methods that maximize a linear approximation of the model'…

New method reduces PDE surrogate model training costs by selectively acquiring time steps.

Proposes a method to balance tasks in multitask learning with a single gradient step update.