arXiv research

Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.

In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient…

Improved sampling method using regularized Stein Variational Gradient Flow.

The paper tackles safe reinforcement learning with convex regularization.

Study shows policy gradient convergence for entropy-regularized MDPs with neural nets in mean-field regime.

MGD with early stopping tends to ridge regularization in least squares regression.

Cross-regularization adapts model complexity during training.

Noise injection before gradient steps helps in regularization for neural networks.

Regularization plays an important role in generalization of deep neural networks, which are often prone to overfitting with their numerous parameters. L1 and L2 regularizers are common regularization tools in machine learning with their simplicity and effectiveness. However, we observe that imposing strong L1 or L2 reg…

Regularizes persistent homology gradients for neural network integration.

Gradient matching method estimates implicit regularization in complex deep learning systems.

We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds…

Proposes a new method to improve deep model security against adversarial deformations.

We study a policy gradient method with L2 regularization for MAB problems.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

Gradient descent implicitly favors group sparsity in neural networks.

Adam's hyperparameters implicitly regularize solutions, penalizing or impeding loss gradients' norms.

New algorithm adds Hessian regularization to improve neural network robustness.

Efficiently performs robust and sparse kernel regression.

Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.

Study shows convergence of stochastic gradient method for unregularized Wasserstein optimization.

New metric shows how different regularization methods affect deep linear networks.

PEGR improves deep learning models' robustness against noisy data.

The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.

SGD with large learning rates can achieve better test accuracy than expected.

Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order…

DRAG decreases regularization to accelerate semi-discrete OT convergence.

New method improves adversarial robustness of neural networks.

This work shows how exploiting gradient alignment can improve distributed and federated learning performance.

A regularized optimization problem over a large unstructured graph is studied, where the regularization term is tied to the graph geometry. Typical regularization examples include the total variation and the Laplacian regularizations over the graph. When applying the proximal gradient algorithm to solve this problem, t…

Entropy-regularized NPG converges linearly with linear function approximation.

In this work we study input gradient regularization of deep neural networks, and demonstrate that such regularization leads to generalization proofs and improved adversarial robustness. The proof of generalization does not overcome the curse of dimensionality, but it is independent of the number of layers in the networ…

Gradient descent in tensor factorization favors low-rank solutions.

Introduces self-regularization for analyzing learning algorithms.

A fast method for discrete OT with group-sparse regularization for class label preservation.

Study compares dropout and l2 regularization in linear models.

Combining explicit and implicit regularization improves deep learning performance without needing depth.

SGD improves generalization by using gradient variability as a proxy for data randomness.

We propose a principled method for gradient-based regularization of the critic of GAN-like models trained by adversarially optimizing the kernel of a Maximum Mean Discrepancy (MMD). We show that controlling the gradient of the critic is vital to having a sensible loss function, and devise a method to enforce exact, ana…

Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit regularization of gradient descent over deep linear neural networks for matrix completion …

Study compares adversarial regularization to sole supervision in machine learning.

The paper studies steady solitons with curvature decay and proves their smoothness.

Stochastic gradient descent regularizes least squares problems by smoothing large singular values.

Develops minibatch stochastic proximal gradient for large-scale learning models.

The paper analyzes rates for a modified gradient descent method using Stein variational gradients.

The paper proposes a gradient-based method for multi-penalty Ridge regression.

We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …

This article is an overview of supervised machine learning problems for regression and classification. Topics include: kernel methods, training by stochastic gradient descent, deep learning architecture, losses for classification, statistical learning theory, and dimension independent generalization bounds. Implicit re…