Study shows how bias in optimization affects robustness in adversarial settings.
problem Understanding and mitigating implicit bias in adversarially robust models.
method Analyzes the implicit bias in robust empirical risk minimization and its impact on generalization.
result Implicit bias in optimization can significantly affect robust generalization.
Survey on why deep learning works despite having more parameters than data.
problem Understanding why deep learning algorithms generalize well despite having more parameters than training data.
method Explains the concept of implicit bias and reviews recent research findings.
result Implicit bias is a key factor in deep learning's ability to generalize.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
This paper analyzes implicit bias in Deep Linear Discriminant Analysis.
problem The implicit bias of Deep Linear Discriminant Analysis.
method Analyzing gradient flow on a L-layer diagonal linear network.
result Under balanced initialization, the network transforms additive updates into multiplicative updates, conserving the (2/L) quasi-norm.
Geometric framework explains and controls implicit bias in machine learning.
problem Understanding and controlling the selection of solutions in overparameterized models.
method Developed a theoretical and constructive framework based on geometric corrections induced by gradient noise and continuous symmetries of the loss.
result Computed the induced bias across various architectures and enabled inverse design to shape the bias.
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
This work extends implicit bias analysis to multiclass classification using a new loss framework.
problem The implicit bias of gradient descent on multiclass data without explicit regularization.
method Employing the PERM framework to introduce a multiclass extension of the exponential tail property.
result Extended implicit bias result to multiclass classification using a new loss framework.
Our paper characterizes how ReLU affects GD's implicit bias in high-dimensional neural networks.
problem Understanding the implicit bias of gradient descent on neural networks.
method Novel primal-dual analysis tracking predictions and coefficients.
result The implicit bias approximates the minimum-ℓ2-norm solution with high probability. Intersectional constraints improve selection outcomes by reducing inequality.
problem Persistent inequality and reduced utility in selection processes due to implicit bias.
method Introducing intersectional constraints to mitigate the adverse effects of implicit bias in selection processes.
result Intersectional constraints can recover almost all the utility achievable in the absence of implicit bias, offering a significant advantage over non-intersectional constraints.
Study shows how steepest descent algorithms' geometric margin increases during training.
problem Understanding implicit bias in steepest descent algorithms for neural networks.
method Analysis of steepest descent algorithms with infinitesimal learning rates in homogeneous neural networks.
result Limit points of training trajectories correspond to KKT points of margin-maximization problems.
Gradient flow in ReLU networks biases towards generalization but makes them vulnerable to adversarial attacks.
problem Generalization vs. Adversarial Robustness in ReLU Networks
method Analysis of gradient flow in two-layer ReLU networks with clustered data.
result Gradient flow biases towards generalization but also makes networks vulnerable to adversarial attacks.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
Over the past two decades, the notion of implicit bias has come to serve as an important component in our understanding of discrimination in activities such as hiring, promotion, and school admissions. Research on implicit bias posits that when people evaluate others -- for example, in a hiring context -- their unconsc…
This work investigates implicit bias in multiclass separable data using a novel geometry-aware optimizer.
problem Understanding implicit bias in overparameterized models on multiclass separable data.
method Introduces NucGD, a geometry-aware optimizer enforcing low-rank structures through nuclear norm constraints.
result NucGD enables scalable training and characterizes the impact of stochastic optimization dynamics.
Study reveals how initialization scale affects training accuracy in linear networks.
problem Understanding implicit bias in linear classification models.
method Asymptotic analysis of gradient flow trajectories and training loss minimization.
result Implicit bias is more complex at reasonable initialization scales and training accuracies.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
New model reduces matrix factorization bias, yielding truly low-rank solutions.
problem Gradient descent's implicit bias in matrix factorization.
method Introducing a new factorization model with constrained factors and diagonal components.
result The new model consistently exhibits a strong implicit bias, yielding truly low-rank solutions.
SSMs can be poisoned with clean labels, leading to generalization failure.
problem The implicit bias of SSMs can be manipulated by including special training examples with clean labels.
method Formal proof and empirical demonstration of the phenomenon.
result SSMs can fail to generalize even with clean labels, due to the inclusion of special training examples.
Study reveals how neural network biases align with adversarial attack frequencies.
problem Correlation between neural network biases and adversarial attacks.
method Fourier transform analysis of network implicit bias and adversarial perturbations.
result Network bias and adversarial attack frequencies are highly correlated.
Mirror flow in shallow neural networks shows similar implicit bias to gradient flow, with key differences in curvature penalties.
problem Analyzing implicit bias in shallow neural networks with mirror flow.
method Characterization through variational problems and scaled potentials.
result Mirror flow with scaled potentials induces a rich class of biases not captured by RKHS norms.
This study investigates how gradient-based methods bias neural networks trained on high-dimensional data.
problem The implicit biases of gradient-based optimization algorithms in neural networks trained on high-dimensional data.
method Investigation of gradient flow and gradient descent in two-layer fully-connected neural networks with leaky ReLU activations.
result Gradient flow and gradient descent lead to neural networks with low-rank solutions and linear decision boundaries.
Adam's bias shifts from full-batch to max-margin of different norms for separable data.
problem Understanding Adam's implicit bias in the incremental batch setting.
method Analyzing incremental Adam on linearly separable data, constructing datasets, and using a proxy algorithm.
result Incremental Adam can converge to different max-margin classifiers depending on the dataset and batching scheme.
SIP corrects model bias in Bayesian ML.
problem Model selection biases predictions in Bayesian ML.
method Sparse Implicit Processes (SIP) for flexible, trainable predictions.
result SIP provides better predictive distributions than initial models.
Batch normalization biases linear models towards uniform margins, improving performance in binary classification.
problem Understanding the implicit bias of batch normalization in linear models and neural networks.
method Analyzing gradient descent convergence on linear models and two-layer CNNs with batch normalization.
result Gradient descent with batch normalization in linear models converges to a uniform margin classifier with an exponential convergence rate.
GNIs induce asymmetric heavy-tailed noise in SGD, affecting network performance.
problem The effect of Gaussian noise injections on SGD dynamics and network performance.
method Developed a Langevin-like SDE driven by asymmetric heavy-tailed noise to model the modified SGD dynamics.
result GNIs induce an implicit bias that varies with noise heaviness and asymmetry, affecting network performance.
Adam optimizes linear classifiers with separable data.
problem Understanding Adam's implicit bias in linear logistic regression.
method Study of Adam's behavior on linearly separable data.
result Adam converges to a linear classifier with maximum ℓ∞-margin. Algorithmic approaches endow deep learning systems with implicit bias that helps them generalize even in over-parametrized settings. In this paper, we focus on understanding such a bias induced in learning through dropout, a popular technique to avoid overfitting in deep learning. For single hidden-layer linear neural …
The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.
problem Convergence analysis for linear systems with unbounded domain.
method Entropic mirror descent with Polyak stepsizes, sublinear and linear convergence results.
result Generalized convergence result for arbitrary convex functions.
Unified framework approximates gradient descent's implicit bias in high dimensions.
problem Understanding gradient descent's behavior in overparameterized settings with convex losses.
method Unified framework for convex losses, including sensitivity analysis.
result Approximation of minimum-norm interpolation in high dimensions.
GD iterates for non-homogeneous deep nets increase margin and converge in direction.
problem Understanding implicit bias in non-homogeneous deep networks.
method Characterization of GD iterates' properties starting from small empirical risk.
result GD iterates converge in direction despite diverging norms, satisfying KKT conditions.
SGD noise has no bias advantage in online learning, contrary to offline learning.
problem The role of SGD noise in online learning.
method Extensive empirical analysis of image and language data.
result Small batch sizes do not confer any implicit bias advantages in online learning.
Linear RNNs exhibit a bias towards shorter memory due to initialization variance.
problem Understanding the performance limitations of RNNs, especially linear ones.
method Kernel regime analysis to show equivalence to 1D-convolutional networks and analyze weightings.
result Linear RNNs with random initialization have a bias towards shorter memory periods.
Local convolutions bias neural networks towards high-frequency adversarial examples.
problem High-frequency adversarial examples in neural networks.
method Analysis of different linear and nonlinear architectures, focusing on the impact of local convolution operations.
result Local convolutions induce an implicit bias towards high frequency features, leading to high-frequency adversarial examples.
Self-attention optimizers converge to optimal weights, revealing bias patterns.
problem Understanding the implicit bias in self-attention mechanisms.
method Analysis of gradient-based optimization in self-attention layers.
result Adaptive step-size strategies can accelerate convergence to optimal weights.
New insights into network generalization show learning rate affects both norm and sharpness.
problem Understanding the generalization of overparameterized networks.
method Empirical analysis and theoretical proof of the trade-off between norm and sharpness.
result Learning rate influences both norm and sharpness, neither alone minimizes generalization error.
This study reveals efficient finite-difference computation for gradient regularization in deep learning.
problem Improving generalization performance in deep learning through gradient regularization.
method Analyzes and reveals a specific finite-difference computation that reduces computational cost and improves generalization performance.
result Finite-difference computation strengthens the implicit bias towards rich regimes and enhances generalization performance.
This work studies the implicit bias of mini-batch SGD in classification.
problem Understanding the implicit bias of mini-batch SGD in multi-class classification.
method Characterizes how batch size, momentum, and variance reduction affect convergence and max-margin behavior under different norms.
result Momentum enables small-batch convergence to an approximate max-margin solution, while variance reduction recovers the exact full-batch bias.
Study shows how networks converge to minimum norm solutions with regularization.
problem Interpolating between known regions in shallow ReLU networks.
method Investigates empirical risk minimizers and weight decay regularizers.
result Empirical risk minimizers converge to minimum norm interpolants under specific conditions.
Gradient matching method estimates implicit regularization in complex deep learning systems.
problem Estimating implicit regularization in modern deep learning systems with complex modifications.
method Gradient matching methods to empirically estimate implicit regularization.
result Empirical estimation of implicit regularization in arbitrary networks, including dropout.
Adam avoids simplicity bias in neural networks, leading to better generalization.
problem Simplicity bias in neural networks trained with SGD.
method Comparison of Adam and GD on binary classification tasks with Gaussian data.
result Adam leads to richer and more diverse features, improving generalization.
Combining explicit and implicit regularization improves deep learning performance without needing depth.
problem Improving deep learning performance without increasing model complexity.
method Proposes an explicit penalty to mirror implicit regularization bias in adaptive gradient optimizers.
result Single-layer networks can achieve low-rank approximations with similar performance to deep linear networks.
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems. We explore the question of whether the specifi…
This paper studies how gradient descent in control systems can perform well on unseen data.
problem The extent of a learned controller's ability to extrapolate to unseen initial states.
method Theoretical study of policy gradient in Linear Quadratic Regulator (LQR) problems, focusing on the role of exploration.
result The performance of a learned controller on unseen initial states depends on the degree of exploration induced by the system.
Matrix SMD converges to unique solution minimizing Bregman divergence.
problem High-dimensional multi-output classification and matrix completion problems.
method Stochastic Mirror Descent with matrix parameters and matrix mirror functions.
result Matrix SMD converges exponentially to the unique solution minimizing Bregman divergence.
The paper analyzes the implicit bias of SGD near loss manifold and provides new insights.
problem Understanding the implicit bias of SGD near loss manifolds in overparametrized models.
method Adapting ideas from Katzenberger (1991) to analyze SGD dynamics using a stochastic differential equation (SDE).
result SGD with label noise locally decreases the sharpness of loss, leading to a global analysis of implicit bias.
A new model shows fairness mechanisms can improve selection utility even without implicit bias.
problem Improving selection fairness without introducing a utility trade-off.
method A model with latent quality and group-dependent variance, comparing fairness mechanisms to group-oblivious selection.
result Demographic parity always increases selection utility, while γ-rules weakly increase it. Gradient flow on ReLU networks converges to a simple model with few regions.
problem Understanding the dynamics of gradient flow in shallow ReLU networks.
method Analysis of gradient flow dynamics on univariate ReLU neural networks.
result Gradient flow converges to a network with at most O(r) linear regions.