Every local minimum in non-convex machine learning is globally optimal.
problem Non-convex optimization challenges in machine learning.
method Proves every local minimum achieves globally optimal value of perturbable gradient basis model.
result Theoretical support for non-convex machine learning similar to convex machine learning.
Node-perturbation learning is a type of statistical gradient descent algorithm that can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. It estimates the gradient of an objective function by using the change in the object function in response to the per…
Unified framework for gradient estimation in combinatorial spaces.
problem Scaling relaxed gradient estimators to large combinatorial distributions.
method Introducing stochastic softmax tricks within the perturbation model framework.
result Stochastic softmax tricks improve model performance and discover more latent structure.
We introduce and analyze stochastic optimization methods where the input to each gradient update is perturbed by bounded noise. We show that this framework forms the basis of a unified approach to analyze asynchronous implementations of stochastic optimization algorithms.In this framework, asynchronous stochastic optim…
Approximate vanishing ideal is a concept from computer algebra that studies the algebraic varieties behind perturbed data points. To capture the nonlinear structure of perturbed points, the introduction of approximation to exact vanishing ideals plays a critical role. However, such an approximation also gives rise to a…
Gradient noise improves privacy-protected optimization performance.
problem Improving privacy in convex optimization while maintaining utility.
method We analyze the effect of gradient perturbation on differentially private convex optimization, focusing on expected curvature.
result Gradient perturbation can achieve a significantly improved utility guarantee for differentially private convex optimization.
New perturbative method improves stochastic gradient descent for binary weights.
problem Improving stochastic gradient descent for binary weights.
method Perturbative expansion around the mean of the sampling distribution, Taylor-corrected estimators, variance reduction techniques.
result Perturbative correction improves convergence of stochastic variational inference.
Novel PCA method for high-dimensional inverse problems.
problem Optimizing large-scale random fields with gradient information.
method Gradient-Sensitive Principal Component Analysis (Gradient-SPCA) that modifies PCA using objective function gradients.
result Improvements in encoding quality for objective function minimization and field distribution.
Sharp results link DLN gradient flow to basis pursuit optimization and GHA phase transitions.
problem Understanding implicit regularization in Diagonal Linear Networks.
method Sharp convergence bounds and characterization of ℓ1 minimizers. result Gradient flow of DLNs with tiny initialization approximates minimizers of basis pursuit optimization problem.
Paper tackles fooling deep networks with minimal perturbations.
problem Easily fooling deep neural networks with high confidence predictions.
method Uses integrated adaptive gradients to generate minimal adversarial perturbations.
result Achieves minimal adversarial perturbations for fooling deep networks.
SGD converges with perturbed forward-backward passes, explained by geometric amplification.
problem Analyzing convergence of SGD with perturbed forward-backward passes in composite optimization.
method Characterized propagation and amplification of perturbations, derived convergence guarantees for non-convex and PL objectives.
result Perturbations cascade through the computational graph, affecting convergence order under specific conditions.
The paper examines how gradient descent stabilizes low-rank matrix factorization in noisy conditions.
problem Stability of low-rank implicit regularization in perturbed deep matrix factorization.
method Derives spectral conditions for gradient descent to exhibit a low-rank phase in noiseless settings and analyzes perturbed dynamics.
result Gradient descent converges to a low-rank solution under perturbation, with explicit dependence on perturbation size.
Paper shows robustness of gradient descent in matrix sensing despite perturbations.
problem Understanding robustness of gradient descent in matrix sensing.
method Developed perturbed gradient flow to capture noise and improve robustness.
result Gradient descent is robust to perturbations in matrix sensing.
The paper shows how to stabilize perturbed Kähler-Ricci solitons.
problem Stabilizing perturbed Kähler-Ricci solitons.
method Normalized Kähler-Ricci flow starting from perturbed metrics.
result The flow converges to an asymptotically conical gradient expanding Kähler-Ricci soliton.
Optimizes basis functions for learning dynamical systems from data.
problem Learning suitable basis functions for dynamical systems from data.
method Gradient-based optimization framework for learning basis functions.
result Efficacy demonstrated on various benchmark problems.
New methods reveal symmetries in Chern-Simons theory.
problem Understanding symmetries in Chern-Simons theory.
method Introduced a special basis in the center of the universal enveloping algebra to present group factors in arbitrary representations.
result Computed Vassiliev invariants and proved the tug-the-hook symmetry of the colored HOMFLY polynomial.
New geometric interpretation explains over-parameterized models and adversarial perturbations.
problem Geometric understanding of over-parameterized regression and adversarial perturbations.
method Alternative geometric interpretation of regression in feature space.
result Adversarial perturbations are a natural feature of biased models due to underlying geometry.
Gradient-based training and pruning for radial basis function networks in materials physics.
problem Interpretable and robust machine learning for materials physics problems.
method Gradient-based training and pruning of radial basis function networks with closed-form optimization criteria.
result Pruned models provide compact and interpretable versions of larger models, offering insights into atom-level migration processes.
Gradient-enhanced GSA uses Poincaré chaos expansions for accurate sensitivity analysis.
problem Accurately estimating Sobol' indices with limited data.
method Integrates sparse, gradient-enhanced regression with Poincaré chaos expansions for derivative-based sensitivity analysis.
result Accurately estimated Sobol' indices using limited data.
Gradient boosts monomial-order-free basis construction algorithms.
problem Lack of theoretical properties in monomial-order-free basis construction algorithms.
method Exploits gradient to sidestep spurious vanishing, achieve consistent output, and remove redundant bases.
result Proposes methods that equip monomial-order-free algorithms with theoretical properties.
Most random ReLU networks are vulnerable to small, Euclidean adversarial perturbations.
problem Vulnerability of ReLU networks to adversarial attacks.
method Analysis of random ReLU networks with decreasing dimensions, using gradient flow and descent.
result Most examples can be perturbed by small Euclidean distances via gradient methods.
We present a version of the equivariant gradient degree defined for equivariant gradient perturbations of an equivariant unbounded self-adjoint operator with purely discrete spectrum in Hilbert space. Two possible applications are discussed.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
New method creates universal perturbations to fool neural network interpretations.
problem Vulnerability of gradient-based saliency maps to adversarial perturbations.
method Gradient-based optimization and PCA-based approach to create UPI.
result Existence and successful application of Universal Perturbation for Interpretation (UPI).
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. ELD compares graphs by their embedded Laplacian eigenvectors, resolving ambiguities.
problem Comparing graphs of different sizes and structures.
method ELD uses symmetrization and perturbation techniques to compare graph embeddings.
result ELD resolves ambiguities in graph comparisons, making it a natural pseudo-metric.
Adversarial perturbations fool deepfake detectors with high accuracy.
problem Improving deepfake detection accuracy against adversarial attacks.
method Used adversarial perturbations and two defenses: Lipschitz regularization and Deep Image Prior (DIP).
result Deepfake detectors achieved 27% accuracy on perturbed images, compared to 95% on unperturbed.
We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that…
This paper aims at achieving a "good" estimator for the gradient of a function on a high-dimensional space. Often such functions are not sensitive in all coordinates and the gradient of the function is almost sparse. We propose a method for gradient estimation that combines ideas from Spall's Simultaneous Perturbation …
Radial-basis-function networks are traditionally defined for sets of vector-based observations. In this short paper, we reformulate such networks so that they can be applied to adjacency-matrix representations of weighted, directed graphs that represent the relationships between object pairs. We re-state the sum-of-squ…
Gradient-based methods introduce noise that penalizes models sensitive to weight perturbations.
problem Noise in gradient-based optimization methods.
method Analysis of Gradient Descent (GD) and Stochastic Gradient Descent (SGD) updating all parameters simultaneously.
result Noise introduced by simultaneous parameter updates penalizes models sensitive to weight perturbations.
LoRA fine-tuning explained with gradient dynamics for low-rank perturbations.
problem Understanding why gradient descent converges to useful low-rank perturbations in LoRA fine-tuning.
method Generalized student-teacher setting with i.i.d. samples and online gradient descent.
result Gradient descent converges to the teacher model in dkO(1) iterations under certain conditions. We study the effectiveness of various approaches that defend against adversarial attacks on deep networks via manipulations based on basis function representations of images. Specifically, we experiment with low-pass filtering, PCA, JPEG compression, low resolution wavelet approximation, and soft-thresholding. We evalu…
Learnable token perturbations boost extrapolation in LLMs.
problem Limited flexibility of current discrete perturbations in large language models.
method Learnable continuous latent vector transformations in embedding space, unbiased estimating equations, stochastic gradient descent optimization.
result Significant gains in out-of-domain settings over state-of-the-art methods.
A new efficient PGD method generates smaller perturbation adversarial examples.
problem Adversarial examples in deep neural networks.
method Modified Project Gradient Descent (PGD) method for ensemble models.
result Generates smaller perturbation adversarial examples than PGD method.
New technique stabilizes singular values in concatenated matrices.
problem How singular values of concatenated matrices relate to individual components.
method Developed perturbation technique extending classical results to concatenated matrices.
result Dominant singular values remain stable under small perturbations in submatrices.
Twistor space constructions and actions are given for full Yang-Mills and conformal gravity using almost complex structures that are not, in general, integrable. These are used as the basis of a derivation of the twistor-string generating functionals for tree level perturbative scattering amplitudes of Yang-Mills and c…
Proposes a new method to improve deep model security against adversarial deformations.
problem Deep neural networks' resistance to adversarial attacks, especially location perturbations.
method Regularizes flow gradients to provide a tighter bound and improve model resistance.
result Models trained with flow gradient regularization show better resistance to adversarial deformations compared to input gradient regularization and adversarial training.
Adversarial training adds dynamic perturbations to neural networks for robustness.
problem Accuracy trade-off and lack of diversity in adversarial examples.
method Dynamic adversarial perturbations in the parameter space of neural networks, updating perturbation biases during training.
result Adversarial training with negligible cost and reduced accuracy trade-off.
The paper studies heat kernels on modified manifolds and bounds their properties.
problem Bounding heat kernels on modified Riemannian manifolds.
method Derives upper bounds and gradient estimates for the heat kernel of (M,ildeg). result Establishes upper bounds and gradient estimates for the heat kernel of modified manifolds.
Although gradient descent (GD) almost always escapes saddle points asymptotically [Lee et al., 2016], this paper shows that even with fairly natural random initialization schemes and non-pathological functions, GD can be significantly slowed down by saddle points, taking exponential time to escape. On the other hand, g…
A new optimizer for deep learning improves accuracy and reduces training time.
problem Training deep neural networks for classification tasks.
method Hybrid Newton/Gradient Descent (NGD) method exploiting convexity of cross-entropy loss.
result Improves validation error and provides qualitative differences in hidden layer basis functions.
New strategy uses interpretability to improve adversarial learning.
problem Improving adversarial learning speed and effectiveness.
method Spatially constrained one-pixel adversarial perturbations guided by gradient-based interpretability.
result Spatially constrained one-pixel adversarial perturbations noticeably improve convergence and attack success.
This paper improves flow models to better handle perturbations in real-world data.
problem Flow models amplify initial errors in perturbed data, leading to poor generalization.
method Utilizes Bernstein-type polynomials to construct Normalizing Flows (NF) for higher robustness.
result Proposed NF framework provides theoretical upper bounds and practical advantages.
DANCE improves saliency maps by adding subtle input variations.
problem Poor performance of saliency methods in saturated gradients, adversarial perturbations, and inter-feature dependence.
method Two-step procedure: 1) Perturbation mechanism, 2) Aggregation of saliency maps.
result DANCE saliency method outperforms existing methods qualitatively and quantitatively.
SGD generalization bounds derived from information theory.
problem Understanding generalization of SGD for non-convex functions.
method Combining information-theoretic bounds with perturbation analysis.
result Upper bounds on SGD's generalization error based on gradient variance and function smoothness.
PWGF escapes saddle points in nonconvex optimization.
problem Escaping saddle points in nonconvex optimization.
method PWGF uses noisy perturbations via Gaussian process to escape saddle points.
result PWGF achieves second-order optimality for nonconvex objectives.
Tseytlin has recently proposed that an action functional exists whose gradient generates to all orders in perturbation theory the Renormalization Group (RG) flow of the target space metric in the worldsheet sigma model. The gradient is defined with respect to a metric on the space of coupling constants which is explici…