Optimizes CNNs by directing gradients along output channels.
problem Improving generalization error in CNNs.
method Output-channel directed re-weighted L2 or Sobolev metrics.
result Improves generalization error by optimizing gradients.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Gradient Directions”
Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better char…
Forward gradients improve neural network training without backpropagation issues.
problem Training neural networks without backpropagation's locking and memorization problems.
method Using directional derivatives in forward differentiation mode, with biased guesses based on feedback from small auxiliary networks.
result Using gradients from a local loss as a candidate direction improves Forward Gradient methods.
PGD-trained models have a preferential direction in their gradients, which improves robustness.
problem Mathematical lack of clarity in the direction of preferential gradient alignment after adversarial training.
method Proposed a novel definition of preferential direction and evaluated it using a metric based on GANs.
result PGD-trained models have higher alignment with the proposed preferential direction than baseline models.
New stochastic gradient descent with random search directions improves efficiency and convergence.
problem Efficiency and convergence of stochastic gradient descent methods.
method Developed a new class of stochastic gradient descent algorithms with random search directions.
result Established almost sure convergence and provided rates of convergence.
Optimal neural network approximation for Wasserstein gradient direction via convex optimization.
problem Approximating Wasserstein gradient direction with limited data.
method Two-layer networks with squared-ReLU activations, SDP relaxation.
result Optimal approximation of Wasserstein gradient direction in two-layer networks.
CProp scales learning rate based on past gradient direction.
problem Adaptive learning rate scheduling during training.
method Gradient scaling method based on past gradient conformity.
result Significant gain in training speed on SGD and adaptive methods.
Gradient-free method solves infinite-dimensional optimization problems.
problem Optimizing functions in infinite-dimensional spaces.
method Uses directional derivatives and a pre-basis for Hilbert space.
result Proves convergence for solving PDEs using PINNs.
Stochastic gradient descent outperforms traditional force-directed methods.
problem Improving graph layout quality and efficiency.
method Applying stochastic gradient descent for stress minimization.
result Stochastic gradient descent is simpler and more robust than traditional methods.
New methods validate a hypothesis explaining how neural nets generalize well.
problem Why over-parameterized nets generalize well despite memorizing training data.
method Developed new algorithms to suppress weak gradient directions without per-example gradients.
result Validated a hypothesis about gradient directions and their role in generalization.
Direct and indirect RL methods classified and compared.
problem Classifying RL methods for sequential decision making.
method Direct RL solves optimal policy directly, indirect RL solves Bellman equation.
result Direct and indirect RL methods are equivalent and can be unified.
To obtain groups with bounded harmonic functions (which are not hyperbolic), one of the most frequent way is to look at some semi-direct products (\eg lamplighter groups). The aim here is to show that many of these semi-direct products do not admit harmonic functions with gradient in , for .
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
In optimization, the negative gradient of a function denotes the direction of steepest descent. Furthermore, traveling in any direction orthogonal to the gradient maintains the value of the function. In this work, we show that these orthogonal directions that are ignored by gradient descent can be critical in equilibri…
Study shows directional convergence for neural networks under spherical symmetry.
problem Learning linear predictors with neural networks under spherically symmetric data.
method Analysis of gradient flow and gradient descent for two-layer and deep linear networks.
result Directional convergence guarantees with exact convergence rate for specific network architectures.
Paper tackles hyper-gradient estimation in decentralized FL over time-varying networks.
problem Excessive communication costs and inability to use robust networks.
method Introduces an optimality condition and uses Push-Sum for averaging model parameters and gradients over time-varying directed networks.
result Derives a hyper-gradient estimator that operates over time-varying directed networks and converges to the true hyper-gradient.
The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Modelsstat.ML
SGD benefits from a directional bias in kernel regression models.
problem Improving generalization in kernel regression models.
method Generalized directional bias property of SGD in kernel regression.
result SGD converges along the eigenvector of the largest eigenvalue of the Gram matrix.
Paper confirms Thom's conjecture for nonlinear evolutions on manifolds.
problem Thom's gradient conjecture for nonlinear evolution equations.
method Extending and settling the conjecture in infinite dimensional problems using Łojasiewicz, L. Simon, and Kurdyka-Mostowski-Parusinski's foundational works.
result Uniqueness of the limiting direction and characterization of convergence rates for both classical and infinite dimensional settings.
Direct optimization is an appealing framework that replaces integration with optimization of a random objective for approximating gradients in models with discrete random variables. A sampling is a framework for optimizing such random objectives over large spaces. We show how to combine these techniques to yiel…
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from hu…
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an appr…
Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
Gradients help find global optima in complex functions.
problem Finding global optima in functions with many local minima.
method A principle for generating search directions from non-local quadratic approximants based on gradients.
result The proposed algorithm and CMA-ES perform better than random reinitialized BFGS.
Push-SAGA is a decentralized algorithm for directed graphs that converges linearly.
problem Finite-sum minimization over directed graphs with stochastic gradients.
method Combines variance reduction, gradient tracking, and consensus algorithms.
result Achieves linear convergence for smooth and strongly convex problems.
Variance reduced stochastic optimization over directed graphs with row and column stochastic weightsmath.OC
AB-SAGA optimizes distributed optimization over directed graphs using variance reduction and stochastic weights.
problem Optimizing distributed stochastic optimization over directed graphs with stochastic weights.
method AB-SAGA combines variance reduction and network-level gradient tracking, using both row and column stochastic weights.
result AB-SAGA converges linearly to the global optimal with a constant step-size and achieves a linear speed-up over centralized methods.
Deep neural networks are usually trained with stochastic gradient descent (SGD), which minimizes objective function using very rough approximations of gradient, only averaging to the real gradient. Standard approaches like momentum or ADAM only consider a single direction, and do not try to model distance from extremum…
Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalstat.ML
SpecGD mitigates misalignment in phase retrieval models with anisotropic inputs.
problem Misalignment during gradient descent in phase retrieval models with anisotropic inputs.
method Spectral gradient descent modifies gradient updates to preserve directional information and remove spike amplification.
result SpecGD removes spike amplification, leading to stable alignment and accelerated noise contraction.
Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute t…
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Ratecs.LG
Stochastic Gradient Descent shows directional bias with moderate learning rates, impacting optimization outcomes.
problem Understanding the bias of SGD with moderate learning rates in practical scenarios.
method Analyzing SGD and GD on an overparameterized linear regression problem.
result SGD converges along large eigenvalue directions, GD along small ones, affecting early stopping outcomes.
The study characterizes heat flow and concentration on directed graphs with a lower Ricci curvature bound.
problem Understanding heat flow and concentration on directed graphs with a specific curvature bound.
method Characterization via gradient estimate and transportation inequality for the heat semigroup.
result Concentration of measure inequality for directed graphs with positive Ricci curvature.
Investigating the interaction between gradient-only line searches and different activation functionsstat.ML
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
New PG losses improve decision optimization in misspecified models.
problem Improving decision optimization in models that are not perfectly specified.
method Introducing Perturbation Gradient (PG) losses to connect decision loss with directional derivatives and optimizing using gradient techniques.
result PG losses yield best-in-class policies asymptotically, even in misspecified settings.
In this paper, we classify 3-dimensional complete gradient Yamabe solitons with divergence-free Cotton tensor. We also give some classifications of complete gradient Yamabe solitons with nonpositively curved Ricci curvature in the direction of the gradient of the potential function.
Deep networks converge in direction, with implications for predictions and margins.
problem Understanding convergence and alignment in deep learning networks.
method Developed a theory of unbounded nonsmooth Kurdyka-Łojasiewicz inequalities for functions definable in an o-minimal structure.
result Network weights, predictions, training errors, and margin distribution converge in direction and align with gradient flow.
Log-density gradient estimation is a fundamental statistical problem and possesses various practical applications such as clustering and measuring non-Gaussianity. A naive two-step approach of first estimating the density and then taking its log-gradient is unreliable because an accurate density estimate does not neces…
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show th…
Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscs.LG
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
Early training of deep neural networks leads to small, directionally converging weights.
problem Training dynamics of deep homogeneous neural networks with small initializations.
method Gradient flow analysis and study of KKT points for neural correlation function.
result Weights converge in direction to KKT points during early training stages.
Constructs explicit solutions to Spin(7)-structures gradient flow.
problem Finding explicit solutions to Spin(7)-structures gradient flow.
method Expressed Spin(7)-torsion tensor and gradient flow in terms of torsion forms; used these formulae to find solutions.
result Found explicit solutions including a shrinking soliton on SU(3) and another on a -bundle over .
Paper tackles efficient policy gradient estimation from off-policy data.
problem Estimating policy gradients from off-policy data is challenging and inefficient.
method Derives asymptotic lower bounds, proposes a meta-algorithm with 3-way robustness, and establishes convergence guarantees.
result Meta-algorithm achieves the lower bound on mean-squared error without parametric assumptions.
New risk class penalizes loss deviations from mean on both sides.
problem Current risks are sensitive to loss tails on the upside and ignore the downside.
method Introduces a bi-directional risk class with flexible tail sensitivity.
result Derives high-probability learning guarantees without gradient clipping.
The policy gradient theorem describes the gradient of the expected discounted return with respect to an agent's policy parameters. However, most policy gradient methods drop the discount factor from the state distribution and therefore do not optimize the discounted objective. What do they optimize instead? This has be…
Gradient descent, when applied to the task of logistic regression, outputs iterates which are biased to follow a unique ray defined by the data. The direction of this ray is the maximum margin predictor of a maximal linearly separable subset of the data; the gradient descent iterates converge to this ray in direction a…
The paper analyzes RLVR's training dynamics, proving convergence depends on aligning update direction with Gradient Gap.
problem Understanding why RLVR works and its limitations.
method Analysis of RLVR's training process at trajectory and token levels, introducing Gradient Gap.
result Convergence depends on aligning update direction with Gradient Gap, with a sharp step-size threshold.
A new pruning method finds sparse minimizers in flat regions of deep neural networks.
problem Finding sparse minimizers in flat regions of deep neural networks.
method Directional pruning using an proximal gradient algorithm.
result Empirical results show promising sparsity (92%) and comparable minima to SGD.
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show that - contrary to the case of isotropic noise - this variance is proportional to t…
Muon replaces matrix gradient with polar factor, optimizing flat spectrum updates
problem Optimization bias in matrix updates
method Using polar factor of gradient
result Muon update maximizes entropy among bounded updates
New method accelerates neural network training by focusing on flat directions.
problem Improving neural network training speed and stability.
method Bulk-SGD, interpolated gradient methods.
result Updates along the Dominant subspace can accelerate convergence but compromise stability.