A new gradient descent method speeds up in flat regions and slows in steep directions.
problem Improving the speed and stability of gradient descent algorithms.
method Introducing a 'power gradient' where each gradient component is replaced by its H-th power, with 0<H<1. result The new gradient descent methods achieve significantly better performances, especially for Nesterov accelerated gradient and AMSGrad.
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
The paper analyzes the variance of different shuffling methods in stochastic gradient descent.
problem Understanding the variance of different shuffling methods in stochastic gradient descent.
method Power spectral density analysis to study the noise sequences of stochastic gradients.
result The stationary variances of iterates decrease in the order of SGD, SGD-RR, and SGD-SO.
Enhanced Gaussian process regression for multi-fidelity data fusion.
problem Combining data of varying fidelity levels for accurate predictions.
method Gradient-enhanced Cokriging method (GE-Cokriging) for QoI and its gradients.
result GE-Cokriging outperforms conventional multi-fidelity Cokriging in predicting QoI and gradients.
We prove gradient estimates for hypersurfaces in the hyperbolic space Hn+1, expanding by negative powers of a certain class of homogeneous curvature functions. We obtain optimal gradient estimates for hypersurfaces evolving by certain powers p>1 of F−1 and smooth convergence of the properly rescale…
GradPower speeds up language model training with minimal code changes.
problem Slower training of large language models.
method Elementwise sign-power transformation applied to gradients.
result Consistently lower terminal loss across various models and datasets.
PowerSGD compresses gradients for faster distributed optimization.
problem Communication bottleneck in data-parallel distributed optimization.
method Low-rank gradient compressor based on power iteration.
result Achieves test performance on par with SGD and consistent speedups.
Optimizes power allocation for WDM in RoFSO systems.
problem Maximizing total capacity with power and eye safety constraints.
method Model-based Stochastic Dual Gradient algorithm and model-free Primal-Dual Deep Learning algorithm.
result Deep Learning algorithm outperforms average equal power allocation.
Looped transformers with LN converge to power method for principal component prediction.
problem Understanding how transformers learn algorithmic procedures.
method Study of principal component prediction with looped linear transformers and layer normalization.
result Gradient descent trains looped transformers with LN to implement the power method for principal component prediction.
Paper uses ensemble learning for more accurate power flow modeling.
problem Improving accuracy and efficiency of power flow modeling.
method Applies polynomial regression and ensemble learning (GB, bagging) to create a more accurate linear power flow model.
result Data-driven model outperforms traditional methods in accuracy and speed.
Deep actor-critic learning optimizes power control in mobile networks.
problem Optimizing power control in large-scale wireless mobile networks.
method Multi-agent deep reinforcement learning with deep deterministic policy gradient.
result The algorithm maximizes a global utility function in a distributed manner.
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
This study proposes a logic architecture for the high-speed and power efficiently training of a gradient boosting decision tree model of binary classification. We implemented the proposed logic architecture on an FPGA and compared training time and power efficiency with three general GBDT software libraries using CPU a…
We prove that the evolution of weight vectors in online gradient descent can encode arbitrary polynomial-space computations, even in very simple learning settings. Our results imply that, under weak complexity-theoretic assumptions, it is impossible to reason efficiently about the fine-grained behavior of online gradie…
Differentiable learning via SGD and GD can simulate various learning problems, depending on precision and minibatch size.
problem Understanding the power of differentiable learning via SGD and GD compared to statistical query (SQ) learning.
method Comparing the learning power of SGD and GD on population and empirical losses with statistical query learning.
result The learning power of SGD and GD depends on the precision of gradient calculations relative to the minibatch size or sample size.
We present a model for random simple graphs with a degree distribution that obeys a power law (i.e., is heavy-tailed). To attain this behavior, the edge probabilities in the graph are constructed from Bertoin-Fujita-Roynette-Yor (BFRY) random variables, which have been recently utilized in Bayesian statistics for the c…
This work establishes properties on diffeological structures for set-valued maps and measures.
problem Establish rigorous properties on diffeological structures for set-valued maps and measures.
method Using diffeologies, the authors link various structures including set-valued maps, relations, gradients, measures, and shape analysis.
result Established rigorous properties on sample diffeologies.
It is a known fact that training recurrent neural networks for tasks that have long term dependencies is challenging. One of the main reasons is the vanishing or exploding gradient problem, which prevents gradient information from propagating to early layers. In this paper we propose a simple recurrent architecture, th…
A new method for efficient computation of Knowledge Gradient in Bayesian optimization.
problem Efficient computation of the Knowledge Gradient for Bayesian optimization.
method One-shot Hybrid KG, a new approach combining previous ideas.
result The new method is cheap to compute and preserves theoretical properties of previous methods.
New findings show neural network training loss follows a power law over time.
problem Understanding the optimization process of neural networks during training.
method Spectral analysis of the integral operator representing the linearized evolution of a large network.
result The loss function in neural network training follows a power law behavior, L(t)∼t−ξ, with exponent ξ determined by network parameters and data characteristics. Study evaluates capacity and trainability of parametrized quantum circuits.
problem Finding the best type of circuits for hybrid quantum-classical algorithms.
method Geometric structure of parameter space, effective quantum dimension, and circuit expressiveness.
result Identifies a transition in quantum geometry leading to decay of quantum natural gradient for deep circuits.
Derives an approximate solution for power utility optimization under predictable returns.
problem Optimizing portfolios with power utility functions under predictable returns.
method Approximate analytical solution using multivariate normal distribution and gradient descent algorithm.
result Gradient descent method provides a viable alternative to Taylor series expansion for portfolio optimization.
New factorial power constants improve optimization convergence rates.
problem Optimization convergence rates depend on various constants.
method Proposes using factorial powers for defining these constants.
result Factorial powers simplify or improve convergence rates of optimization methods.
Greedy PIG adapts integrated gradients for better feature attribution.
problem Interpreting deep learning model predictions.
method Unified discrete optimization framework for feature attribution and selection.
result Greedy PIG improves feature attribution on various tasks.
This work analyzes neural scaling laws using power-law data spectra and derives analytical expressions for generalization error.
problem Understanding how neural network performance scales with key factors like data size and model complexity.
method Statistical mechanics techniques applied to one-pass stochastic gradient descent in a student-teacher framework.
result Derivation of analytical expressions for generalization error under power-law data spectra and identification of conditions for power-law scaling.
New attack method improves neural network security.
problem Vulnerability of deep neural networks to adversarial attacks.
method Ensemble adversarial attack with long-term gradient memory.
result Enhanced robustness against adversarial attacks.
SGLB boosts machine learning with Langevin diffusion for multimodal loss functions.
problem Dealing with multimodal loss functions in machine learning.
method Stochastic Gradient Langevin Boosting (SGLB) based on Langevin diffusion equation.
result SGLB guarantees global convergence for multimodal loss functions.
New method uses diffusion models to solve inverse problems.
problem Solving ill-posed inverse problems with powerful priors.
method Formulate posterior sampling as a regularized Wasserstein gradient flow in latent space.
result Demonstrates improved performance on standard benchmarks.
Stein variational gradient decent (SVGD) has been shown to be a powerful approximate inference algorithm for complex distributions. However, the standard SVGD requires calculating the gradient of the target density and cannot be applied when the gradient is unavailable. In this work, we develop a gradient-free variant …
Gradient boosting with randomized trees reduces discontinuities and complexity.
problem Discontinuities in regression functions due to sparse training data.
method Gradient boosting machine with partially randomized decision trees.
result Improves robustness and computational efficiency of gradient boosting.
The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge in this scenario is how to reduce the variance of policy gradient estimates for reliable policy updates. In this paper, we combine the followi…
Interpreting gradient methods as fixed-point iterations, we provide a detailed analysis of those methods for minimizing convex objective functions. Due to their conceptual and algorithmic simplicity, gradient methods are widely used in machine learning for massive data sets (big data). In particular, stochastic gradien…
New Stein operator improves robustness in model inference.
problem Improving robustness in inference for unnormalized models.
method Density-power weighted Stein operator (γ-Stein operator). result Robust methods for goodness-of-fit testing and posterior approximation.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
Gradient descent benefits from tangent kernel advantages under specific conditions.
problem Comparing gradient descent with tangent kernel methods in learning.
method Analysis of gradient descent and tangent kernel methods under different conditions.
result Gradient descent can achieve small error only if tangent kernel methods have a non-trivial advantage, but this advantage can be very small.
Untuned SGD converges but with an exponential dependence on smoothness, adaptive methods prevent this.
problem The exponential dependence on smoothness in untuned SGD's convergence rate.
method Untuned SGD with arbitrary stepsize η, adaptive methods like NSGD, AMSGrad, and AdaGrad.
result Adaptive methods prevent the exponential dependence on smoothness in SGD.
LLMs learn peaked distributions slowly due to power-law losses.
problem Slow convergence of loss in training large language models.
method Systematic analysis of toy models and empirical evaluation of LLMs.
result Power-law time scaling with an exponent of 1/3 for learning peaked distributions.
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD(λ) - preconditioned gradient descent method. result ScaledGD(λ) converges at a constant linear rate after a logarithmic number of iterations. Understanding the influence of features in machine learning is crucial to interpreting models and selecting the best features for classification. In this work we propose the use of principles from coalitional game theory to reason about importance of features. In particular, we propose the use of the Banzhaf power inde…
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
New method for efficient sketching of gradients and Hessians.
problem Memory constraints in training machine learning models.
method A novel framework for scalable gradient and HVP sketching tailored for modern hardware.
result Theoretical guarantees and practical applications in training data attribution and Hessian spectrum analysis.
Paper uses ML to predict SME defaults with interpretability.
problem Lack of interpretability in ML models for SME default prediction.
method Model-agnostic approach using Accumulated Local Effects and Shapley values.
result eXtreme Gradient Boosting algorithm provides highest classification power with interpretability.
In this paper, we study the convexity, interior gradient estimate, Liouville type theorem and asymptotic behavior at infinity of translating solutions to mean curvature flow as well as the nonlinear flow by powers of the mean curvature.
Graph neural networks are explained through energy gradient flow and framelet decomposition.
problem Understanding and improving graph neural networks.
method Viewing framelet-based models as gradient flows of energy, proposing a generalized energy via framelet decomposition.
result The proposed model leads to more flexible dynamics, enhancing graph neural networks.
New framework for tracking varying bounds in time series forecasting.
problem Forecasting bounded time series with varying bounds.
method Extended log-likelihood estimation, online maximum likelihood estimation, Normalized Gradient Descent (NGD) for quasiconvex optimization.
result Derive an Online Normalized Gradient Descent algorithm for online bound tracking.
A method to improve gradient boosting models using stacking.
problem Improving the performance of gradient boosting models.
method Proposes a stacking algorithm to learn a meta-model for ensembles of gradient boosting models.
result The proposed approach can be extended to differentiable combination models like neural networks.
Graph Neural Tangent Kernels combine GNNs and GKs for better graph classification.
problem Limited expressive power of graph kernels and difficulties in training graph neural networks.
method Graph Neural Tangent Kernels (GNTKs) are infinitely wide multi-layer GNNs trained by gradient descent.
result GNTKs achieve strong performance on graph classification datasets.
First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. W…