Kernel-based online learning has often shown state-of-the-art performance for many online learning tasks. It, however, suffers from a major shortcoming, that is, the unbounded number of support vectors, making it non-scalable and unsuitable for applications with large-scale datasets. In this work, we study the problem …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “kernel-based gradient descent”
Paper proposes adaptive parameter selection for KGD algorithms.
problem Improving parameter selection for kernel-based gradient descent.
method Integrates bias-variance analysis with splitting method, introduces empirical effective dimension.
result Adaptive parameter selection strategy achieves optimal generalization error bound.
In this paper, we propose an adaptive stopping rule for kernel-based gradient descent (KGD) algorithms. We introduce the empirical effective dimension to quantify the increments of iterations in KGD and derive an implementable early stopping strategy. We analyze the performance of the adaptive stopping rule in the fram…
ASVGD accelerates SVGD for efficient sampling.
problem Slow SVGD in high-dimensional sampling.
method Accelerated gradient flow in a metric space of probability densities, using Nesterov's method and momentum-based updates.
result ASVGD outperforms SVGD and other methods in sampling efficiency.
Study local convergence of GDA for training GANs with kernel-based discriminators.
problem Analyzing the local dynamics of GDA for GANs with kernel-based discriminators.
method Linearization of a non-linear dynamical system, under an isolated points model assumption.
result Showed phase transitions indicating convergence, oscillation, or divergence of GDA.
This paper presents a stochastic behavior analysis of a kernel-based stochastic restricted-gradient descent method. The restricted gradient gives a steepest ascent direction within the so-called dictionary subspace. The analysis provides the transient and steady state performance in the mean squared error criterion. It…
We develop a multi-kernel based regression method for graph signal processing where the target signal is assumed to be smooth over a graph. In multi-kernel regression, an effective kernel function is expressed as a linear combination of many basis kernel functions. We estimate the linear weights to learn the effective …
The paper improves QD policy ensembles using distribution ratio estimators.
problem Training diverse and high-quality reinforcement learning agents.
method Using Stein variational gradient descent and distribution ratio estimators.
result The method generates diverse and high-quality reinforcement learning agents.
Neural networks can interpolate noisy data and still generalize well.
problem Generalization of neural networks trained on noisy data.
method Two-layer neural networks trained to interpolation by gradient descent on corrupted labels.
result Neural networks can achieve zero training error and optimal test error.
Generalization in deep neural networks can be analyzed using minimax rates for gradient methods.
problem Generalization performance of over-parameterized neural networks
method Establishing a connection between gradient-based methods and kernel methods
result Deriving minimax-optimal rates for GD and SGD under polynomial network width scaling
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
KSOS improves kernel learning for dynamical systems via global optimization.
problem Challenges in selecting optimal kernels and tuning parameters in traditional kernel-based methods.
method Global optimization framework with kernel-based surrogate functions.
result KSOS consistently outperforms gradient descent in predicting dynamical systems.
This paper analyzes convergence of FL for neural networks using NTK.
problem Theoretical guarantees of FL for neural networks with explicit forms and multi-step updates are unexplored.
method FL-NTK framework for federated learning of ReLU neural networks trained by gradient descent.
result FL-NTK converges to a global-optimal solution at a linear rate with proper learning parameters.
Paper introduces a new, tractable measure of model complexity.
problem Need for a reliable measure of model complexity.
method Mathematically rigorous measure based on gradient similarities.
result Generalizes to various model types and insights into double descent.
Stochastic gradient descent algorithms for training linear and kernel predictors are gaining more and more importance, thanks to their scalability. While various methods have been proposed to speed up their convergence, the model selection phase is often ignored. In fact, in theoretical works most of the time assumptio…
New algorithm for nonparametric IV regression using stochastic gradients.
problem Identifying causal effects in the presence of unobservable confounders.
method Functional stochastic gradient descent for NPIV regression.
result Superior stability and competitive performance compared to existing methods.
Convolutional Neural Networks, as most artificial neural networks, are commonly viewed as methods different in essence from kernel-based methods. We provide a systematic translation of Convolutional Neural Networks (ConvNets) into their kernel-based counterparts, Convolutional Kernel Networks (CKNs), and demonstrate th…
New method estimates variable importance for large models efficiently.
problem Estimating variable importance for large, opaque models is computationally challenging.
method Combining early stopping and warm-start techniques for scalable variable importance estimation.
result The method provides theoretical guarantees and demonstrates improved accuracy and computational efficiency.
Proposes learning manifold implicitly via heat kernel.
problem Direct manifold learning methods lack flexibility for down-stream applications.
method Implicit manifold learning using heat kernel.
result Framework achieves state-of-the-art results for data generation and Bayesian inference.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
Taming Nonconvexity in Kernel Feature Selection -- Favorable Properties of the Laplace Kernelmath.ST
Laplace kernel feature selection offers statistical guarantees for nonparametric models with few samples.
problem Statistical guarantees for kernel-based feature selection in nonconvex optimization problems.
method Sharp characterization of the gradient of the objective function for Laplace kernel feature selection.
result Model-selection consistency for Laplace kernel-based feature selection in nonparametric settings with samples.
ASVGD accelerates SVGD for efficient sampling from Gaussian targets.
problem Efficient sampling from Gaussian distributions using SVGD.
method Accelerated gradient flow in a metric space of probability densities, including momentum and Wasserstein regularization.
result ASVGD achieves optimal convergence rate for Gaussian targets, independent of covariance.
SVH-PSL uses Stein Variational Gradient Descent and Hypernetworks to improve Pareto set learning for expensive MOO.
problem Fragmented surrogate models and pseudo-local optima in expensive multi-objective optimization problems.
method SVH-PSL integrates Stein Variational Gradient Descent (SVGD) with Hypernetworks to address fragmentation and pseudo-local optima.
result SVH-PSL significantly improves the quality of the learned Pareto set, offering a promising solution for expensive MOO.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
A new method de-randomizes MCMC dynamics using the Stein operator.
problem Estimating complex target distributions in Bayesian inference.
method De-randomized kernel-based particle samplers that discretize the fiber-gradient Hamiltonian flow.
result GSVGD de-randomizes complex MCMC dynamics, maintaining high sample quality.
Optimizes LLM prompts using logged user feedback.
problem Naive approaches to optimizing LLM prompts suffer from high variance and bias.
method Kernel-based off-policy gradient method leveraging sentence similarity.
result Substantially reduces variance and suppresses bias in optimizing prompts.
In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
Kernel-based mean-field games use MMD penalties for interaction and target costs.
problem Optimizing mean-field games with specific cost functions.
method Kernel structure, random Fourier U-statistics, neural network training.
result Sample-level convergence theorem and rate of convergence proved.
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Sliced kernelized Stein discrepancy improves goodness-of-fit tests and model learning in high dimensions.
problem The curse-of-dimensionality in kernelized Stein discrepancy (KSD).
method Sliced Stein discrepancy and its scalable variants using optimal one-dimensional projections.
result Significantly outperforms KSD and baselines in goodness-of-fit tests and improves model learning.
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcond-mat.dis-nn
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Reparameterizes mirror descent as gradient descent for efficient sparse learning.
problem Efficiently training small sparse networks with mirror descent.
method Develops a framework to convert mirror descent updates into gradient descent updates on different parameters.
result Mirror descent can be reparameterized as gradient descent on modified parameters, facilitating standard backpropagation.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
Gravilon improves gradient descent for neural networks.
problem Improving efficiency and accuracy of gradient descent methods.
method Uses geometric modification of gradient step lengths.
result Promising experimental results on MNIST classification.
Langevin Dynamics for Adaptive Inverse Reinforcement Learning of Stochastic Gradient Algorithmscs.LG
Paper proposes Langevin dynamics for adaptive IRL of stochastic gradient algorithms.
problem Estimating reward functions from noisy gradient estimates of stochastic gradient agents.
method Generalized Langevin dynamics algorithm for IRL.
result Proposed algorithms asymptotically generate samples proportional to exp(R(θ)).
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.
problem Optimal convergence rates for non-convex functions under gradient noise.
method Geometric interpretation of the PL-condition to analyze convergence rates.
result Convergence rates of SGD and stochastic gradient descent match those of strongly convex quadratics.
Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.
problem Gradient descent's stability and sharpness behavior at the edge of instability.
method Cubic Taylor expansion analysis of gradient descent dynamics.
result Gradient descent at edge of stability implicitly follows projected gradient descent.
This work proposes a new method for variational inference using Wasserstein gradient descent.
problem Optimizing variational parameters to match a true posterior distribution.
method Reinterpreting VI as an optimization problem over a variational parameter space, using Wasserstein gradient descent.
result The proposed Wasserstein gradient descent can be seen as a generalization of existing optimization techniques in VI.