Gradient descent benefits from tangent kernel advantages under specific conditions.
problem Comparing gradient descent with tangent kernel methods in learning.
method Analysis of gradient descent and tangent kernel methods under different conditions.
result Gradient descent can achieve small error only if tangent kernel methods have a non-trivial advantage, but this advantage can be very small.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
Changing kernel bandwidth during training improves kernel regression performance.
problem Improving kernel regression performance with varying model complexity.
method Investigated changing the bandwidth of a translational-invariant kernel during training for kernel regression using gradient descent.
result Kernel regression exhibits double descent behavior with decreasing model complexity (bandwidth).
Kernel ridgeless regression with random features shows good generalization without explicit regularization.
problem Generalization of kernel ridgeless regression without explicit regularization.
method Investigation of ridgeless regression with random features and stochastic gradient descent, exploring the effect of random features error and spectral density optimization.
result Random features error exhibits the double-descent curve, leading to improved generalization.
Deep networks are mathematically equivalent to kernel machines learned by gradient descent.
problem Understanding the learned representations of deep learning models.
method Using gradient descent to learn deep networks, showing they are equivalent to kernel machines.
result Deep network weights are a superposition of training examples, revealing the learned function.
We demonstrate that distributed block coordinate descent can quickly solve kernel regression and classification problems with millions of data points. Armed with this capability, we conduct a thorough comparison between the full kernel, the Nyström method, and random features on three large classification tasks from va…
The paper extends kernel ridge regression to product kernels and reveals new convergence behaviors.
problem Understanding kernel ridge regression in large dimensions with various kernels.
method Established a broad family of large dimensional kernels and derived convergence rates.
result Revealed new phenomena including minimax optimality, saturation effect, and multiple descent behavior.
Stochastic gradient descent optimizes Nyström samples for kernel matrix approximation.
problem Optimizing Nyström samples for kernel matrix approximation.
method Stochastic gradient descent applied to multisets of landmark points (Nyström samples) using a surrogate criterion (radial SKD).
result Local minimization of the radial SKD yields improved Nyström approximation accuracy.
New method uses multiple kernels to improve SVGD performance.
problem Sub-optimal performance of single kernel in SVGD.
method Combines multiple kernels to approximate optimal kernel, using Kernelized Stein Discrepancy (KSD) and constructing Multiple Kernel SVGD (MK-SVGD).
result Consistently matches or outperforms competing methods in experiments.
SimpleMKKM improves multi-kernel clustering efficiency.
problem Efficient multi-kernel clustering.
method Re-formulated minimization-maximization problem into a smooth minimization, solved with gradient descent.
result Outperforms state-of-the-art multi-kernel clustering alternatives.
This paper extends mirror descent to Banach spaces with reproducing kernels.
problem Optimizing in Banach spaces with reproducing kernels.
method Mirror descent algorithm adapted for Banach spaces with reproducing kernels.
result Mirror descent achieves linear convergence in certain conditions and standard convergence in a constrained setting.
KSD Descent uses KSD to sample from a target distribution efficiently.
problem Sampling from complex target distributions efficiently.
method Wasserstein gradient flow of KSD, using L-BFGS optimization.
result KSD Descent can sample from a target distribution using a set of particles.
Generalizes neural tangent kernel analysis for two-layer networks with noise and regularization.
problem Limitations of NTK analysis in deep learning practice.
method Generalized NTK analysis for two-layer neural networks with weight decay and gradient noise.
result Noisy gradient descent with weight decay exhibits 'kernel-like' behavior and converges linearly.
Paper connects neural networks to Gaussian processes for understanding double-descent.
problem Understanding the double-descent phenomenon in neural networks.
method Uses techniques from random matrix theory and Gaussian processes.
result Establishes a connection between NNGP and random matrix theory for neural networks.
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
Neural networks can learn useful representations that kernels can't.
problem Learning functions that depend on only a few relevant directions.
method Gradient descent on a two-layer neural network.
result Improved sample complexity for learning polynomials.
Neural networks generalize well despite overfitting due to high capacity.
problem Understanding why deep neural networks generalize well in overparameterized settings.
method High-dimensional asymptotic analysis of generalization under kernel regression with Neural Tangent Kernel.
result Test error exhibits non-monotonic behavior and can have additional peaks and descents in the overparameterized regime.
Ad-SVGD optimizes kernel parameters for SVGD, improving inference performance.
problem Efficiently approximating posterior distributions in Bayesian inference.
method Adaptive kernel selection for SVGD dynamics.
result Ad-SVGD outperforms standard heuristics in various tasks.
Paper analyzes SVGD algorithm for non-asymptotic convergence.
problem Optimizing a set of particles to approximate a target probability distribution.
method Finite time analysis of SVGD algorithm, providing descent lemma and convergence rates.
result SVGD algorithm decreases the objective at each iteration and converges to the target distribution.
Deep learning with noisy gradient descent outperforms linear estimators in high dimensions.
problem Theoretical explanation of deep learning's superiority over linear methods.
method Theoretical analysis of excess risk of a deep learning estimator trained by noisy gradient descent.
result Deep learning achieves a faster learning rate than linear estimators, especially in high dimensions.
Theoretical analysis explains why models generalize after overfitting in modular addition.
problem Understanding why models generalize after overfitting in modular addition.
method Theoretical analysis and gradient descent behavior of two-layer quadratic networks and Transformers.
result Two-layer quadratic networks and simple Transformers generalize well after initially overfitting, indicating grokking.
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
The paper analyzes rates for a modified gradient descent method using Stein variational gradients.
problem Improving the accuracy of gradient descent methods for complex target distributions.
method Derives finite-particle rates for regularized Stein variational gradient descent (R-SVGD).
result Establishes explicit non-asymptotic bounds for time-averaged empirical measures.
Study shows minimizing the norm of the ERM solution stabilizes kernel ridge-less regression.
problem Stability of kernel ridge-less regression.
method Minimizing the norm of the ERM solution to minimize CV stability.
result Interpolating solution with minimum norm minimizes CV stability.
Paper proposes adaptive parameter selection for KGD algorithms.
problem Improving parameter selection for kernel-based gradient descent.
method Integrates bias-variance analysis with splitting method, introduces empirical effective dimension.
result Adaptive parameter selection strategy achieves optimal generalization error bound.
NVGD uses neural networks to infer distributions without kernel choices.
problem Challenges in choosing kernel functions for SVGD.
method NVGD parameterizes the witness function of the Stein discrepancy with a neural network.
result NVGD achieves good performance on various inference problems.
This work closes the theory-practice gap for distributed optimization methods by introducing a new regularity condition.
problem Existing convergence conditions for distributed optimization methods are violated by nearly all kernels used in practice.
method Introduces Hessian relative uniform continuity (HRUC) to guarantee convergence under mild conditions.
result Derives convergence guarantees for mirror descent-based gradient tracking without restrictive assumptions.
SGD benefits from a directional bias in kernel regression models.
problem Improving generalization in kernel regression models.
method Generalized directional bias property of SGD in kernel regression.
result SGD converges along the eigenvector of the largest eigenvalue of the Gram matrix.
Quantum models show improved performance in overparameterized regimes.
problem Overfitting in quantum machine learning models.
method Analytical demonstration and numerical experiments on quantum kernel methods.
result Quantum models can operate in the modern, overparameterized regime without overfitting.
One-pass SGD converges in overparametrized neural networks with random data.
problem Understanding convergence of SGD in neural networks with streaming data.
method Overparameterized two-layer neural networks, one-pass SGD, random initialization, NTK eigen-decomposition, VC dimension, McDiarmid's inequality.
result Prediction error converges in expectation under one-pass SGD in overparametrized neural networks.
New method improves optimization algorithms without Lipschitz smoothness.
problem Improving optimization algorithms in the absence of Lipschitz smoothness.
method Dual kernel conditioning (DKC) to provide dual Lipschitz continuity.
result First complexity bounds and iterate convergence for random reshuffling mirror descent.
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
Gradient descent achieves optimal learning for elliptic PDEs via Sobolev norms.
problem Learning elliptic PDEs from noisy data.
method Gradient descent on Sobolev norm objective functions.
result Gradient descent achieves statistical optimality for elliptic PDEs.
The paper analyzes high-dimensional kernel regression, showing different risk curves based on data and regularization.
problem Characterizing generalization properties of high-dimensional kernel ridge regression.
method Bias-variance decomposition of the expected excess risk, considering different regularization schemes and data eigen-profiles.
result The risk curve of kernel regression can be double-descent-like, bell-shaped, or monotonic, depending on n, d, and regularization level.
We analyze kernel matrices in polynomial high-dimensional settings and explain double descent in KRR.
problem Understanding the spectrum of kernel matrices in polynomial high-dimensional settings and its implications for KRR risk.
method Generalized decomposition of kernel matrices into low-rank spike matrix, identity, and Gegenbauer matrix.
result The test error in KRR can exhibit double descent behavior, depending on effective regularization and signal-to-noise ratio.
The study analyzes how many neurons are needed for two-layer neural networks trained with gradient descent.
problem Determining the minimum number of neurons required for effective training of shallow neural networks.
method Analyzes two-layer neural networks in the NTK regime, trained with gradient descent. Derives fast rates of convergence and tracks the number of hidden neurons required for generalization.
result Derives fast rates of convergence and improves on existing results for the number of hidden neurons needed for generalization.
Gradient descent on MMD GAN parameter space converges globally to target distribution.
problem Convergence of gradient descent in Maximum Mean Discrepancy (MMD) GANs.
method Proposes a parametric kernelized gradient flow that mimics the min-max game in gradient regularized MMD GAN.
result Gradient descent on the generator's parameter space in gradient regularized MMD GAN is globally convergent to the target distribution under certain conditions.
Bayesian inference problems require sampling or approximating high-dimensional probability distributions. The focus of this paper is on the recently introduced Stein variational gradient descent methodology, a class of algorithms that rely on iterated steepest descent steps with respect to a reproducing kernel Hilbert …
TKRR improves KRR performance by aligning target functions with kernels.
problem Improving kernel ridge regression performance through target alignment.
method Focuses on truncated kernel ridge regression (TKRR) with an additional spectral truncation parameter.
result TKRR can achieve faster rates than full KRR, reaching parametric rates.
Unified derivation of high-dimensional linear models using stochastic gradient descent.
problem Performance analysis of high-dimensional linear models trained with stochastic gradient descent.
method Derivation of a deterministic equivalence for the two-point function of a random matrix resolvent.
result Unified understanding of model performance including previously known and novel results.
Study on how initialization scale affects neural network training regimes.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple depth-D models and empirical testing on complex models.
result Scale of initialization controls transition between kernel and rich regimes.
This paper presents a stochastic behavior analysis of a kernel-based stochastic restricted-gradient descent method. The restricted gradient gives a steepest ascent direction within the so-called dictionary subspace. The analysis provides the transient and steady state performance in the mean squared error criterion. It…
OKSVM optimizes RBF kernel hyperparameter for SVMs, improving classification performance.
problem Intrinsic dependence of RBF kernel hyperparameter on SVM performance.
method Gradient descent method for automatic hyperparameter learning and SVM weights adjustment.
result OKSVM outperforms classical SVM regardless of initial RBF hyperparameter values.
Gradient descent learns over-param neural nets better than NTK.
problem Learning over-parametrized neural networks with ReLU activations.
method Gradient descent from random initialization on a Gaussian input distribution.
result Gradient descent achieves population loss o(1/d), while NTK achieves Ω(1/d). Generalization in deep neural networks can be analyzed using minimax rates for gradient methods.
problem Generalization performance of over-parameterized neural networks
method Establishing a connection between gradient-based methods and kernel methods
result Deriving minimax-optimal rates for GD and SGD under polynomial network width scaling
Kernel-based online learning has often shown state-of-the-art performance for many online learning tasks. It, however, suffers from a major shortcoming, that is, the unbounded number of support vectors, making it non-scalable and unsuitable for applications with large-scale datasets. In this work, we study the problem …
Gradient descent methods for deep ReLU networks achieve optimal generalization rates.
problem Generalization of gradient descent methods for deep neural networks
method Establishing minimax-optimal rates for GD and SGD with deep ReLU networks
result Gradient descent methods for deep ReLU networks achieve optimal generalization rates
SRF improves kernel approximation and GP regression performance.
problem Efficient kernel approximation and Bayesian kernel learning in large-scale regression problems.
method Stein variational gradient descent to generate high-quality random features and approximate spectral measure posteriors.
result SRF outperforms traditional approaches in kernel approximation and GP regression.