Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
A new method reduces the cost of solving large-scale linear models.
problem Solving large-scale linear statistical models efficiently.
method Sequential Preconditioned Conjugate Gradient Method (SPCG).
result SPCG achieves OLS prediction accuracy with fewer iterations and less time.
New recommendations improve Gaussian process accuracy and stability.
problem Numerical instabilities and poor test likelihoods in iterative Gaussian process learning.
method Investigated CG tolerance, preconditioner rank, and Lanczos decomposition rank. Recommended small CG tolerance and large root decomposition size.
result L-BFGS-B optimizer achieves convergence with fewer gradient updates, improving Gaussian process accuracy.
A new iterative K-FAC algorithm reduces training time and memory usage.
problem Training deep learning models efficiently.
method Uses conjugate gradient to approximate Fisher information matrix without generating the matrix or factors.
result Time and memory complexity of iterative CG-FAC is less than standard K-FAC.
New algorithms improve direction finding using prior signal knowledge.
problem Efficiently estimate signal direction from sensor data.
method Multi-step knowledge-aided iterative conjugate gradient algorithms.
result MS-KAI-CG algorithms outperform existing techniques in simulations.
The computational and storage complexity of kernel machines presents the primary barrier to their scaling to large, modern, datasets. A common way to tackle the scalability issue is to use the conjugate gradient algorithm, which relieves the constraints on both storage (the kernel matrix need not be stored) and computa…
This manuscript proposes a probabilistic framework for algorithms that iteratively solve unconstrained linear problems Bx=b with positive definite B for x. The goal is to replace the point estimates returned by existing methods with a Gaussian posterior belief over the elements of the inverse of B, which can …
Improved Gaussian process regression with tighter log marginal likelihood bounds.
problem Improving predictive performance in Gaussian process regression models.
method Lower bound on log marginal likelihood using conjugate gradients.
result Improved predictive performance compared to other conjugate gradient based approaches.
New algorithm selects best preconditioner for iterative methods.
problem Choosing optimal preconditioner for solving linear systems.
method Sketching methods to estimate preconditioner stability.
result Provable method selects minimal stability preconditioner.
New method uses robust estimators for Newton's method in empirical risk minimization.
problem Improving robustness in empirical risk minimization.
method Robust Newton's method with gradient and Hessian replaced by robust estimators.
result Faster convergence rates in high-dimensional settings.
Warm-start strategies speed up GP inference by 19x.
problem Efficient sequential inference in Gaussian processes.
method Three warm-start strategies exploiting smaller linear systems.
result Warm-starting achieves up to 19x speed-up in convergence.
Conjugate gradient methods improve efficiency for high-dimensional GLMMs.
problem Efficiency bottleneck in computing high-dimensional GLMM precision matrices.
method Combining spectral analysis and random graph theory with conjugate gradient methods.
result CG-based methods achieve linear scaling in cost with model parameters and observations.
Deep learning designs effective preconditioners for water engineering problems.
problem Solving large linear systems in water engineering applications.
method Convolutional Neural Network (CNN) for designing preconditioning matrices.
result Learned preconditioners improve convergence rates beyond existing methods.
This paper speeds up iterative GP inference with warm starting.
problem Improving scalability of Gaussian process inference.
method Warm starting sequential posteriors using known solutions.
result Significant speed-ups and improved Bayesian optimisation performance.
A new network reduces MIMO detection complexity.
problem Reducing computational complexity in massive MIMO systems.
method Learned conjugate gradient descent network (LcgNet) that learns step-sizes and integrates a quantizer.
result The network achieves promising performance with significantly reduced complexity.
NeuralIF uses neural networks to improve preconditioning for faster CG convergence.
problem Improving convergence of conjugate gradient method for large-scale sparse systems.
method Data-driven approach using graph neural networks to generate incomplete factorization.
result Data-driven preconditioners accelerate convergence of conjugate gradient method.
This work proves convergence of adaptive resampling for random Fourier features.
problem Sampling Fourier frequencies well for high-dimensional data.
method Data adaptive resampling of Fourier frequencies, asymptotically optimal.
result Proves convergence of adaptive resampling method for regression and classification problems.
DE via conjugate policies improves exploration and policy performance.
problem Effective exploration in policy gradient methods.
method DE via conjugate policies.
result DE improves policy performance and exploration effectiveness.
The paper studies the solution of stochastic optimization problems in which approximations to the gradient and Hessian are obtained through subsampling. We first consider Newton-like methods that employ these approximations and discuss how to coordinate the accuracy in the gradient and Hessian to yield a superlinear ra…
The conjugate gradient (CG) method is an efficient iterative method for solving large-scale strongly convex quadratic programming (QP). In this paper we propose some generalized CG (GCG) methods for solving the ℓ1-regularized (possibly not strongly) convex QP that terminate at an optimal solution in a finite numb…
Matrix factorization was used to generate investment recommendations for investors. An iterative conjugate gradient method was used to optimize the regularized squared-error loss function. The number of latent factors, number of iterations, and regularization values were explored. Overfitting can be addressed by either…
The paper speeds up hyperparameter optimisation in Gaussian processes.
problem Scaling hyperparameter optimisation to large datasets.
method Improvements to linear system solvers (pathwise gradient, warm starting, early stopping).
result Speed-ups of up to 72x and residual norm decreases of up to 7x.
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…
Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. We study two forms of sketching that…
EarlyStopping package helps prevent overfitting in iterative learning procedures.
problem Preventing overfitting in iterative learning procedures.
method Early stopping implemented in a Python package.
result Demonstrates how to use the EarlyStopping package to explore implicit regularization.
A fast metric learning framework using Gershgorin disc alignment.
problem Learning effective metrics for graph-based data.
method Fast projection-free metric learning via Gershgorin disc alignment.
result Efficiently computed graph metric matrices outperform competing methods.
Unified analysis of conjugate gradients and accelerated methods using duality gap.
problem Minimizing convex quadratic functions efficiently.
method Approximate Duality Gap Technique to unify conjugate gradients and accelerated methods.
result Unified and self-contained proof of conjugate gradients without relying on Chebyshev polynomials.
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can signific…
The study proves triviality and rigidity results for Ricci solitons and estimates their conjugate radius.
problem Understanding the properties and behavior of Ricci solitons.
method Analytical proofs and estimates for various types of Ricci solitons.
result Upper bounds and estimates for conjugate radius of Ricci solitons.
Two randomized algorithms improve hypergraph learning accuracy and efficiency.
problem Efficiently learning and tagging images in hypergraphs.
method Block randomized SVD and conjugate gradient method.
result Both methods achieve high accuracy and reduce computational requirements.
NCG methods improve shape optimization efficiency.
problem Shape optimization problems
method Nonlinear conjugate gradient methods
result NCG methods are efficient for shape optimization
Harmonic functions of two variables are exactly those that admit a conjugate, namely a function whose gradient has the same length and is everywhere orthogonal to the gradient of the original function. We show that there are also partial differential equations controlling the functions of three variables that admit a c…
Study compares methods for computing hypergradients in machine learning problems.
problem Computing exact hypergradients in machine learning is difficult.
method Investigates reverse mode iterative differentiation and approximate implicit differentiation methods.
result Unified analysis provides iteration complexity bounds and hierarchy of methods.
Proposes an extension of BayesCG for solving multiple linear systems.
problem Solving multiple linear systems efficiently and accurately.
method Extension of Bayesian conjugate gradient algorithm.
result Extension improves efficiency and accuracy of solving multiple linear systems.
A new method speeds up deep neural network training.
problem Nonconvex optimization in deep neural networks.
method Scaled conjugate gradient method for nonconvex optimization.
result The method converges faster and achieves lower scores in practical applications.
Improved kernel ridge regression using conjugate gradients.
problem Efficiently solving large-scale kernel ridge regression problems.
method Structured Gaussian regression model with low-rank approximation and conjugate gradients.
result Enhanced approximation of kernel ridge regressor/Gaussian process posterior mean.
New iterative methods improve scalability of Gaussian process approximations for large data.
problem Scalability issues in Gaussian process approximations for large spatial data.
method Iterative methods combined with preconditioners to reduce computational costs.
result Preconditioners accelerate convergence and improve predictive variances.
A new method for estimating large-scale linear models with improved precision.
problem Estimating large-scale linear statistical models efficiently.
method Sequential Least-Squares Estimators with Fast Randomized Sketching (SLSE-FRS), integrating Sketch-and-Solve and Iterative-Sketching methods.
result SLSE-FRS produces high-precision estimators, outperforming state-of-the-art methods.
We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard for…
NLCG optimizes DNN training, especially with large mini-batches.
problem Improving convergence speed in large-scale DNN training.
method Stochastic Preconditioned Nonlinear Conjugate Gradient (SP-NLCG) algorithm.
result NLCG improves DNN training accuracy by over 10 percentage points at large mini-batch sizes.
We present a new operation to be performed on elements in a Garside group, called cyclic sliding, which is introduced to replace the well known cycling and decycling operations. Cyclic sliding appears to be a more natural choice, simplifying the algorithms concerning conjugacy in Garside groups and having nicer theoret…
CNN outperforms other methods in gravity inversion.
problem Estimating subsurface density from gravitational field data.
method CNN, VAEs, GANs, iterative solvers (GD, GMRES, LGMRES, ICG).
result CNN provides the most reliable reconstructions.
A method uses CG to create efficient channels for ideal observers.
problem Computational intractability of ideal observers for high-dimensional image data.
method Conjugate gradient (CG) method for constructing efficient channels.
result CG-based channels approximate IO and HO performance efficiently.
The paper deals with learning probability distributions of observed data by artificial neural networks. We suggest a so-called gradient conjugate prior (GCP) update appropriate for neural networks, which is a modification of the classical Bayesian update for conjugate priors. We establish a connection between the gradi…
The matrix completion problem consists of finding or approximating a low-rank matrix based on a few samples of this matrix. We propose a new algorithm for matrix completion that minimizes the least-square distance on the sampling set over the Riemannian manifold of fixed-rank matrices. The algorithm is an adaptation of…
The techniques and analysis presented in this paper provide new methods to solve optimization problems posed on Riemannian manifolds. A new point of view is offered for the solution of constrained optimization problems. Some classical optimization techniques on Euclidean space are generalized to Riemannian manifolds. S…
New method for identifying autoregressive systems on manifolds.
problem Identifying autoregressive systems on Stiefel and Grassmann manifolds.
method Defining parameters as orthogonal group elements, averaging over observations, conjugate gradient descent on manifolds.
result System parameters can be estimated efficiently using the proposed algorithm.
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.