We introduce a hybrid stochastic estimator to design stochastic gradient algorithms for solving stochastic optimization problems. Such a hybrid estimator is a convex combination of two existing biased and unbiased estimators and leads to some useful property on its variance. We limit our consideration to a hybrid SARAH…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Hybrid Gradient Descent”
Hybridizes CEM and gradient descent for efficient model-predictive control.
problem Efficiently planning optimal action sequences in high-dimensional spaces.
method Interleaves Cross-Entropy Method (CEM) and gradient descent steps.
result Faster convergence and avoidance of local optima compared to CEM.
A new method for optimizing functions without gradients, improving efficiency and convergence.
problem Optimizing functions without gradient information in machine learning.
method Hybrid Gradient Descent (HGE) using random and coordinate-wise gradient estimates.
result The proposed method achieves optimal convergence rates in convex cases and generalizes to non-convex cases.
Hybrid model for online nonlinear prediction using LSTM and soft GBDT.
problem Online nonlinear prediction with manual feature selection and model selection issues.
method End-to-end architecture with LSTM for feature extraction and soft GBDT for regression, jointly optimized.
result Significant performance improvements over conventional methods on real datasets.
Coded computation techniques provide robustness against straggling servers in distributed computing, with the following limitations: First, they increase decoding complexity. Second, they ignore computations carried out by straggling servers; and they are typically designed to recover the full gradient, and thus, canno…
Hybrid regularization avoids double descent in random feature models.
problem Avoiding the double descent phenomenon in random feature models.
method Combines early stopping and weight decay, using GCV for hyperparameter selection.
result Hybrid method successfully avoids double descent and achieves comparable generalization.
Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The g…
New optimization method combines gradient clipping and non-Euclidean smoothness.
problem Improving optimization in non-Euclidean spaces for machine learning.
method Hybrid of steepest descent and conditional gradient, incorporating weight decay.
result Achieves optimal convergence rate and demonstrates effectiveness in deep learning.
Stein variational neural network ensembles improve diversity and uncertainty estimation.
problem Lack of proper Bayesian justification and diversity guarantees in deep neural network ensembles.
method Particle-based inference methods, specifically Stein variational gradient descent (SVGD), operating in weight space, function space, and hybrid settings.
result SVGD methods improve diversity and uncertainty estimation, approaching the true Bayesian posterior more closely.
This paper improves traffic flow modeling by using multi-gradient descent algorithms for physics-informed machine learning.
problem Combining physics-based and data-driven approaches in traffic flow modeling.
method Introducing multi-gradient descent algorithms to explore the Pareto front in a multi-objective setting.
result Multi-gradient descent algorithms significantly outperform scalarization-based methods in complex PIML scenarios.
SBS uses SVGD to optimize continuous functions globally.
problem Global optimization of continuous Sobolev functions.
method Stein Boltzmann Sampling (SBS) with SVGD.
result SBS and its variants are highly competitive in global optimization.
A new hybrid algorithm reduces stochastic gradient evaluations for nonconvex optimization.
problem Solving stochastic composite nonconvex optimization problems efficiently.
method Proposes a new hybrid variance-reduced proximal gradient method with a stochastic gradient estimator.
result Achieves optimal stochastic oracle complexity bound with one less gradient evaluation.
Hybrid Policy Optimization tackles reinforcement learning in hybrid spaces, improving performance over PPO.
problem Credit assignment issues and biased gradients in hybrid discrete-continuous action spaces.
method Mixed gradient estimator combining pathwise and score-function gradients, reformulating problems in hybrid form.
result HPO substantially outperforms PPO on inventory control and switched systems, with performance gaps increasing with continuous action dimension.
A new optimizer for deep learning improves accuracy and reduces training time.
problem Training deep neural networks for classification tasks.
method Hybrid Newton/Gradient Descent (NGD) method exploiting convexity of cross-entropy loss.
result Improves validation error and provides qualitative differences in hidden layer basis functions.
RS-PINN uses randomized smoothing to speed up high-dimensional PDE simulations without sacrificing accuracy.
problem High computational cost and bias in PINNs for high-dimensional PDEs.
method Introduces Gaussian noise for stochastic smoothing of PINNs, enabling Monte Carlo derivative approximation.
result Proposes bias correction techniques and a hybrid method to optimize the bias-variance trade-off.
Hybrid tensor networks improve machine learning by combining quantum and classical methods.
problem Limitations of regular tensor networks in machine learning.
method Quantum-classical hybrid tensor networks (HTN) combining tensor networks and classical neural networks.
result HTN overcomes limitations of regular tensor networks and enables deep learning training.
Motivated by the gap between theoretical optimal approximation rates of deep neural networks (DNNs) and the accuracy realized in practice, we seek to improve the training of DNNs. The adoption of an adaptive basis viewpoint of DNNs leads to novel initializations and a hybrid least squares/gradient descent optimizer. We…
A hybrid strategy forecasts short-term loads using Warm-start Gradient Tree Boosting.
problem Lack of effective short-term load forecasting methods.
method Hybrid strategy integrating four different inference models: tree-based ensemble method Warm-start Gradient Tree Boosting (WGTB).
result Demonstrates effectiveness of hybrid strategy on real datasets.
A new hybrid-ordered SGD method reduces communication and complexity for non-convex optimization.
problem Balancing communication, computational complexity, and convergence rate in distributed non-convex optimization.
method Hybrid-ordered distributed SGD with pre-shared scalers and periodic vector communication.
result Order-wise faster convergence compared to existing methods.
New algorithmic view of ℓ2 regularization using ODEs and path-following methods.
problem Optimizing convex loss functions with ℓ2 regularization.
method Established an equivalence between ℓ2-regularized solution paths and ODEs, proposing path-following algorithms based on homotopy methods and numerical ODE solvers.
result The solution path can be viewed as a hybrid of gradient descent and Newton method, providing novel schemes to choose grid points and reducing computational cost.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
Study benchmarks methods for learning non-Cartesian k-space trajectories and reconstruction.
problem Benchmarking methods for learning non-Cartesian k-space trajectories and reconstruction.
method Comparing PILOT, BJORK, and HybLearn schemes to learn non-Cartesian k-space trajectories and reconstruction.
result HybLearn scheme outperforms other methods in learning and comparing non-Cartesian k-space trajectories and reconstruction.
Model predicts S&P500 volatility more accurately than existing models.
problem Improving accuracy of volatility and market risk forecasts.
method Stacked model using Gradient Descent Boosting, Random Forest, SVM, and Artificial Neural Network.
result The model outperforms other models in forecasting S&P500 volatility.
In this work we show that Evolution Strategies (ES) are a viable method for learning non-differentiable parameters of large supervised models. ES are black-box optimization algorithms that estimate distributions of model parameters; however they have only been used for relatively small problems so far. We show that it …
This paper provides a geometrical derivation of the Hybrid Minimum Principle (HMP) for autonomous hybrid systems whose state manifolds constitute Lie groups which are left invariant under the controlled dynamics of the system, and whose switching manifolds are defined as smooth embedded time invariant subma…
Traditionally, neural networks are parameterized using optimization procedures such as stochastic gradient descent, RMSProp and ADAM. These procedures tend to drive the parameters of the network toward a local minimum. In this article, we employ alternative "sampling" algorithms (referred to here as "thermodynamic para…
In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
Synthetic images rendered by graphics engines are a promising source for training deep networks. However, it is challenging to ensure that they can help train a network to perform well on real images, because a graphics-based generation pipeline requires numerous design decisions such as the selection of 3D shapes and …
Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.
problem How to optimize deep neural networks without explicit regularization.
method Backward error analysis to calculate implicit gradient regularization and demonstrate its effectiveness empirically.
result Implicit gradient regularization biases gradient descent toward flat minima, improving model robustness and test errors.
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcond-mat.dis-nn
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
Gravilon improves gradient descent for neural networks.
problem Improving efficiency and accuracy of gradient descent methods.
method Uses geometric modification of gradient step lengths.
result Promising experimental results on MNIST classification.
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Gradient descent can take exponentially long to escape saddle points in 2D.
problem Worst-case inefficiency of gradient descent in non-convex optimization.
method Analysis of gradient descent's performance on 2D functions.
result Gradient descent can take exponentially long to escape saddle points.
SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.
problem Optimal convergence rates for non-convex functions under gradient noise.
method Geometric interpretation of the PL-condition to analyze convergence rates.
result Convergence rates of SGD and stochastic gradient descent match those of strongly convex quadratics.
The min-max problem, also known as the saddle point problem, is a class of optimization problems which minimizes and maximizes two subsets of variables simultaneously. This class of problems can be used to formulate a wide range of signal processing and communication (SPCOM) problems. Despite its popularity, most exist…
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
problem Combining robust on-policy methods with efficient offline data for hybrid RL.
method Integrates off-policy training on offline data into on-policy NPG framework.
result Achieves state-of-the-art theoretical guarantees and maintains on-policy NPG guarantees.
Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.
problem Gradient descent's stability and sharpness behavior at the edge of instability.
method Cubic Taylor expansion analysis of gradient descent dynamics.
result Gradient descent at edge of stability implicitly follows projected gradient descent.
Semi-decentralized federated learning combines device-to-server and device-to-device communications for faster convergence.
problem Faster convergence in federated learning with decentralized model training.
method Two timescale hybrid federated learning (TT-HF) with cooperative D2D model aggregations.
result Achieves sublinear convergence rate of O(1/t) with adaptive control algorithm.
This work proposes a new method for variational inference using Wasserstein gradient descent.
problem Optimizing variational parameters to match a true posterior distribution.
method Reinterpreting VI as an optimization problem over a variational parameter space, using Wasserstein gradient descent.
result The proposed Wasserstein gradient descent can be seen as a generalization of existing optimization techniques in VI.