Proposes VSGD optimizer combining probabilistic and gradient-based methods.
problem Uncertainty modeling in deep neural networks.
method Combines probabilistic and gradient-based approaches using SVI.
result VSGD outperforms Adam and SGD on image classification tasks.
Introduces new gradient-based methods for machine learning problems.
problem New challenges in machine learning due to decision-making and multi-agent problems.
method Gradient-based optimization and variational inequalities.
result Shifts focus from pattern recognition to decision-making and multi-agent problems.
Novel BSG method for efficient stochastic optimization.
problem Efficient optimization of non-convex surfaces in stochastic settings.
method Binary search combined with first order gradient optimization.
result BSG produces more promising results and better generalization than other methods.
This paper surveys gradient-based multi-objective deep learning methods.
problem Balancing multiple conflicting objectives in deep learning models.
method Gradient-based techniques adapted from Multi-Objective Optimization.
result Comprehensive survey of gradient-based multi-objective deep learning algorithms.
A novel gradient-based method optimizes decision trees for complex tasks.
problem Training decision trees with arbitrary differentiable loss functions.
method Gradient-based optimization using first and second derivatives of loss functions.
result Improves accuracy and flexibility in decision tree optimization.
Differentiable pipeline replaces non-differentiable CAE components for shape optimization.
problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.
Extends gradient-based optimization to spline functions.
problem Limitations of standard differentiable programming methods.
method Derives Jacobian of spline functions and uses it in predictive models.
result Improved performance in various applications.
Survey of theoretical foundations for policy optimization in control.
problem Understanding the theoretical properties of gradient-based methods in control and reinforcement learning.
method Interdisciplinary review of optimization landscape, convergence, and sample complexity for various control problems.
result Recent theoretical results on stability and robustness in learning-based control.
Gradient-based training and pruning for radial basis function networks in materials physics.
problem Interpretable and robust machine learning for materials physics problems.
method Gradient-based training and pruning of radial basis function networks with closed-form optimization criteria.
result Pruned models provide compact and interpretable versions of larger models, offering insights into atom-level migration processes.
New method creates universal perturbations to fool neural network interpretations.
problem Vulnerability of gradient-based saliency maps to adversarial perturbations.
method Gradient-based optimization and PCA-based approach to create UPI.
result Existence and successful application of Universal Perturbation for Interpretation (UPI).
Gradient-based optimization improves variational empirical Bayes regression.
problem Sparse, large-scale multiple regression models.
method Gradient-based optimization (GradVI) for variational empirical Bayes (VEB) regression.
result GradVI produces similar predictive performance to CAVI but converges faster and is faster in certain settings.
We propose a technique for increasing the efficiency of gradient-based inference and learning in Bayesian networks with multiple layers of continuous latent vari- ables. We show that, in many cases, it is possible to express such models in an auxiliary form, where continuous latent variables are conditionally determini…
New method improves combinatorial optimization by overcoming inefficient sampling.
problem Wandering in contours: sampling similar solutions for long periods.
method Reheated Gradient-based Discrete Sampling with a novel mechanism.
result Superior performance across various combinatorial optimization problems.
DenseHMM improves HMMs by learning dense representations that enable gradient-based optimization.
problem Learning dense representations for hidden states and observables in HMMs.
method DenseHMM uses kernelized transition probabilities and two optimization schemes.
result DenseHMM achieves superior performance and expressiveness compared to standard HMMs.
A new method for categorical variational inference using discrete normalizing flows.
problem Challenges in optimizing variational approximations for discrete latent variables.
method Differentiable reparameterization using a mixture of discrete normalizing flows.
result Improves optimization of evidence lower bound and reduces sensitivity to hyperparameters.
Graph neural networks (GNNs) which apply the deep neural networks to graph data have achieved significant performance for the task of semi-supervised node classification. However, only few work has addressed the adversarial robustness of GNNs. In this paper, we first present a novel gradient-based attack method that fa…
GFM models neural network training as a dynamical system to forecast final weights.
problem Computational intensity and inefficiency in training deep neural networks.
method Gradient Flow Matching (GFM) treats training as a dynamical system with learned vector fields.
result GFM achieves forecasting accuracy competitive with Transformer-based models and significantly outperforms classical baselines.
Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these …
New method combines QQA and gradient-based sampling for combinatorial optimization.
problem Scalability challenges in learning-based solvers for combinatorial optimization.
method Integrates gradient-based update through continuous relaxation with Quasi-Quantum Annealing (QQA) and parallel communication.
result Achieves superior speed-quality trade-offs for large-scale instances.
Develops shuffling gradient-based methods for nonconvex-concave minimax optimization.
problem Nonconvex-concave minimax optimization problems.
method Two shuffling gradient-based algorithms for nonconvex-linear and nonconvex-strongly concave settings.
result Achieves state-of-the-art oracle complexity in nonconvex optimization and best-known complexity bounds for nonconvex-strongly concave setting.
Paper proposes HTAF for stable training of binary neural networks.
problem Challenges in training binary neural networks with gradient-based optimization.
method HTAF is a smooth approximation to the Heaviside function that enables stable training.
result HTAF enables stable training of various binary neural networks with gradient-based optimization.
GSSBO reduces GP fitting time in Bayesian optimization.
problem High computational cost of fitting Gaussian process surrogate models in Bayesian optimization.
method Gradient-based sample selection to reduce the number of samples used in GP fitting.
result Sublinear regret bounds and significant reduction in computational cost.
A gradient-based method learns the structure of TAN for Bayesian network classifiers.
problem Learning the structure of Bayesian networks is difficult.
method A distribution over graph structures learned via gradient-based optimization.
result Consistently outperforms random and Chow-Liu TAN structures.
New method preserves convergence rates in gradient-based optimization.
problem How to discretize gradient-based optimization systems while preserving stability and convergence rates.
method Geometric framework for dissipative symplectic integration.
result Dissipative symplectic integrators preserve rates of convergence up to a controlled error.
A scalable gradient-based framework for sparse portfolio selection.
problem Sparse minimum-variance portfolio selection with cardinality constraint.
method Gradient-based optimization with Boolean relaxation and tunable parameter.
result Matches commercial solvers in most instances, differing by a few assets with negligible error in portfolio variance.
Optimizes shortfall risk using gradient-based methods.
problem Optimizing utility-based shortfall risk measures.
method Gradient-based stochastic optimization, non-asymptotic bounds derivation.
result Non-asymptotic convergence rate for optimizing UBSR.
FDS tackles long horizon hyperparameter optimization issues.
problem Memory scaling and gradient degradation in long horizon tasks.
method Forward-mode differentiation with sharing (FDS).
result Significantly outperforms greedy gradient-based alternatives.
Paper presents a novel gradient-based method for training models and hyperparameters simultaneously.
problem Achieving generalization in machine learning models.
method A novel gradient-based framework that trains parameters and hyperparameters simultaneously.
result Significantly smaller runtime compared to benchmark methods for equivalent prediction scores.
We analyze the convergence of gradient-based optimization algorithms that base their updates on delayed stochastic gradient information. The main application of our results is to the development of gradient-based distributed optimization algorithms where a master node performs parameter updates while worker nodes compu…
Neural networks perform differently when regression is treated as classification.
problem Understanding why neural networks perform better when regression is treated as classification.
method Analyzing two-layer ReLU networks and their feature spaces, focusing on the cross entropy loss vs. square loss.
result The support of the measure induced by the square loss differs from that of the cross entropy loss, indicating optimization difficulties.
Gradient-based methods are often used for policy optimization in deep reinforcement learning, despite being vulnerable to local optima and saddle points. Although gradient-free methods (e.g., genetic algorithms or evolution strategies) help mitigate these issues, poor initialization and local optima are still concerns …
Randomized gradient-based ensemble improves prediction accuracy.
problem Improving prediction accuracy in machine learning.
method Randomization and gradient-based aggregation of weakly-correlated estimators.
result The method outperforms existing techniques in terms of increased accuracy.
The paper proposes a gradient-based method for multi-penalty Ridge regression.
problem Optimizing multiple regularization hyperparameters for linear regression.
method Gradient-based optimization through matrix differential calculus.
result The method outperforms traditional regularization techniques like LASSO and Ridge.
K-Means and RBF networks are shown to be equivalent under certain conditions.
problem Discrete clustering vs. continuous optimization in machine learning.
method Established variational and gradient-based equivalence between K-Means and RBF networks.
result Gradient-based updates of RBF centers recover K-Means centroid update rule.
Characterizes corridors in loss surfaces for gradient-based optimization.
problem Understanding and mitigating training instabilities in gradient-based optimization.
method Characterizes corridors as regions where gradient descent and gradient flow trajectories are linearly related.
result Corridors indicate regions without implicit regularization effects, leading to better learning rate adaptation schemes.
EvoGrad improves efficiency in meta-learning and hyperparameter optimization.
problem Efficiently compute hypergradients for larger network architectures.
method Uses evolutionary techniques to estimate hypergradients without second-order derivatives or longer computational graphs.
result Significant improvements in efficiency, enabling scaling to bigger architectures.
Causal structure learning has been a challenging task in the past decades and several mainstream approaches such as constraint- and score-based methods have been studied with theoretical guarantees. Recently, a new approach has transformed the combinatorial structure learning problem into a continuous one and then solv…
We introduce a fully stochastic gradient based approach to Bayesian optimal experimental design (BOED). Our approach utilizes variational lower bounds on the expected information gain (EIG) of an experiment that can be simultaneously optimized with respect to both the variational and design parameters. This allows the …
We study local complexity measures for stochastic convex optimization problems, providing a local minimax theory analogous to that of Hájek and Le Cam for classical statistical problems. We give complementary optimality results, developing fully online methods that adaptively achieve optimal convergence guarantees. Our…
Proposes efficient calibration method for LIBOR Market Model with stochastic volatility.
problem Calibrating LIBOR Market Model with stochastic volatility.
method Derives analytical gradient of swaptions prices for DDSVLMM and uses it for gradient-based optimization.
result Analytical gradient-based calibration is highly competitive and efficient for DDSVLMM.
Gradient-based framework for optimizing text prompts in diffusion models.
problem Efficiently optimizing prompts in text-to-image diffusion models with large domain space and non-differentiable embeddings.
method Formulated as discrete optimization over language space, designed compact subspaces, and introduced shortcut text gradient.
result Empirically discovered prompts that enhance or destroy image faithfulness.
Optimal selection of a subset of items from a given set is a hard problem that requires combinatorial optimization. In this paper, we propose a subset selection algorithm that is trainable with gradient-based methods yet achieves near-optimal performance via submodular optimization. We focus on the task of identifying …
Fully automating machine learning pipelines is one of the key challenges of current artificial intelligence research, since practical machine learning often requires costly and time-consuming human-powered processes such as model design, algorithm development, and hyperparameter tuning. In this paper, we verify that au…
Adaptor 'E' extends gradient-based optimizers to explore loss landscapes, improving generalization.
problem Finding lower and better-generalizing minima in deep learning.
method Proposes an adaptor 'E' to extend gradient-based optimizers, encouraging exploration along landscape valleys.
result Adapted optimizers increase test accuracy by an average of 2.5% in large-batch training tasks.
We consider a class of a nested optimization problems involving inner and outer objectives. We observe that by taking into explicit account the optimization dynamics for the inner objective it is possible to derive a general framework that unifies gradient-based hyperparameter optimization and meta-learning (or learnin…
This work bridges competitive learning with gradient-based learning for faster feature extraction.
problem Lack of powerful feature extractors in competitive learning methods.
method Introduces gradient-based competitive layers for feature extraction.
result Demonstrates theoretical equivalence and faster convergence of gradient-based competitive layers.
We propose a novel method for gradient-based optimization of black-box simulators using differentiable local surrogate models. In fields such as physics and engineering, many processes are modeled with non-differentiable simulators with intractable likelihoods. Optimization of these forward models is particularly chall…
A novel decentralized deep learning algorithm using gradient-based optimization.
problem Decentralized deep learning in networked systems without a central server.
method Heavy-ball acceleration method and consensus protocol for model and gradient-momentum sharing.
result The proposed algorithm outperforms competing methods in various communication topologies.