arXiv research

Generalizes NTK for surrogate gradient learning in neural networks.

Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute t…

Gradient-informed BNNs improve Bayesian optimization performance.

Hyperboost uses gradient boosting for hyperparameter optimization, outperforming state-of-the-art methods.

A new training method uses multilevel minimization for machine learning.

GUESS improves surrogate model accuracy with adaptive sampling.

New PG losses improve decision optimization in misspecified models.

Study proposes a differentiable surrogate loss function for optimizing $F_β$ score in binary classification with imbalanced data.

FRA-Attack improves adversarial transferability for closed-source MLLMs by aligning visual focus across models.

We address the problem of training models with black-box and hard-to-optimize metrics by expressing the metric as a monotonic function of a small number of easy-to-optimize surrogates. We pose the training problem as an optimization over a relaxed surrogate space, which we solve by estimating local gradients for the me…

A new method uses natural gradients for efficient distribution optimization.

Differentiable pipeline replaces non-differentiable CAE components for shape optimization.

Learning with non-modular losses is an important problem when sets of predictions are made simultaneously. The main tools for constructing convex surrogate loss functions for set prediction are margin rescaling and slack rescaling. In this work, we show that these strategies lead to tight convex surrogates iff the unde…

We propose a novel method for gradient-based optimization of black-box simulators using differentiable local surrogate models. In fields such as physics and engineering, many processes are modeled with non-differentiable simulators with intractable likelihoods. Optimization of these forward models is particularly chall…

The problem of maximizing precision at the top of a ranked list, often dubbed Precision@k (prec@k), finds relevance in myriad learning applications such as ranking, multi-label classification, and learning with severe label imbalance. However, despite its popularity, there exist significant gaps in our understanding of…

New algorithm optimizes PAC-Bayes bound without surrogate loss.

New method uses surrogate gradients to train efficient spiking networks on neuromorphic hardware.

Proposes a method for inference in high-dimensional classification with non-differentiable surrogate losses.

Surrogate models provide a low computational cost alternative to evaluating expensive functions. The construction of accurate surrogate models with large numbers of independent variables is currently prohibitive because it requires a large number of function evaluations. Gradient-enhanced kriging has the potential to r…

Stein variational gradient decent (SVGD) has been shown to be a powerful approximate inference algorithm for complex distributions. However, the standard SVGD requires calculating the gradient of the target density and cannot be applied when the gradient is unavailable. In this work, we develop a gradient-free variant …

MFNets constructs efficient multifidelity surrogates from diverse information sources.

Develops a gradient-enhanced approach for online estimation in high-dimensional generalized linear models with streaming data.

New framework tackles high-dimensional reliability analysis using surrogate models and active subspaces.

A new framework improves reinforcement learning algorithms with policy guarantees.

New method optimizes black-box functions using generative models and Wasserstein distance.

We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be optimized via stochastic gradient descent and we prove tight bounds on the so-called "calibration func…

Bayesian surrogate models reduce uncertainty in high-dimensional design optimisation problems.

New research shows existing information-theoretic methods can't establish minimax rates for gradient descent in stochastic convex optimization.

We consider the black-box adversarial setting, where the adversary has to generate adversarial perturbations without access to the target models to compute gradients. Previous methods tried to approximate the gradient either by using a transfer gradient of a surrogate white-box model, or based on the query feedback. Ho…

UREs lead to overfitting in complex models, especially in complementary label learning.

Sparse Bayesian learning improves rational approximations for complex-valued models.

New algorithms handle online prediction with bandit and delayed feedback, improving regret bounds.

New method speeds up Bayesian inverse problem solving with neural operators.

Develops a transparent surrogate model for complex data.

In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as …

Improved MALA method for neural networks uncertainty quantification.

Proposes efficient Bayesian logistic regression for large sparse datasets.

PDMP samplers improve Bayesian PDE coefficient inference.

New method optimises learning via surrogate PAC-Bayes bounds.

A standard introduction to online learning might place Online Gradient Descent at its center and then proceed to develop generalizations and extensions like Online Mirror Descent and second-order methods. Here we explore the alternative approach of putting Exponential Weights (EW) first. We show that many standard meth…

PPO's gradients are heavy-tailed, affecting learning; a robust estimator improves performance.

NoFAS combines variational inference and adaptive surrogate models for efficient inference of computationally expensive models.

Proposes SPFB method for optimizing partition functions in stochastic learning.

This work improves neural likelihood surrogates for stochastic models with a score-augmented loss.

SVH-PSL uses Stein Variational Gradient Descent and Hypernetworks to improve Pareto set learning for expensive MOO.

We propose a class of very simple modifications of gradient descent and stochastic gradient descent. We show that when applied to a large variety of machine learning problems, ranging from logistic regression to deep neural nets, the proposed surrogates can dramatically reduce the variance, allow to take a larger step …

Unified approach for federated learning using MM optimization.

New methods improve black-box adversarial attacks with fewer queries.