Gradient-informed BNNs improve Bayesian optimization performance.
problem Improving Bayesian optimization with gradient information.
method Gradient-informed loss function for Bayesian neural networks.
result Gradient information enhances BNN predictions and BO convergence.
New research shows existing information-theoretic methods can't establish minimax rates for gradient descent in stochastic convex optimization.
problem Establishing minimax rates for gradient descent in stochastic convex optimization using information-theoretic methods.
method Examined several information-theoretic frameworks including input-output mutual information bounds, conditional mutual information bounds, PAC-Bayes bounds, and their variants.
result Proved that none of the examined information-theoretic frameworks can establish minimax rates for gradient descent in stochastic convex optimization.
AngularGrad optimizes CNNs by considering gradient direction, improving convergence.
problem Dying gradient problem and inefficiency in exploiting gradient curvature.
method AngularGrad considers gradient direction/angle, generating a score for step size control.
result AngularGrad outperforms state-of-the-art optimizers in benchmark tests.
Policy gradient method proves convergence in imperfect-information games.
problem Policy gradient methods in imperfect-information games (EFGs).
method Policy gradient approach with best-iterate convergence.
result Policy gradient leads to provable best-iterate convergence in self-play EFGs.
VB uses natural gradients in information geometry.
problem Estimating or computing natural gradients in VB.
method Natural-gradient descent algorithm and Bayesian Learning Rule.
result Simplification of Bayes' rule and generalization of quadratic surrogates.
Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.
We investigate connections between information-theoretic and estimation-theoretic quantities in vector Poisson channel models. In particular, we generalize the gradient of mutual information with respect to key system parameters from the scalar to the vector Poisson channel model. We also propose, as another contributi…
Develops methods to estimate gradient of EIG for Bayesian Experimental Design.
problem Optimizing Bayesian inference through efficient experimental design.
method Introduces UEEG-MCMC and BEEG-AP methods for estimating EIG gradient.
result Both methods improve upon existing benchmarks in EIG optimization.
A new metric GNQ audits LLMs for privacy risks during training.
problem Auditing LLMs for privacy risks during training is computationally hard.
method Gradient Uniqueness (GNQ) metric derived from gradient descent, BS-Ghost GNQ for efficiency.
result GNQ successfully predicts sequence extractability and reveals risk heterogeneity.
A new method for automatic gradient tree boosting using information theory.
problem Automatic selection of tree complexity and number in gradient boosting.
method Optimism of greedy leaf splitting procedure modeled as a Cox-Ingersoll-Ross process, leading to an information criterion for model selection.
result The method achieves significant speedups (10-1400) compared to xgboost without sacrificing predictive power.
The article reviews how gradient flow systems on hypergraphs connect to information geometry and nonequilibrium physics.
problem Understanding the geometry of perturbed gradient flow systems on hypergraphs.
method Formulating modern nonequilibrium principles within the framework of perturbed gradient flow systems on hypergraphs.
result New concepts like moduli spaces and thermodynamical area are introduced to understand speed limits.
A deep neural network is a hierarchical nonlinear model transforming input signals to output signals. Its input-output relation is considered to be stochastic, being described for a given input by a parameterized conditional probability distribution of outputs. The space of parameters consisting of weights and biases i…
This work improves OOD detection using deep generative models by approximating Fisher information metrics.
problem Deep generative models often incorrectly infer higher likelihoods for out-of-distribution data.
method Approximating Fisher information metrics using gradient norms of data points.
result The method outperforms existing OOD detection techniques.
Improved SVRG method using BB techniques for faster convergence.
problem Improving the convergence speed of stochastic variance reduction methods.
method Incorporates Barzilai-Borwein (BB) techniques as second-order information into SVRG.
result Proves linear convergence of the proposed method and its variants.
Gradient-free framework for Bayesian experimental design in complex systems.
problem Optimal experimental design in systems where gradient information is unavailable.
method Combines EKI and ALDI for optimization and sampling, with approximations for scalable utility estimation.
result Demonstrates robust, accurate, and efficient experimental design in various complex systems.
SGD generalization bounds derived from information theory.
problem Understanding generalization of SGD for non-convex functions.
method Combining information-theoretic bounds with perturbation analysis.
result Upper bounds on SGD's generalization error based on gradient variance and function smoothness.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
Gradient GA uses gradient information to improve molecular design.
problem Random walk exploration limits genetic algorithms' quality and speed in molecular design.
method Gradient GA incorporates gradient information from the objective function into genetic algorithms, using a differentiable neural network and Discrete Langevin Proposal.
result Significantly improves convergence speed and solution quality over traditional genetic algorithms.
Unified framework analyzes privacy risks from gradients in distributed learning.
problem Analyzing inference privacy risks from gradients in machine learning.
method Unified game-based framework for various attacks, including attribute, property, distributional, and user disclosures.
result Demonstrates inefficacy of data aggregation for privacy against inference attacks.
CAGES optimizes expensive RL problems by efficiently learning gradients from multiple sources.
problem Optimizing expensive-to-evaluate functions in high-dimensional spaces.
method Cost-Aware Gradient Entropy Search (CAGES) for multi-fidelity Bayesian optimization.
result Significant performance improvements on synthetic and RL benchmark problems.
Lower bounds show many sampling algorithms need many gradient queries.
problem Sampling from strongly log-concave densities in high dimensions.
method Information theory and stochastic gradient methods.
result Lower bound on number of gradient queries needed.
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed gradient methods operating with staled and compressed gradients. Non-asymptotic bo…
A scalable method for BED with implicit models using approximate gradients.
problem Efficiently estimating posterior distribution and maximizing MI for implicit models.
method Stochastic approximate gradient ascent with smoothed variational MI estimator.
result Significantly improves scalability of BED in high-dimensional problems.
Neural network learning is usually time-consuming since backpropagation needs to compute full gradients and backpropagate them across multiple layers. Despite its success of existing works in accelerating propagation through sparseness, the relevant theoretical characteristics remain under-researched and empirical stud…
The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.
problem Estimating gradients of smooth functions using noisy function evaluations.
method Information-theoretic lower bounds and finite difference method analysis.
result The finite difference method is not minimax optimal, suggesting room for improvement in gradient estimation.
NGD improves multivariate Gaussian inference by optimizing Fisher information.
problem Efficiently optimizing multivariate Gaussian models.
method Natural Gradient Descent applied to multivariate Gaussian parameters.
result NGD updates are more efficient for symmetric covariance matrices.
We study the problem of supervised linear dimensionality reduction, taking an information-theoretic viewpoint. The linear projection matrix is designed by maximizing the mutual information between the projected signal and the class label (based on a Shannon entropy measure). By harnessing a recent theoretical result on…
This research enhances ML models using gradient information from neural networks.
problem Improving the accuracy of machine learning models.
method Leveraging gradients extracted from neural networks to improve model performance.
result Gradient information can effectively enhance machine learning models with existing datasets.
New attack recovers user-level information from large batch images.
problem Recovering private information from user-level gradients in distributed learning.
method Proposes a gradient inversion attack using a denoising diffusion model as a prior.
result Demonstrates recovery of realistic facial images and private attributes.
Introduces new info-geometric structure for dynamics on graphs and hypergraphs.
problem Modeling dynamics on discrete structures like graphs and hypergraphs.
method Introduces two dually flat structures: one on vertex space and another on edge space.
result Extends gradient flows to include nonequilibrium dynamics.
Develops accelerated methods for optimization using low-dimensional projected-gradient information.
problem Optimization with low-dimensional projected-gradient information and Nesterov acceleration.
method Randomized-subspace Nesterov accelerated gradient methods for smooth convex and strongly convex optimization.
result Established accelerated oracle-complexity guarantees and unified basis for comparing sketch families.
Optimizes SGLD noise structure for better generalization bounds.
problem Improving generalization bounds for large models trained with SGLD.
method Manipulates the noise structure in SGLD to optimize information-theoretical bounds.
result Optimal noise covariance is the square root of the expected gradient covariance under certain constraints.
agtboost speeds up gradient tree boosting with automatic complexity adjustment.
problem Speeding up and simplifying gradient tree boosting computations.
method Adaptive gradient tree boosting with automatic complexity adjustment and feature importance.
result Significant decrease in computation time and simplification of model complexity.
GRAFFL learns Bayesian models without gradients for federated learning.
problem Privacy protection in federated learning during gradient updates.
method Gradient-free approach using summary statistics from SuffiAE.
result Feasibility and usefulness demonstrated in Gaussian mixture model.
OMGD algorithm optimizes online convex optimization with switching costs and delayed gradients.
problem Optimizing online convex optimization with switching costs and delayed gradients.
method Proposed an online multiple gradient descent (OMGD) algorithm for quadratic and linear switching costs.
result OMGD achieves optimal dynamic regret in the limited information setting.
In this paper, we propose and analyze zeroth-order stochastic approximation algorithms for nonconvex and convex optimization, with a focus on addressing constrained optimization, high-dimensional setting and saddle-point avoiding. To handle constrained optimization, we first propose generalizations of the conditional g…
Improved sampling from high-dimensional Gaussians using smoothed scores.
problem Sampling from high-dimensional Gaussian distributions with gradient information.
method Using smoothed scores, which are gradients of the logarithms of Gaussian-convolved densities, to overcome approximation barriers.
result Improved sampling efficiency with a complexity of \(O\left(\left(\logκ+\log(e\sqrt d/δ_{
m TV})
ight)\log(e\sqrt d/δ_{
m TV})
ight)\) smoothed-score queries.
Stochastic gradient algorithms have been the main focus of large-scale learning problems and they led to important successes in machine learning. The convergence of SGD depends on the careful choice of learning rate and the amount of the noise in stochastic estimates of the gradients. In this paper, we propose a new ad…
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…
Enhances optimization and sampling methods using ensemble-based gradient inference.
problem Improving ensemble-based methods for optimization and sampling.
method Ensemble-based gradient inference (EGI) to extract higher-order derivatives from particle ensembles.
result Augmented algorithms outperform gradient-free variants, especially in multimodal and non-Gaussian settings.
Bayesian optimization has been successful at global optimization of expensive-to-evaluate multimodal objective functions. However, unlike most optimization methods, Bayesian optimization typically does not use derivative information. In this paper we show how Bayesian optimization can exploit derivative information to …
This paper studies an acceleration technique for incremental aggregated gradient ({\sf IAG}) method through the use of \emph{curvature} information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications. Our technique utilizes a c…
Studying various functionals and associated gradient ows are known problems in differential geometry. The perpose of this article is to provide a general overview of curvature functionals in Finsler geometry and use their information for introducing different gradient ows on Finsler manifolds.
A neural network approach for feature selection using mutual information.
problem Feature ranking and selection leading to sub-optimal solutions for class separability.
method Stochastic mutual information gradient estimation for dimensionality reduction.
result The network projects features onto an output space maximizing mutual information with class labels.
MFAI uses gradient boosted trees to leverage auxiliary info for scalable Bayesian matrix factorization.
problem Matrix factorization struggles with poor data quality, especially high sparsity and low SNR.
method Integrates gradient boosted trees into probabilistic matrix factorization framework.
result MFAI effectively leverages auxiliary information, improving model performance.
Stein variational gradient descent (SVGD) is a particle-based inference algorithm that leverages gradient information for efficient approximate inference. In this work, we enhance SVGD by leveraging preconditioning matrices, such as the Hessian and Fisher information matrix, to incorporate geometric information into SV…
The paper analyzes rates for a modified gradient descent method using Stein variational gradients.
problem Improving the accuracy of gradient descent methods for complex target distributions.
method Derives finite-particle rates for regularized Stein variational gradient descent (R-SVGD).
result Establishes explicit non-asymptotic bounds for time-averaged empirical measures.
In this work, we improve upon the stepwise analysis of noisy iterative learning algorithms initiated by Pensia, Jog, and Loh (2018) and recently extended by Bu, Zou, and Veeravalli (2019). Our main contributions are significantly improved mutual information bounds for Stochastic Gradient Langevin Dynamics via data-depe…