Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its or…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Renyi divergence”
New divergences help audit DP in high dimensions.
problem Challenges in auditing DP in high-dimensional data.
method Propose kernel Rényi divergence and its regularized version for auditing.
result Regularized kernel Rényi divergence can be estimated from samples in high dimensions.
The paper improves semi-supervised learning using -divergences and -Rényi divergences.
problem Improving semi-supervised learning with noisy pseudo-labels.
method Inspired by -divergences and -Rényi divergences, the paper develops new empirical risk functions and regularization techniques.
result The new methods show better performance than traditional self-training methods, especially in noisy pseudo-label scenarios.
New variational formula for Rényi divergences improves neural network estimation in high dimensions.
problem Estimating Rényi divergences in high-dimensional systems.
method Derive and apply a variational formula for Rényi divergences over various function spaces.
result Neural network estimators of Rényi divergences are consistent under certain conditions.
Rényi Neural Processes replace KL divergence with Rényi divergence to improve NP performance.
problem Parameterization coupling in Neural Processes leads to prior misspecification.
method Propose Rényi Neural Processes (RNP) by replacing KL divergence with Rényi divergence.
result Significant performance improvements in real-world problems, including better log-likelihoods.
RényiCL uses Rényi divergence for robust contrastive learning with stronger data augmentations.
problem Learning useful representations from multiple data views with hard augmentations.
method RényiCL employs Rényi divergence for contrastive learning, using a novel variational objective to manage hard negative sampling.
result RényiCL achieves better performance with stronger augmentations compared to other methods.
New divergences improve estimation and GAN training performance.
problem Improving estimation and training in machine learning models.
method Function-space regularized Rényi divergences.
result New divergences reduce variance and improve training performance.
Tail-Sensitive KL and Rényi Convergence of Unadjusted Hamiltonian Monte Carlo via One-Shot Couplingsstat.ML
This work improves convergence guarantees for unadjusted HMC in KL and Rényi divergences.
problem Understanding convergence properties of unadjusted HMC in divergences like KL and Rényi.
method One-shot couplings to establish regularization and lift convergence bounds.
result Quantitative control of relative density mismatch and warm-start requirements.
The paper solves portfolio selection using Rényi divergence and optimization.
problem Single-period portfolio selection under CRRA utility.
method Information-theoretic lens, Rényi divergence, Rényi entropy, Blahut-Arimoto-style alternating optimization.
result CRRA portfolio selection is equivalent to a Rényi information-projection problem.
New PAC-Bayesian bounds for multi-view learning using Rényi divergence.
problem Applying PAC-Bayesian theory to multi-view learning.
method Introducing novel PAC-Bayesian bounds based on Rényi divergence for multi-view learning.
result Efficient optimization algorithms that align with theoretical bounds.
Paper introduces RCaI, a risk-sensitive control method using Rényi divergence.
problem Risk-sensitive control in reinforcement learning.
method RCaI extends CaI using Rényi divergence variational inference.
result Risk-sensitive optimal policy can be obtained by solving a soft Bellman equation.
All you need is logcs.IT
A new family of multi-distribution divergences is characterized for fairness and other problems.
problem Comparing more than two probability distributions in a consistent way.
method Characterized a new family of multi-way coincidence divergences.
result The new family of multi-distribution Rényi divergences is necessary and arises from several independent routes.
All you need is logcs.IT
All you need is log
problem Comparing multiple probability distributions
method Characterizing multi-distribution Rényi divergences
result Found a canonical multi-distribution Rényi calculus
This paper introduces the variational Rényi bound (VR) that extends traditional variational inference to Rényi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is co…
Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main cont…
New sampler improves uniform sampling over convex bodies with fewer queries.
problem Improving uniform sampling over convex bodies with fewer queries.
method Proximal sampler with uniform ergodicity and annealing scheme.
result Converges in Rényi-infinity divergence with query complexity.
LMC algorithm converges to target in Chi-squared and Renyi divergence.
problem Sampling from target distribution using LMC with strong dissipativity and smoothness conditions.
method LMC algorithm with strong dissipativity and first-order smoothness, initialized with Gaussian.
result LMC reaches ε-neighborhood of target in Chi-squared and Renyi divergence in O(λ²dε⁻¹) steps.
Unified framework for deriving generalization bounds in supervised learning.
problem Generalization error bounds in supervised learning.
method Data Processing Inequality PAC-Bayesian framework.
result Unified bounds on binary Kullback-Leibler generalization gap for various divergences.
This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distri…
We consider nonparametric estimation of , Renyi- and Tsallis- divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on cor…
We derive the optimal differential privacy (DP) parameters of a mechanism that satisfies a given level of Rényi differential privacy (RDP). Our result is based on the joint range of two -divergences that underlie the approximate and the Rényi variations of differential privacy. We apply our result to the moments acc…
Disagreement is an essential element of science and life in general. The language of probabilities and statistics is often used to describe disagreements quantitatively. In practice, however, we want much more than that. We want disagreements to be resolved. This leaves us with a substantial knowledge gap which is ofte…
Proposes an active RBI framework using Rényi information measures for more informed decision-making.
problem Optimal latent variable estimates in real-time settings with streaming noisy observations.
method Unified inference and query selection steps through Rényi entropy and α-divergence; new objective called Momentum for exploration.
result Analytically demonstrates superior performance compared to conventional methods like mutual information.
Rényi Pufferfish Privacy: General Additive Noise Mechanisms and Privacy Amplification by Iterationcs.CR
Enhances privacy in machine learning through Rényi Pufferfish mechanisms.
problem Designing general and efficient Pufferfish mechanisms that maintain privacy and utility.
method Introduces a Rényi divergence-based variant of Pufferfish, generalizes the Wasserstein mechanism, and proves privacy amplification results.
result Extends the applicability of Pufferfish framework and provides stronger privacy guarantees.
New method for privacy amplification without sampling for matrix factorization.
problem Privacy amplification for differentially private model training with matrix factorization.
method Sampling-free bounds based on Rényi divergence and conditional composition.
result Stronger privacy guarantees for small ε, applicable to various matrices.
New algorithm uniformly samples high-dimensional convex bodies efficiently.
problem Uniform sampling of high-dimensional convex bodies.
method Stochastic diffusion perspective to show contraction to the target distribution.
result Achieves state-of-the-art runtime complexity with strong guarantees on output.
Reduces policy space complexity for reinforcement learning.
problem Efficiency in exploring vast policy spaces in reinforcement learning.
method Uses Rényi divergence and norm to determine sample size for accurate policy approximation.
result Established error bounds for sample size requirements in model-based and model-free settings.
The paper develops divergences for Gaussian processes and RKHS settings.
problem Estimating divergences in infinite-dimensional spaces.
method Formulations of Alpha Log-Det divergences, continuity in norm, laws of large numbers, consistent estimation from finite samples.
result Infinite-dimensional divergences can be estimated from finite-dimensional versions with dimension-independent sample complexities.
New framework controls generalization for heavy-tailed data in RLHF and SGLD.
problem Heavy-tailed data in modern learning pipelines.
method Tail-dependent information-theoretic framework for sub-Weibull data.
result Sharp generalization bounds for heavy-tailed data.
Differential privacy is a de facto standard in data privacy, with applications in the public and private sectors. A way to explain differential privacy, which is particularly appealing to statistician and social scientists is by means of its statistical hypothesis testing interpretation. Informally, one cannot effectiv…
We study the asymptotic consistency properties of -Rényi approximate posteriors, a class of variational Bayesian methods that approximate an intractable Bayesian posterior with a member of a tractable family of distributions, the member chosen to minimize the -Rényi divergence from the true posterior. Unique to o…
New GAN loss functions improve image generation quality and stability.
problem Improving the performance of GANs in generating high-quality images.
method Introducing least kth-order GAN (LGAN) and Rényi-centric GAN loss functions.
result The proposed loss functions lead to better image quality and stability.
For nonlinear supervised learning models, assessing the importance of predictor variables or their interactions is not straightforward because it can vary in the domain of the variables. Importance can be assessed locally with sensitivity analysis using general methods that rely on the model's predictions or their deri…
Enhanced DeepONet framework with uncertainty quantification for complex operators.
problem Learning complex operators with uncertainty quantification.
method Generalised variational inference (GVI) using Rényi's α-divergence.
result Superior predictive accuracy and uncertainty quantification.
The Dirichlet mechanism protects privacy while minimizing KL divergence.
problem Minimizing KL divergence while protecting sensitive data privacy.
method Using the exponential mechanism with the KL divergence loss function, resulting in the Dirichlet mechanism.
result Proved a probability tail bound on KL divergence and derived a lower bound for sample complexity.
New method improves understanding of machine learning model performance.
problem Understanding how well machine learning models generalize from training data to unseen data.
method Auxiliary Distribution Method to derive new generalization error bounds.
result Upper bounds on generalization errors are tighter and more applicable.
Surprisal-Rényi Free Energystat.ML
SRFE clarifies KL divergences without unifying learning frameworks.
problem Inductive biases of KL divergences and their limitations.
method Introducing SRFE, a log-moment-based functional of the likelihood ratio.
result SRFE recovers KL divergences as limits and reveals a mean-variance tradeoff.
Faster algorithm for sampling logconcave densities in high dimensions.
problem Cubic barrier in sampling logconcave densities from a cold start.
method Two key ingredients: weaker distance sampling and refined log-Sobolev inequality.
result First sub-cubic sampling algorithms for isotropic position.
The paper derives a new theorem for predicting batches of data.
problem Finding lower bounds on minimal batch regret.
method Derives a conditional version of the regret-capacity theorem.
result Reveals a connection between conditional Rényi divergence and conditional Sibson's mutual information.
Improved privacy analysis for stochastic gradient descent.
problem Analyzing privacy leakage in noisy stochastic gradient descent.
method Modeling Rényi divergence dynamics with Langevin diffusions, proving exponential privacy loss convergence for smooth and strongly convex objectives.
result Privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size.
Study on privacy leakage in noisy gradient descent algorithms.
problem Information leakage of iterative randomized learning algorithms about training data.
method Analyzes the dynamics of Rényi differential privacy loss in noisy gradient descent algorithms.
result Privacy loss converges exponentially fast for smooth and strongly convex loss functions.
Study on affine surface areas and their inequalities for convex bodies.
problem Understanding weighted affine surface areas in convex bodies.
method Investigating valuations, isoperimetric inequalities, and connections to divergences.
result Established isoperimetric inequalities for weighted affine surface areas.
Optimal Clustering of Discrete Mixtures: Binomial, Poisson, Block Models, and Multi-layer Networksmath.ST
Paper optimizes clustering for multi-layer networks and discrete mixtures.
problem Optimizing clustering in multi-layer networks and discrete mixtures.
method Two-stage method: tensor-based initialization and likelihood-based refinement.
result Achieves minimax optimal error rate for multi-layer networks and discrete mixtures.
We propose a direct estimation method for Rényi and f-divergence measures based on a new graph theoretical interpretation. Suppose that we are given two sample sets and , respectively with and samples, where is a constant value. Considering the -nearest neighbor (-NN) graph of in the j…
Geometric Renyi Differential Privacy: Ricci Curvature Characterized by Heat Diffusion Mechanismsstat.ML
Paper develops novel privacy mechanism for Riemannian manifold data using geometric analysis and heat diffusion.
problem Privacy-preserving estimation of generalized Frechet mean on Riemannian manifolds.
method Characterizes Renyi divergence via Harnack inequalities, introduces mechanisms based on heat diffusion and Langevin process.
result Proposes mechanisms for nonnegative and general Riemannian manifolds with detailed utility analyses.
Threshold found for hyperbolicity in random Coxeter groups.
problem Determining the hyperbolicity threshold in random Coxeter groups.
method Analyzing random right-angled Coxeter groups via Erdős-Rényi graphs and combinatorial properties.
result Threshold for relative hyperbolicity in random Coxeter groups.
A novel stepwise VI method using vine copulas for complex latent dependence.
problem Modeling complex latent dependence structures in probabilistic models.
method Stepwise estimation of vine copula parameters using Rényi divergence and a stopping criterion.
result Our method outperforms mean-field VI and is more parsimonious in complex applications.
This short note highlights some links between two lines of research within the emerging topic of trustworthy machine learning: differential privacy and robustness to adversarial examples. By abstracting the definitions of both notions, we show that they build upon the same theoretical ground and hence results obtained …