Study compares chi-squared divergence and KL-divergence posteriors for PAC-Bayesian bounds.
problem Investigates optimal posteriors for PAC-Bayesian bounds using chi-squared divergence.
method Analyzes bounds for three distance functions, derives FP equations for computation.
result Chi-squared divergence based posteriors have weaker bounds and worse test errors.
A deep learning approach for clustering time series of varying lengths.
problem Clustering time series with variable lengths and temporal relations.
method Recurrent Deep Divergence-based Clustering framework.
result Outperforms previous methods on benchmark datasets.
Paper studies S-rectangular DR-RL models for robust reinforcement learning with near-optimal sample complexity.
problem Addressing distributional discrepancies in reinforcement learning environments.
method Empirical value iteration algorithm for divergence-based S-rectangular DR-RL models.
result Near-optimal sample complexity bound of O(∣S∣∣A∣(1−γ)−4ε−2). Approaches KL divergence for learning multi-sense word distributions.
problem Capturing the polysemy and uncertainty of words in word embeddings.
method Modeling words as multi-sense Gaussian mixtures and using KL divergence for learning.
result The proposed approach effectively captures word entailment and distribution similarity.
Theoretical proof shows COMs are a type of contrastive divergence model with improved sampling.
problem Improving sampling quality in offline model-based optimization.
method Showed COMs are contrastive divergence models, proposed Langevin MCMC sampler, and decoupled model.
result Improved sampling quality achieved by decoupling model and using Langevin MCMC.
Develops a KL-divergence-based deep learning method for survival analysis with short data.
problem Challenges of training deep learning models with limited survival data.
method Kullback-Leibler (KL) divergence to integrate external and internal data.
result Proposed model achieves better performance and higher robustness.
New methods solve non-Lipschitz smooth problems with guaranteed convergence.
problem Non-Lipschitz smooth problems in machine learning and signal processing.
method Bregman-divergence based algorithms for relatively smooth problems.
result Guaranteed convergence to second-order stationary points for any relatively smooth problem.
Paper derives a new lower bound for KL-divergence using HCRB.
problem Estimating KL-divergence between distributions.
method Using Hammersley-Chapman-Robbins bound and information geometry.
result New lower bound for KL-divergence derived from HCRB.
New method for bidirectional generative modeling using adversarial gradient estimation.
problem Bidirectional generative modeling with various f-divergences. method Adversarial gradient estimation for f-divergence optimization. result Similar algorithms for different f-divergences with varying scaling. New privacy framework improves data analysis security.
problem Weaknesses in existing privacy definitions, especially in composition and subsampling.
method Introduces f-differential privacy, a new relaxation that avoids composition and subsampling issues. result Privacy guarantees converge to Gaussian differential privacy (GDP) under composition.
We consider the problem of maximum a posteriori (MAP) inference in discrete graphical models. We present a parallel MAP inference algorithm called Bethe-ADMM based on two ideas: tree-decomposition of the graph and the alternating direction method of multipliers (ADMM). However, unlike the standard ADMM, we use an inexa…
New algorithm estimates barycenters of distributions using Frank-Wolfe.
problem Estimating the average of arbitrary probability distributions.
method Frank-Wolfe optimization for Sinkhorn divergence, incrementally populating support.
result Converges in both discrete and continuous distributions, with proven rates.
The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asympt…
Optimal posterior distributions improve SVM classifiers and parameter selection.
problem Improving SVM classifiers and selecting optimal regularization parameters.
method PAC-Bayesian approach with optimal posterior identification for stochastic classifiers.
result Optimal posteriors yield tight risk bounds and improved SVM performance.
Study on geometric Jensen-Shannon divergence for Gaussian measures in Hilbert space.
problem Computing divergence between Gaussian measures in infinite-dimensional Hilbert space.
method Closed form expression and regularization for divergence calculation.
result Closed form expression and regularization for Geometric Jensen-Shannon divergence.
New PAC-Bayesian bounds for multi-view learning using Rényi divergence.
problem Applying PAC-Bayesian theory to multi-view learning.
method Introducing novel PAC-Bayesian bounds based on Rényi divergence for multi-view learning.
result Efficient optimization algorithms that align with theoretical bounds.
Improved UDA framework using f-divergence measures.
problem Addressing distribution shifts in machine learning.
method Refined f-divergence-based discrepancy and f-domain discrepancy. result Novel target error and sample complexity bounds.
A new method for averaging model predictions using minimum divergence.
problem Improving model averaging methods, especially in small samples.
method Minimum divergence framework for model weight calculation.
result Empirically outperforms standard model averaging methods.
Sharp bounds for high-probability estimation of discrete distributions.
problem Estimating discrete distributions with high probability under χ2-divergence. method Sharp upper and lower bounds for the classical Laplace estimator, and characterization of minimax high-probability risk for any estimator.
result Sharp bounds for high-probability estimation of discrete distributions can be achieved through a simple smoothing strategy.
Flexible framework integrates machine learning and DRO for uncertain parameter prediction.
problem Limited joint observations of uncertain parameters and covariates.
method Wasserstein, sample robust optimization, and phi-divergence-based ambiguity sets.
result Validation of theoretical and practical benefits in limited data scenarios.
LLM safety alignment explained as divergence estimation.
problem Aligning large language models to avoid harmful outputs.
method Presented a theoretical framework showing alignment methods as divergence estimators.
result KLDO method improves safety alignment using compliance-refusal datasets.
EBMs improve continual learning without external memory or regularization.
problem Improving continual learning without external memory or regularization.
method Energy-Based Models with contrastive divergence training objective.
result EBMs outperform baseline methods on various benchmarks.
A new VIS approach improves log-likelihood estimation in latent variable models.
problem Challenges in achieving high log-likelihood with VI for complex posterior distributions.
method Uses forward χ2 divergence to optimize proposal distribution for better log-likelihood estimation. result Consistently outperforms state-of-the-art baselines in log-likelihood and parameter estimation.
Hutch++ optimizes trace estimation for generative models, reducing variance and improving quality.
problem High variance and scalability issues in Hutchinson estimators for generative models.
method Hutch++ is an optimal stochastic trace estimator designed to minimize training variance while maintaining transport optimality.
result Hutch++ leads to higher quality generations and effective variance reduction in various applications.
Paper introduces Wasserstein total correlation for disentangled representation learning.
problem Learning disentangled representations from data.
method Adversarial training of a critic to estimate Wasserstein total correlation in variational and Wasserstein autoencoders.
result Proposed method achieves comparable disentanglement performance with less reconstruction loss.
The paper develops a method to approximate arbitrary Bregman divergences from supervision.
problem Approximating an arbitrary Bregman divergence from supervision.
method Develops a formulation and algorithm for learning arbitrary Bregman divergences by approximating their convex generating function via a piecewise linear function.
result The method achieves a generalization error of Op(m−1/2) for metric learning, matching known bounds. Outliers are ubiquitous in modern data sets. Distance-based techniques are a popular non-parametric approach to outlier detection as they require no prior assumptions on the data generating distribution and are simple to implement. Scaling these techniques to massive data sets without sacrificing accuracy is a challeng…
New deep clustering network uses divergence measures for unlabeled data.
problem Discovering cluster structure in unlabeled data without supervision.
method Discriminative loss function incorporating geometric regularization.
result Competitive performance on synthetic and real datasets.
New framework using Jensen-Shannon divergence improves domain adaptation theory.
problem Incoherence between empirical domain adversarial training and theoretical H-divergence. method Established new theoretical framework based on Jensen-Shannon divergence, derived bi-directional upper bounds.
result Framework exhibits flexibilities for various transfer learning problems.
Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs t…
Robust GW distance improves graph data alignment.
problem Outliers in GW distance lead to inaccurate comparisons.
method Optimistically perturbed marginal constraints within a Kullback-Leibler divergence-based ambiguity set.
result RGW reduces inaccuracies in graph data alignment.
ATL learns from many streaming processes without labeled data.
problem Knowledge transfer across many streaming processes with covariate shift and drifts.
method Autonomous transfer learning with generative and discriminative phases, KL divergence optimization, and elastic network structure.
result Improved performance and faster training speed compared to existing methods.
New method estimates densities using Sobolev regularization, outperforming existing algorithms.
problem Non-parametric density estimation with clear inductive bias.
method Regularizes Sobolev norm of density, approximates kernel via sampling, uses natural gradients for optimization.
result Method ranks second best on ADBench anomaly detection benchmark.
This work extends t-SNE to f-divergences for better visualization of data.
problem Visualizing high-dimensional data with improved accuracy and structure capture.
method Extending t-SNE to f-divergences, analytically and empirically evaluating different types of latent structure.
result Different f-divergences perform better for different types of latent structure.
Conditional Restricted Boltzmann Machines (CRBMs) are rich probabilistic models that have recently been applied to a wide range of problems, including collaborative filtering, classification, and modeling motion capture data. While much progress has been made in training non-conditional RBMs, these algorithms are not a…
The paper introduces a new risk assessment framework using φ-divergence.
problem Assessing risk and decision-making in uncertain conditions.
method Introduces a novel framework called the φ-Divergence Quadrangle.
result Provides a more nuanced understanding of risk through φ-divergence.
Enhances privacy in machine learning through Rényi Pufferfish mechanisms.
problem Designing general and efficient Pufferfish mechanisms that maintain privacy and utility.
method Introduces a Rényi divergence-based variant of Pufferfish, generalizes the Wasserstein mechanism, and proves privacy amplification results.
result Extends the applicability of Pufferfish framework and provides stronger privacy guarantees.
Extends reinforcement learning alignment to scalar rewards, improving math reasoning.
problem Designing reinforcement learning algorithms for general LLM alignment.
method Introduces f-GRPO and f-HAL, estimating f-divergences between reward-aligned and unaligned distributions.
result Improves math reasoning RLVR tasks and mitigates reward hacking.
This work tightens generalization error bounds using Wasserstein distance.
problem Improving expected generalization error bounds in machine learning.
method Introduces bounds based on Wasserstein distance for various settings.
result New, tighter bounds based on relative entropy and other information measures.
Adapts VAEs for Bayesian inverse problems, quantifying uncertainty.
problem Bayesian inverse problems in scientific simulations.
method UQ-VAE: hybrid framework combining divergence-based variational inference and adjustable hyperparameters.
result Flexible, adaptive training of neural networks for posterior distribution.
Robust VAE improves model performance on corrupted data.
problem Outliers in training data degrade model performance.
method Applying robust statistics to VAEs using beta-divergence.
result Improved robustness to outliers in generated representations.
New algorithm enhances generative modeling for bounded domains.
problem Ad-hoc thresholding techniques for boundary enforcement in diffusion models.
method Reflected Schrödinger Bridge algorithm for entropy-regularized optimal transport.
result Generative modeling in diverse bounded domains with optimal transport properties.
We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defin…
Paper improves differential privacy analysis for machine learning.
problem Quantifying privacy leakage in noisy gradient descent.
method Shifted interpolation in f-differential privacy. result First exact privacy analysis for strongly convex optimization.
Study online RL with mismatched dynamics, achieving sublinear regret.
problem Exploration challenges in online RL with mismatched training and deployment dynamics.
method Introduce supremal visitation ratio, propose efficient algorithm with f-divergence. result Achieves sublinear regret in online RMDPs with optimal dependence on supremal visitation ratio and interaction episodes.
This paper extends the Risk Quadrangle framework for risk management and optimization.
problem Integrating risk management, optimization, and statistical estimation.
method Review and extension of the Risk Quadrangle framework with new quadrangles.
result New quadrangles offer novel approaches to risk-sensitive decision-making.
A new method learns robust policies from offline data with latent structures.
problem Conservative policies under unrealistic dynamics shifts.
method d-RRMDP framework with f-divergence regularization and R2PVI algorithm. result R2PVI learns robust policies with superior computational efficiency.
Study improves reinforcement learning for stable long-term performance.
problem Distributionally robust average-reward reinforcement learning for stable long-term performance.
method Proposes two algorithms to achieve near-optimal sample complexity.
result Achieves a sample complexity of O(∣S∣∣A∣tmix2ε−2) for estimating optimal policy and robust average reward.