E2M optimizes tensor density estimation by relaxing α-divergence to KL-divergence.
problem Analytical challenges in traditional α-divergence optimization for tensor-based density estimation. method E2M algorithm: relaxes optimization to KL-divergence, then applies tensor many-body approximation. result Flexible modeling of various low-rank structures and their mixtures.
MIM learns joint distributions with mutual information and low divergence.
problem Learning joint distributions over observations and latent variables.
method Probabilistic auto-encoder with three design principles: low divergence, high mutual information, and low marginal entropy.
result MIM learns representations with high mutual information, consistent encoding and decoding distributions, effective latent clustering, and comparable data log likelihood to VAE.
Fairness-aware learning is a novel framework for classification tasks. Like regular empirical risk minimization (ERM), it aims to learn a classifier with a low error rate, and at the same time, for the predictions of the classifier to be independent of sensitive features, such as gender, religion, race, and ethnicity. …
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving Gaussian Process regression efficiency with low-rank approximations.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation.
result Bounds on the divergence and error between exact and approximate GP models.
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving the efficiency of Gaussian Process regression while maintaining accuracy.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation, and bounds the divergence and error between exact and approximate models.
result Theoretical bounds on the divergence and error between exact and approximate Gaussian Process models are provided.
New theory explains GAN's high quality but low diversity.
problem Lack of theoretical justification for non-saturating GAN training.
method Showed non-saturating GAN training approximately minimizes a specific f-divergence.
result Non-saturating GAN training minimizes a particular f-divergence.
We compare two models of corporate default by calculating the Jeffreys-Kullback-Leibler divergence between their predicted default probabilities when asset correlations are either high or low. Our main results show that the divergence between the two models increases in highly correlated, volatile, and large markets, b…
New method for factor analysis using nuclear and ℓ0 norms.
problem Finding a low-rank plus sparse decomposition from noisy covariance matrix.
method Formulated an optimization problem with nuclear norm, ℓ0 norm, and KL divergence. Used alternating minimization algorithm. result Algorithm effectively decomposes covariance matrices in synthetic and real datasets.
New method improves imitation learning from expert observations.
problem Challenges in imitation learning from observation setting.
method Reparameterized Variational Divergence Minimization.
result Our method outperforms baseline approaches in low-dimensional tasks.
This paper introduces a new method to train normalizing flows using precision-recall divergences.
problem Training generative models with mode dropping and low-quality samples.
method Introduces PR-divergences and proposes a novel generative model to minimize precision-recall trade-offs.
result Normalizing flows can be trained to achieve specific precision-recall trade-offs using PR-divergences.
New divergences improve estimation and GAN training performance.
problem Improving estimation and training in machine learning models.
method Function-space regularized Rényi divergences.
result New divergences reduce variance and improve training performance.
New research shows DDPM can adapt to data's intrinsic low dimensionality efficiently.
problem Theoretical inefficiency of DDPM in high-dimensional data.
method Investigates how DDPM can exploit intrinsic low dimensionality of data.
result Proves DDPM's iteration complexity scales nearly linearly with intrinsic dimension k. The paper presents methods to improve uncertainty calibration in Bayesian Neural Networks.
problem Uncalibrated Bayesian Neural Networks often lead to overconfidence.
method The paper uses alpha-divergences from Information Geometry for calibration.
result Calibration using alpha-divergences provides better uncertainty estimates and is more efficient.
Two methods factor out prior knowledge from low-dimensional embeddings.
problem Visualizing data without considering background knowledge.
method JEDI for tSNE and CONFETTI for any embedding.
result Embeddings reveal meaningful structure hidden by prior knowledge.
Black box variational inference (BBVI) with reparameterization gradients triggered the exploration of divergence measures other than the Kullback-Leibler (KL) divergence, such as alpha divergences. In this paper, we view BBVI with generalized divergences as a form of estimating the marginal likelihood via biased import…
Method identifies low-dimensional structure in high-dimensional probability measures.
problem Identifying low-dimensional structure in high-dimensional probability measures.
method Extends prior work on minimizing majorizations of the Kullback-Leibler divergence to identify optimal approximations within a specific class of measures.
result Connection between dimensional logarithmic Sobolev inequality and approximations with the ansatz.
We lay the foundations for a theory of divergence-measure fields in noncommutative stratified nilpotent Lie groups. Such vector fields form a new family of function spaces, which generalize in a sense the BV fields. They provide the most general setting to establish Gauss-Green formulas for vector fields of low regul…
This work bounds classification error in machine learning for low Bayes error conditions.
problem Understanding the error mismatch between Bayes error and model-based classification error.
method Applying classification error bounds to study the relationship with Kullback-Leibler divergence and proposing a linear approximation for low Bayes error conditions.
result A linear approximation of the classification error bound for low Bayes error conditions is proposed.
Deep networks can approximate high-dimensional distributions from low-dimensional ones.
problem Approximating high-dimensional distributions from low-dimensional ones.
method Proved neural networks can transform low-dimensional distributions to high-dimensional ones with arbitrary closeness measured by Wasserstein distances and maximum mean discrepancy.
result Upper bounds of the approximation error are obtained in terms of the width and depth of neural network.
The paper introduces a new divergence measure for variational autoencoders to improve reconstruction and generation.
problem Balancing reconstruction and generalizability in latent space of variational autoencoders.
method Presented a regularisation mechanism based on skew-geometric Jensen-Shannon divergence.
result The skew-geometric Jensen-Shannon divergence leads to better reconstruction and generation in variational autoencoders.
Improves BBVI for high-dimensional Gaussian approximations by using low-rank approximations.
problem Scalability issues with BBVI for high-dimensional multivariate Gaussian approximations.
method Extends BaM framework to handle full covariance matrices by integrating patch step for low-rank parameterization.
result Shows improved efficiency and scalability on synthetic and real-world high-dimensional inference problems.
Robust VAE detects anomalies in corrupted data.
problem Detect anomalies in data with high corruption.
method Robust Variational Autoencoder (VAE) with four modifications.
result Establishes robustness to outliers and suitability to low-rank modeling.
Paper shows robust generative learning with minimal assumptions on target distributions.
problem Learning generative models with minimal assumptions on target distributions.
method Lipschitz-regularized α-divergences with minimal assumptions. result Stable learning across various target distributions with minimal assumptions.
Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.
problem Training Large Language Models (LLMs) with Reinforcement Learning (RL) causes numerical divergence between inference and training.
method Dynamic Vocabulary Pruning (DVP) constrains the RL objective to a safe vocabulary that excludes low-probability tokens.
result DVP stabilizes training by reducing systematic bias introduced by the extreme tail of the token distribution.
We introduce a temperature into the exponential function and replace the softmax output layer of neural nets by a high temperature generalization. Similarly, the logarithm in the log loss we use for training is replaced by a low temperature logarithm. By tuning the two temperatures we create loss functions that are non…
A new method for decomposing non-negative tensors using energy-based modeling.
problem Challenges in traditional tensor decomposition methods, especially global optimization and rank selection.
method Energy-based modeling of tensors, considering interactions between modes for global optimization.
result Demonstrates effectiveness in tensor completion and approximation, revealing a relationship between many-body and low-rank approximations.
MsIGN tackles high-dimensional Bayesian inference using multiscale structure.
problem High-dimensional Bayesian inference challenges due to the curse of dimensionality.
method MsIGN generates samples from coarse to fine scale, minimizing Jeffreys divergence.
result MsIGN outperforms previous approaches in posterior approximation and mode capture.
Mirror descent algorithm recovers low-rank matrices in matrix sensing.
problem Matrix sensing with low-rank matrices under certain conditions.
method Discrete-time mirror descent applied to empirical risk with Bregman divergence analysis.
result Mirror descent converges to a matrix minimizing a specific nuclear norm-related quantity.
The paper solves portfolio selection using Rényi divergence and optimization.
problem Single-period portfolio selection under CRRA utility.
method Information-theoretic lens, Rényi divergence, Rényi entropy, Blahut-Arimoto-style alternating optimization.
result CRRA portfolio selection is equivalent to a Rényi information-projection problem.
p-SNE embeds Poisson count data into low dimensions preserving structure.
problem Embedding high-dimensional sparse Poisson data into a low-dimensional space.
method p-SNE (Poisson Stochastic Neighbor Embedding) using KL divergence and Hellinger distance.
result p-SNE recovers meaningful structure in real-world count datasets.
Improved VAE for heavy-tailed data using Student's t-distributions.
problem Over-regularization in VAEs with Gaussian priors.
method Proposed t3VAE framework with Student's t-distributions for prior, encoder, and decoder. result Significantly outperforms other models on heavy-tailed datasets.
New method uses SoS densities and α-divergences for efficient sequential transport maps.
problem Efficiently generating samples from approximated densities.
method Sequential transport maps using Sum-of-Squares (SoS) densities and α-divergences.
result Convex optimization problems with efficient semidefinite programming solutions.
The paper develops new algorithms for KL-divergence NMF, proving convergence and performance.
problem Improving NMF for nonnegative data with KL divergence.
method Collect and analyze properties of KL objective function, propose and test new algorithms.
result Guaranteed non-increasing objective function for one proposed algorithm, global convergence.
Deep neural networks can estimate Q-values efficiently on low-dimensional state-action spaces.
problem Estimating the performance of a reinforcement learning policy using data from a different policy.
method Deep fitted Q-evaluation method leveraging manifold structure and convolutional neural networks.
result Sharp error bound for fitted Q-evaluation depends on intrinsic dimension and function space mismatch.
A semi-supervised framework using stochastic interpolation and latent representations.
problem Challenges in conditional generative modeling with scarce labeled data.
method Combines conditional stochastic interpolation with low-dimensional latent representations.
result Significantly improves sample complexity and achieves faster convergence rate.
New study shows Gaussian samplers struggle with heavy-tailed targets, while stable samplers excel.
problem The difficulty of sampling from heavy-tailed distributions using Gaussian versus stable oracles.
method Comparison of Gaussian and stable oracles for proximal samplers.
result Gaussian samplers have a fundamental barrier for high-accuracy guarantees in heavy-tailed sampling, while stable samplers excel.
Preconditioned NFs speed up sampling from complex posterior distributions in inverse problems.
problem Sampling from posterior distributions of inverse problems with expensive forward operators.
method Preconditioning a conditional normalizing flow (NF) to speed up training.
result Significant speed-ups achieved compared to training NFs from scratch.
Simulates risk-neutral markets using neural spline flows.
problem Creating realistic risk-neutral market simulations.
method Developed a low-dimensional martingale representation and used neural spline flows for sampling.
result The calibrated simulator is closest to historical data with respect to Kullback-Leibler divergence.
Quantum models face barren plateaus, but specific losses can be trainable.
problem Barren plateaus and loss concentration in quantum generative models.
method Investigated explicit and implicit losses, and their interplay.
result Explicit losses lead to new barren plateaus, while implicit losses can be trainable.
Generative flows learn distributions on low-dimensional manifolds robustly via Wasserstein proximals.
problem Learning distributions supported on low-dimensional manifolds robustly.
method Combining Wasserstein-1 and Wasserstein-2 proximal operators to formulate well-posed continuous-time generative flows.
result The combination of Wasserstein-1 and Wasserstein-2 proximals ensures the well-posedness of generative flows, leading to unique and robust learning.
The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the stan…
CNFs learn on manifolds using PPD, improving likelihood and sample quality.
problem Training CNFs on manifolds efficiently and accurately.
method Minimizing PPD, a novel divergence, to train CNFs on manifolds.
result CNFs trained with PPD achieve state-of-the-art results on manifold benchmarks.
The paper develops a method to model high-dimensional data with many variables and weak signals.
problem Modeling high-dimensional dependent data with many explanatory variables and low signal-to-noise ratio.
method Penalized regression for high-dimensional data, factor modeling of residuals, high-dimensional white noise testing, projected Principal Component Analysis.
result Established asymptotic properties of the proposed method for high-dimensional data.
In the covariate shift learning scenario, the training and test covariate distributions differ, so that a predictor's average loss over the training and test distributions also differ. In this work, we explore the potential of extreme dimension reduction, i.e. to very low dimensions, in improving the performance of imp…
A new method speeds up computation of Sinkhorn divergences to linear time.
problem Expensive computation of Sinkhorn divergences for comparing probability distributions.
method Using positive features to approximate ground costs, reducing computation time to linear.
result Sinkhorn divergences can be computed in linear time, scaling as O(nr).
Filling invariants are measurements of a metric space describing the behaviour of isoperimetric inequalities. In this article we examine filling functions and higher divergence functions. We prove for a class of stratified nilpotent Lie groups that in the low dimensions the filling functions grow as fast as the ones of…
Bayesian coresets improved with random sampling and quasi-Newton optimization.
problem Efficiently approximate Bayesian posterior distributions for computationally expensive inference.
method Randomly select a subset of data points, then optimize weights using quasi-Newton method.
result First algorithm with high-probability KL divergence bound on coreset quality.
This paper introduces a neural sampler for scalable sampling from complex distributions.
problem Efficiently sampling from high-dimensional un-normalized distributions.
method Neural implicit sampler trained with KL and Fisher divergence methods.
result The neural sampler generates large batches of samples with low computational costs.