Bayesian priors and penalties are equivalent in variational inference.
problem Understanding the relationship between Bayesian priors and penalties in variational inference.
method Characterizing the regularizers that can arise in variational inference and providing a systematic way to compute the prior corresponding to a given penalty.
result Equivalence between Bayesian priors and penalties in variational inference.
ETM identifies field-specific keywords in text classification.
problem Unsupervised text classification with field-specific keywords.
method Weighted Lasso penalty and pairwise Kullback-Leibler divergence penalty for topic separation.
result ETM improves topic coherence by 22% and 10% compared to LDA.
Paper proposes f-DPG for aligning language models with preferences.
problem Aligning language models with user preferences.
method Uses f-divergence to approximate target distributions and minimizes a forward KL from it using DPG.
result Jensen-Shannon divergence often outperforms forward KL divergence, leading to significant improvements.
AutoTransfer improves subject transfer learning for biosignal datasets.
problem Subject transfer learning for challenging biosignal datasets.
method Regularization framework with mutual information or divergence penalties.
result Improves subject transfer learning performance on EEG, EMG, and ECoG datasets.
GANs can learn without always minimizing a divergence, contrary to recent theories.
problem The need for GANs to minimize a divergence at every step is overly restrictive.
method Empirical counterexamples and gradient penalties.
result GANs can learn distributions without always minimizing a specific divergence.
Unified framework for reinforcement learning using entropic regularization.
problem Learning optimal controllers for unknown MDPs without diverging to dangerous regions.
method Entropic proximal policy optimization with α-divergences. result Unified perspective on actor-critic architectures and asymptotic analysis of solutions.
Random Fourier features model reconstructs wind fields from sparse measurements.
problem Reconstructing wind fields from limited data.
method Random Fourier features approximating velocity field with adaptive sampling.
result Random Fourier features model outperforms benchmarks.
Proposes HGP to improve survival analysis models.
problem Improving survival analysis models by addressing density function changes.
method Imposes constraints on local data points by regularizing the hazard function gradient.
result HGP enhances survival analysis model performance.
In this paper we study nonconvex penalization using Bernstein functions. Since the Bernstein function is concave and nonsmooth at the origin, it can induce a class of nonconvex functions for high-dimensional sparse estimation problems. We derive a threshold function based on the Bernstein penalty and give its mathemati…
DPEs use KL divergence to approximate BNNs, improving uncertainty estimates for active learning.
problem Improving uncertainty estimates in active learning for visual classification.
method Regularized ensemble approach with KL divergence penalty for variational inference.
result DPEs steadily improve active learning performance with increased annotation budgets.
A new method for non-negative matrix factorization using generalized dual divergence.
problem Non-negative matrix factorization for various noise structures.
method Theoretical framework based on generalized dual Kullback-Leibler divergence, with algorithms developed and proven convergence using Expectation-Maximization.
result Generalizes existing methods and provides an alternative for non-negative matrix factorizations.
Optimal transport with f-divergence regularization using generalized Sinkhorn algorithm.
problem Optimal transport with f-divergence regularization. method Generalized Sinkhorn algorithm for solving optimal transport problems with various f-divergences. result Strong duality holds, optimums are attained, and convergence to an optimal solution is guaranteed under certain conditions.
There has been a growing interest in mutual information measures due to their wide range of applications in Machine Learning and Computer Vision. In this paper, we present a generalized structured regression framework based on Shama-Mittal divergence, a relative entropy measure, which is introduced to the Machine Learn…
Study extends DRO with IPMs, linking robustness to regularization and GANs.
problem Addressing robustness of deep neural networks to adversarial attacks.
method Distributionally Robust Optimization (DRO) with Integral Probability Metrics (IPMs).
result DRO under any IPM corresponds to a family of regularization penalties.
This paper explores policy improvement using various f-divergences, enhancing stability in reinforcement learning.
problem Ensuring stability in reinforcement learning algorithms through policy improvement with trust regions.
method The paper considers a general class of f-divergences and derives policy update rules, including the KL divergence as a special case.
result The study reveals different policy updates and evaluations for various f-divergences, including Pearson χ2-divergence and KL divergence. New method improves sparse signal reconstruction using 1RSB-AMP.
problem Sparse signal reconstruction with improved accuracy.
method Developed 1RSB-AMP and 1RSB-SE for SCAD penalty minimization.
result 1RSB-AMP achieves improved reconstruction compared to RS-AMP.
Develops a statistical learning framework for personalized asset allocation.
problem Continuous-action decision-making with a large number of characteristics.
method Discretization approach with generalized penalties for penalized regression.
result Improves financial well-being with individualized optimal asset allocation.
The paper proposes a method to select tuning parameters for high-dimensional data analysis.
problem Selecting the tuning parameter in penalized likelihood methods for high-dimensional data.
method Optimizing the generalized information criterion (GIC) with an appropriate model complexity penalty.
result The proposed model complexity penalty should diverge at the rate of some power of log p.
2D-PT improves sampling in constrained optimization problems.
problem Sampling Boltzmann distributions with soft constraints.
method Two-dimensional extension of parallel tempering.
result 2D-PT achieves near-ideal mixing in constrained problems.
The paper introduces MU for NMF with β-divergences and disjoint constraints.
problem Nonnegative matrix factorization with constraints.
method Design multiplicative updates for NMF based on β-divergences with disjoint constraints. result Multiplicative updates satisfy constraints and decrease the objective function.
New algorithms improve robust estimation in contaminated Gaussian models.
problem Simultaneous estimation of location and variance matrix in contaminated Gaussian models.
method Tractable adversarial algorithms with spline discriminators for robust estimation.
result Achieve minimax optimal rates or near-optimal rates under Huber's contamination model.
LSGANs improve GANs by using least squares loss, leading to better image quality and stability.
problem Vanishing gradients in GANs during training.
method Introducing LSGANs with least squares loss for both discriminator and generator.
result LSGANs generate higher quality images and are more stable during training.
New RL algorithm learns good actions from offline data, reducing uncertainty and divergence.
problem Limited applicability of current RL algorithms in real-world settings due to high costs of exploration.
method Proposes an algorithm for batch RL using a fixed offline dataset, with penalties for policy and value constraints.
result Compared favorably to state-of-the-art methods on 32 continuous-action benchmarks.
Study EM and GD for clustering with penalties for misspecification and high dimensions.
problem Clustering with misspecification and high-dimensional data.
method Model-based Gaussian Mixture Models, EM algorithm, GD optimization with AD, penalized likelihood.
result GD outperforms EM on high-dimensional data but both have poor cluster interpretation.
Generative models often misrepresent class frequencies; this paper calibrates them.
problem Miscalibration of class frequencies in generative models.
method Formulated as constrained optimization, using surrogate objectives to approximate constraints.
result Significant reduction in calibration error across various models and applications.
Unified framework for smooth convex regularization of optimal transport problems.
problem Optimizing transport plans with regularization.
method Unified framework based on matrix nearness problems with Bregman divergences.
result Regularized optimal transport equivalent to matrix nearness problem.
The paper calibrates robust optimization models to reduce sensitivity to model errors.
problem Reducing sensitivity of expected reward to model errors in empirical optimization.
method Develops a theory for data-driven calibration of robustness parameter δ using resampling methods.
result Substantial variance reduction is possible at little cost if δ is properly calibrated.
New NMF model for better audio source separation.
problem Blind audio source separation from single-microphone recordings.
method Minimum-volume beta-divergence NMF with multiplicative updates.
result Model identifies and interprets sources more effectively.
A single algebraic identity unifies information-theoretic variational results.
problem Deriving and generalizing classical information-theoretic variational results
method Proving a single algebraic mixed coincidence identity
result Unified derivation of classical cornerstones of information theory
Faster algorithm for sampling logconcave densities in high dimensions.
problem Cubic barrier in sampling logconcave densities from a cold start.
method Two key ingredients: weaker distance sampling and refined log-Sobolev inequality.
result First sub-cubic sampling algorithms for isotropic position.
NMF with specific constraints is equivalent to LDA.
problem Dimensionality reduction of non-negative data.
method NMF with ℓ1 normalization constraints and Dirichlet prior. result NMF with these constraints is equivalent to LDA.
VED framework learns low-dimensional latent representations of physical systems.
problem Learning latent representations of complex physical systems.
method Variational Encoder-Decoder (VED) framework with KL divergence and covariance regularization.
result VED achieves lower-dimensional latent representations with improved feature disentanglement.
Momentum SGD fails to track nonstationary optima due to drift amplification.
problem Tracking nonstationary optima in stochastic optimization.
method Theoretical analysis of SGD and momentum variants under strong convexity and smoothness.
result Momentum incurs a drift-amplification penalty that diverges as the momentum parameter approaches 1, leading to systematic lag.
SAIL-RevKL improves SAIL's convergence by regularizing the objective function.
problem Convergence of self-improving online LLM alignment algorithms.
method Proposed SAIL-RevKL, a regularized objective function to improve optimization landscape.
result Proved SAIL-RevKL satisfies the Polyak-Lojasiewicz (PL) condition with near-linear sample complexity.
Binacox detects multiple cut-points in high-dimensional Cox models for genetic cancer data.
problem Detecting multiple cut-points in high-dimensional Cox models with many continuous features.
method Combines one-hot encoding with binarsity penalty for feature selection and regularization.
result Significantly outperforms state-of-the-art survival models in terms of C-index and computational speed.
New methods stabilize EEG classification performance across subjects.
problem Large performance drop in EEG classification models on unseen subjects.
method Regularization techniques using divergence estimation.
result Significant increase in balanced accuracy on test subjects.
The paper explores nonconvex penalties for deep learning regularization.
problem Overfitting in deep learning neural networks.
method Examines and evaluates nonconvex penalties for DNN regularization.
result Nonconvex penalties, under certain conditions, can perform well in DNNs.
Bayesian learning is often hampered by large computational expense. As a powerful generalization of popular belief propagation, expectation propagation (EP) efficiently approximates the exact Bayesian computation. Nevertheless, EP can be sensitive to outliers and suffer from divergence for difficult cases. To address t…
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
We propose a new method of learning a sparse nonnegative-definite target matrix. Our primary example of the target matrix is the inverse of a population covariance or correlation matrix. The algorithm first estimates each column of the target matrix by the scaled Lasso and then adjusts the matrix estimator to be symmet…
Paper proposes algorithms for robust 1-bit compressive sensing with nonconvex penalties.
problem Recovering sparse signals from one-bit measurements.
method Develops algorithms based on convex and nonconvex penalties, providing analytical solutions.
result Analytical solutions for several nonconvex penalties are found, making the recovery process faster and more efficient.
This work improves GAN stability with theoretical conditions.
problem GANs exhibit unstable behavior during training.
method Developed a theoretical framework and conditions for GAN stability.
result Constructs a GAN that fulfills stability conditions.
Gradient descent can optimize penalty parameters for non-smooth penalties.
problem Optimizing penalty parameters for non-smooth penalties in regression problems.
method Modified gradient descent algorithm for non-smooth penalty functions.
result Decreased generalization error with tuned penalty parameters.
Gradient penalty improves GAN performance by inducing a large-margin classifier.
problem Improving GAN performance and addressing vanishing gradients.
method A unifying framework of expected margin maximization, showing gradient penalties induce large-margin classifiers.
result Gradient penalties reduce vanishing gradients and produce better generated outputs.
Study examines insider trading with penalties, finding optimal penalties increase quickly for small orders.
problem Analyzing the impact of penalties on insider trading behavior and market efficiency.
method Formal economic model with penalty functions, existence and uniqueness theorems, and optimization.
result Optimal penalties increase quickly for small orders, signaling extreme events and incorporating information into prices.
The alternating direction method of multipliers (ADMM) has been successfully applied to solve structured convex optimization problems due to its superior practical performance. The convergence properties of the 2-block ADMM have been studied extensively in the literature. Specifically, it has been proven that the 2-blo…
A new approach to risk-sensitive reinforcement learning tackles computational challenges.
problem Computational challenges in estimating risk-sensitive policies for MDPs with finite state and action spaces.
method Proposes a new risk measure called 'caution' and uses a stochastic primal-dual method with KL divergence.
result Demonstrates improved reliability in reward accumulation without additional computational costs.
The paper studies robust risk measures with linear penalties under uncertain distributions.
problem Risk measurement under distributional uncertainty.
method Robust distortion risk measures with linear penalty function under distributional constraints.
result Explicit characterization of optimal quantile distribution and value function.