Paper connects contrastive learning to MI maximization and establishes robust methods for nonlinear ICA and subspace estimation.
problem Understanding and improving unsupervised representation learning and density ratio estimation.
method The paper connects contrastive learning to MI maximization, establishes new recovery conditions for nonlinear ICA, and proposes a practical outlier-robust method for nonlinear subspace estimation.
result The proposed methods can be seen as maximizing MI, performing nonlinear ICA, or estimating nonlinear subspaces, and are robust to outliers.
Maximizes robustness in Bayesian experimental design under model uncertainty.
problem Brittleness of Bayesian experimental design under model misspecification.
method Formulates as a max--min game, uses Sibson's α-MI, and adopts PAC-Bayes framework.
result Establishes robust belief update and conditional information gain measure.
Bayesian nonparametric framework improves robustness of MI estimation.
problem Challenges in exact MI computation in high dimensions with intractable likelihoods.
method Uses a Dirichlet process posterior to regularize MI loss, reducing sensitivity to fluctuations and outliers.
result Significant improvements in convergence over EDF-based methods, enhancing robustness and accuracy.
Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.
problem Generalization bounds for machine learning models with VC classes.
method Unified derivation of conditional PAC-Bayesian and mutual information bounds, including MAC-Bayesian bounds.
result Nontrivial bounds for general VC classes and faster rates for specific conditions.
MIRRAMS framework tackles robust tabular learning under unseen missingness shifts.
problem Challenges in achieving robust predictive performance due to shifts in missingness distribution between training and test inputs.
method Introduces MI robustness conditions and MIRRAMS framework to enforce these conditions without specific missingness assumptions.
result Consistently outperforms existing state-of-the-art baselines and maintains stable performance under diverse missingness conditions.
This research improves EEG-based MI-BCI systems to be more resilient to emotional arousal.
problem Lack of robustness in EEG-based MI-BCI systems due to emotional arousal.
method Subjects were exposed to VR environments to induce high and low arousal states. Machine learning models were trained on proxy subjects instead of arousal states. MI models were trained for each subject.
result MI-BCI systems are made more resilient to emotional perturbations.
New bounds derived using conditional f-information for machine learning models.
problem Improving generalization bounds in machine learning.
method Introducing novel information-theoretic generalization bounds via conditional f-information. result Derives generalization bounds applicable to both bounded and unbounded loss functions.
The aim of this paper is to compare the performances of the optimal strategy under parameters mis-specification and of a technical analysis trading strategy. The setting we consider is that of a stochastic asset price model where the trend follows an unobservable Ornstein-Uhlenbeck process. For both strategies, we prov…
Study examines financial market structure changes during the COVID-19 crash using a novel MI approach.
problem Analyzing nonlinear dependencies among major stocks during market crashes.
method Conditional p-threshold mutual information (MI) and Minimum Spanning Tree (MST) framework.
result Financial networks become more integrated during crashes, with increased periphery vulnerability.
A new framework for robust and coherent counterfactual transports.
problem Estimating joint distributions over counterfactual outcomes in personalized decision-making and treatment risk assessment.
method Counterfactual cocycles that use algebraic structure to provide coherence and identifiability guarantees, bridging the gap between bijective SCMs and OT methods.
result Counterfactual cocycles provide state-of-the-art performance and noise-robustness across synthetic benchmarks and a real-world study.
Mutual Information (MI) is often used for feature selection when developing classifier models. Estimating the MI for a subset of features is often intractable. We demonstrate, that under the assumptions of conditional independence, MI between a subset of features can be expressed as the Conditional Mutual Information (…
We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large samp…
Identity-link IRT improves TVD-MI scores without curvature violations.
problem Preserving additivity in TVD-MI scores for efficient LLM evaluation.
method Derives clipped-linear model from Gini entropy maximization, using identity link.
result Identity-link yields lower curvature violations (median curl 0.080-0.150) compared to probit/logit.
Improves industry classification for diversified companies.
problem Traditional industry classification struggles with multi-sector conglomerates.
method Bayesian Non-Parametrics, Markov Updating, and hierarchical modeling.
result MIS-2 provides a measurable improvement over GICS in predicting future correlations.
It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples. Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial per…
Label aggregation makes learning robust to noisy labels.
problem Learning from noisy labels.
method Label aggregation and risk consistency.
result Aggregated labels lead to stronger consistency guarantees.
TMLE improves causal effect estimation in missing data scenarios with various positivity violations.
problem Estimating causal effects in studies with missing data and positivity violations.
method Targeted Maximum Likelihood Estimation (TMLE) with various missing data methods.
result Complete cases with TMLE incorporating an outcome-missingness model exhibit lower bias and greater robustness against positivity violations.
Proposes MIGE for accurate MI gradient estimation in high-dimensional settings.
problem Intractability of MI in continuous and high-dimensional settings.
method Score estimation of implicit distributions for gradient estimation of MI.
result MIGE provides tight and smooth gradient estimation of MI in high-dimensional settings.
A scalable method for BED with implicit models using approximate gradients.
problem Efficiently estimating posterior distribution and maximizing MI for implicit models.
method Stochastic approximate gradient ascent with smoothed variational MI estimator.
result Significantly improves scalability of BED in high-dimensional problems.
SAEs struggle with feature consistency across runs, hindering MI reliability.
problem Inconsistency of learned SAE features across different training runs.
method Propose using the Pairwise Dictionary Mean Correlation Coefficient (PW-MCC) to measure feature consistency.
result High levels of feature consistency (0.80 for TopK SAEs on LLM activations) are achievable with appropriate architectural choices.
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE ex…
Stock price movement reveals complex interdependencies that are simplified through linear correlation.
problem Exploring the spectral dynamics of the Indonesian capital market using structural network representations.
method Combining three dependency estimators (Pearson, MI adaptive binning, and MI-kNN) with two graph filtering schemes (MST and PMFG) and four community decoders.
result MI adaptive binning is shown to be more proportional than kNN for detecting residual information.
New algorithm learns Koopman operator online, with complexity control and convergence guarantees.
problem Online learning of Koopman operator for general nonlinear systems.
method Sparse online learning via stochastic approximation, RKHS action, CME operator.
result Provably convergent algorithm with finite-time guarantees in mis-specified setting.
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
New MI bounds improve estimation in deep generative models.
problem Estimating mutual information without density information is intractable.
method Importance sampling, Annealed Importance Sampling, Generalized IWAE, MINE-AIS.
result Improved bounds for estimating mutual information in deep models.
Unified framework connects NCE, MIS, RLR, and bridge sampling for EBMs.
problem Challenges in parameter estimation for intractable likelihood EBMs.
method Unified framework connecting NCE, RLR, MIS, and bridge sampling.
result Unified perspective clarifies relationships among existing methods.
New method estimates mutual information using normalizing flows.
problem Mutual information estimation in high-dimensional data.
method Normalizing flows to map data to target distributions with known MI.
result Theoretical guarantees and practical advantages demonstrated.
Proposes CLUB for reliable MI minimization in high dimensions.
problem Estimating and minimizing mutual information in high-dimensional spaces.
method Contrastive Log-ratio Upper Bound (CLUB) for MI minimization.
result CLUB provides reliable estimation of mutual information.
New methods estimate point-wise dependency from neural MI models.
problem Estimating point-wise dependency between different events.
method Developed two methods: Probabilistic Classifier and Density-Ratio Fitting.
result Demonstrated effectiveness in MI estimation, self-supervised representation learning, and cross-modal retrieval.
A typical goal of supervised dimension reduction is to find a low-dimensional subspace of the input space such that the projected input variables preserve maximal information about the output variables. The dependence maximization approach solves the supervised dimension reduction problem through maximizing a statistic…
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
The asymptotic distribution of the Markowitz portfolio is derived, for the general case (assuming fourth moments of returns exist), and for the case of multivariate normal returns. The derivation allows for inference which is robust to heteroskedasticity and autocorrelation of moments up to order four. As a side effect…
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Study robustness of global feature effect explanations in machine learning models.
problem Vulnerability of global feature effect explanations to data and model perturbations.
method Theoretical bounds and experimental evaluation of partial dependence plots and accumulated local effects.
result Quantifies the gap between best and worst-case scenarios of misinterpreting machine learning predictions globally.
InfoBridge uses diffusion bridges to estimate mutual information accurately.
problem Estimating mutual information between random variables.
method Formulated mutual information estimation as a domain transfer problem using diffusion bridge models.
result Demonstrated unbiased estimator for various data types.
Develops a SAS approach for high-dimensional risk prediction using unlabeled data.
problem Challenges in risk modeling with EHR data due to lack of direct disease outcomes and high dimensionality.
method Surrogate Assisted Semi-supervised Learning (SAS) approach leveraging unlabeled and labeled data.
result Valid inference for predicted risk even when underlying model is dense and mis-specified.
A method for finding most influential sets reduces a complex problem to a sequence of simpler top-k problems.
problem Identifying most influential subsets in complex models.
method Reduces the problem to a sequence of top-k problems using Dinkelbach's method. result The method returns a globally optimal set for the univariate ratio objective, including partial linear models.
New method improves self-supervised representation learning using probabilistic modeling and Monte Carlo integration.
problem Improving self-supervised representation learning for multimodal data.
method Discriminative probabilistic modeling with multiple importance sampling (MIS) for robust Monte Carlo integration.
result Proposes a novel non-parametric method for approximating conditional probability densities through convex optimization.
Paper refines InfoNCE for accurate mutual information estimation.
problem Indirect connection of InfoNCE to mutual information estimation.
method Introduces InfoNCE-anchor with an auxiliary anchor class for consistent density ratio estimation.
result InfoNCE-anchor yields a plug-in MI estimator with significantly reduced bias.
A new contrastive MI estimator improves efficiency and tightness.
problem Efficient and tight mutual information estimation.
method Contrastive Fenchel-Legendre optimization.
result The FLO estimator is tight and converges under stochastic gradient descent.
LMI approximates mutual information in high dimensions using learned low-dimensional representations.
problem Estimating mutual information between high-dimensional variables is challenging due to sample size limitations.
method Developed a method called latent MI (LMI) approximation that applies a nonparametric MI estimator to low-dimensional representations learned by a simple model architecture.
result LMI can approximate MI well for variables with >10^3 dimensions if their dependence structure has low intrinsic dimensionality.
Causal discovery is a fundamental problem in statistics and has wide applications in different fields. Transfer Entropy (TE) is a important notion defined for measuring causality, which is essentially conditional Mutual Information (MI). Copula Entropy (CE) is a theory on measurement of statistical independence and is …
Greedy MI maximization method outperforms existing approaches in nonlinear models.
problem Maximizing mutual information in nonlinear models with non-Gaussian noise.
method Greedy approaches based on log-Sobolev inequalities for computationally inexpensive MI lower bounds.
result Proposed method outperforms random selection and Gaussian approximations.
Improves BED scalability for implicit models.
problem Designing experiments for implicit models with intractable data distributions.
method Hybrid gradient approach combining variational MI estimator, ES, and SGA.
result Significantly improves scalability of BED for implicit models.
We stabilize MI-based losses by adding a regularization term, improving their performance and stability.
problem Instability of MI-based losses in machine learning.
method Added a novel regularization term to stabilize MI-based losses.
result Regularization stabilizes training and improves the performance of MI-based losses.
We propose an active learning method for discovering low-dimensional structure in high-dimensional Gaussian process (GP) tasks. Such problems are increasingly frequent and important, but have hitherto presented severe practical difficulties. We further introduce a novel technique for approximately marginalizing GP hype…
The market impact (MI) of Volume Weighted Average Price (VWAP) orders is a convex function of a trading rate, but most empirical estimates of transaction cost are concave functions. How is this possible? We show that isochronic (constant trading time) MI is slightly convex, and isochoric (constant trading volume) MI is…
Study quantized models' privacy against membership inference attacks.
problem Privacy risk in quantized machine learning models.
method Proposed a new MIS indicator for post-training quantization procedures, minimizing empirical loss.
result Demonstrated effectiveness of new MIS indicator in assessing and ranking privacy risk.