In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex lo…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “mutual regularization”
Proposes a framework to maximize mutual information in VAE models for better latent code representation.
problem Lack of explicit measurement of the quality of learned representations in VAE models.
method Variational Mutual Information Maximization Framework for VAE.
result Maximizes mutual information between latent codes and observations, improving latent code representation.
We stabilize MI-based losses by adding a regularization term, improving their performance and stability.
problem Instability of MI-based losses in machine learning.
method Added a novel regularization term to stabilize MI-based losses.
result Regularization stabilizes training and improves the performance of MI-based losses.
Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it ta…
Proposes a framework to improve VAE latent codes using mutual information.
problem Lack of explicit measure for VAE latent variable quality.
method Variational Mutual Information Maximization Framework.
result Improves relationships between latent codes and observations.
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the ent…
We estimate Radon-Nikodym derivatives using regularization in reproducing kernel Hilbert spaces.
problem Estimating Radon-Nikodym derivatives in various applications.
method General regularization scheme in reproducing kernel Hilbert spaces.
result High order accuracy in reconstructing Radon-Nikodym derivatives at any point.
We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn differe…
Mutual information bounds generalization error in variational classifiers.
problem Controlling overfitting in variational classifiers.
method Derive bounds on generalization error using mutual information.
result Mutual information bounds the generalization error in variational classifiers.
New insights explain why -VAEs fail at disentanglement.
problem Disentanglement performance of -VAEs peaks at intermediate and collapses as regularization increases.
method Formalized information-theoretic mechanism, introduced -VAE to stabilize disentanglement.
result Strong regularization pressure leads to mutual information collapse in -VAEs.
Enhances financial data signal-to-noise ratio using auto-encoders and mutual regularization.
problem Improving signal-to-noise ratio in financial data.
method Combining target and context variables, using auto-encoders with mutual regularization to learn common ground.
result Discover new regularities in financial time-series data.
Proposes mutual information for regression without prior knowledge.
problem Regression tasks without prior model knowledge.
method Mutual information learning formulation, SGD convergence analysis.
result High dimensionality can be beneficial with a threshold.
Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations. However, recent studies demonstrate that, when equipped with expressive generative distributions (aka. decoders), VAE suff…
We introduce several methods to define the self-inductance of a single loop as the regularization of divergent integrals which we obtain by applying Neumann (or Weber) formula for the mutual inductance of a pair of loops to the case when two loops are identical.
Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcs.LG
Our work proves CSF can recover ground-truth features in RL, improving understanding of feature learning.
problem Understanding the role of representation and mutual information in reinforcement learning.
method Investigates Contrastive Successor Features (CSF) method for identifiable representation learning in reinforcement learning.
result Proves CSF can recover ground-truth features up to a linear transformation.
Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.
problem Uncertainty in capturing the exceptional performance of learning algorithms using existing information-theoretic generalization bounds.
method Examined the relationship between mutual information and generalization in stochastic convex optimization.
result Mutual information is necessary for true risk minimization in stochastic convex optimization, indicating existing bounds fall short.
AutoTransfer improves subject transfer learning for biosignal datasets.
problem Subject transfer learning for challenging biosignal datasets.
method Regularization framework with mutual information or divergence penalties.
result Improves subject transfer learning performance on EEG, EMG, and ECoG datasets.
This work improves generalisation bounds using chaining and information theory.
problem Improving generalisation bounds for supervised learning algorithms.
method Developed a theoretical framework linking generalisation bounds to their chained counterparts, derived new bounds using Wasserstein distance.
result Chained generalisation bounds can be tighter than standard bounds, especially for concentrated hypothesis distributions.
Standard accuracy metrics indicate that modern reading comprehension systems have achieved strong performance in many question answering datasets. However, the extent these systems truly understand language remains unknown, and existing systems are not good at distinguishing distractor sentences, which look related but…
Variational autoencoders optimize an objective that combines a reconstruction loss (the distortion) and a KL term (the rate). The rate is an upper bound on the mutual information, which is often interpreted as a regularizer that controls the degree of compression. We here examine whether inclusion of the rate also acts…
This paper introduces GEMINI, a new metric for unsupervised neural network training that avoids the need for regularizations.
problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised the mutual information by changing its core distance, introducing the Generalised Mutual Information (GEMINI).
result Some GEMINIs do not require regularizations when training and can automatically select the number of clusters.
Learning to cooperate with friends and compete with foes is a key component of multi-agent reinforcement learning. Typically to do so, one requires access to either a model of or interaction with the other agent(s). Here we show how to learn effective strategies for cooperation and competition in an asymmetric informat…
Collaborative filtering (CF) has been successfully employed by many modern recommender systems. Conventional CF-based methods use the user-item interaction data as the sole information source to recommend items to users. However, CF-based methods are known for suffering from cold start problems and data sparsity proble…
Study entropic regularization of Gaussian measures and processes on Hilbert space.
problem Regularizing 2-Wasserstein distance for infinite-dimensional Gaussian measures and processes.
method Minimum Mutual Information property, closed form formulas, Fréchet differentiability, Sinkhorn barycenter equation.
result Entropic 2-Wasserstein distance and Sinkhorn divergence are Fréchet differentiable in Hilbert space.
Due to the advantage of achieving a better performance under weak regularization, elastic net has attracted wide attention in statistics, machine learning, bioinformatics, and other fields. In particular, a variation of the elastic net, adaptive elastic net (AEN), integrates the adaptive grouping effect. In this paper,…
Latent Variable Models (LVMs) are a large family of machine learning models providing a principled and effective way to extract underlying patterns, structure and knowledge from observed data. Due to the dramatic growth of volume and complexity of data, several new challenges have emerged and cannot be effectively addr…
Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representationscs.LG
A novel method integrates feature and topology views for unsupervised graph representation learning.
problem Lack of mutual information across feature and topology views in graph representation learning.
method Proposes a multi-view representation learning module and a common representation learning module using mutual information maximization and reconstruction loss minimization.
result Demonstrates effectiveness in integrating feature and topology views, achieving comparable or better performance than supervised methods.
New bounds improve neural network generalization through slicing.
problem Difficulty in evaluating mutual information in high dimensions for neural networks.
method Slicing the parameter space and using disintegrated mutual information and k-sliced mutual information.
result Slicing improves generalization and offers significant computational and statistical advantages.
Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…
In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter re…
We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit an…
Graph Information Bottleneck (GIB) optimizes graph representations for robustness against adversarial attacks.
problem Challenges in learning graph representations due to structure and feature information.
method GIB is an information-theoretic principle that balances expressiveness and robustness by maximizing mutual information between representation and target, while constraining mutual information with input data.
result GIB-based models are more robust to adversarial attacks, achieving up to 31% improvement.
Label-Based Diversity Measure Among Hidden Units of Deep Neural Networks: A Regularization Methodcs.LG
New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.
problem Overfitting in deep neural networks.
method Introduces a new redundancy measure based on mutual information to improve generalization.
result Reduction of redundancy improves generalization capacity, reducing overfitting.
Two novel methods estimate multiple FDR directions for binary categorical responses.
problem Estimating multiple FDR directions for categorical responses.
method Information maximization and square loss mutual information.
result Statistical consistency of the proposed methods established.
A new method improves model generalization by recognizing representations.
problem Generalization issues in existing Information Bottlenecks.
method Recognizable Information Bottleneck (RIB) using f-CMI and density ratio matching.
result Improves model generalization through recognizability regularization.
New bounds for optimal transport using Gaussian processes and rate-distortion functions.
problem Finding bounds for entropic optimal transport with mutual information constraints.
method Lifting technique to construct a Gaussian process and applying the majorizing measure theorem.
result Maximum expected inner product is equivalent to a truncated integral involving the rate-distortion function.
Bayesian nonparametric framework improves robustness of MI estimation.
problem Challenges in exact MI computation in high dimensions with intractable likelihoods.
method Uses a Dirichlet process posterior to regularize MI loss, reducing sensitivity to fluctuations and outliers.
result Significant improvements in convergence over EDF-based methods, enhancing robustness and accuracy.
Variational inference with a factorized Gaussian posterior estimate is a widely used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information …
There are two types of deep generative models: explicit and implicit. The former defines an explicit density form that allows likelihood inference; while the latter targets a flexible transformation from random noise to generated samples. While the two classes of generative models have shown great power in many applica…
A method removes treatment-covariate dependence for counterfactual prediction without adversarial training.
problem Counterfactual prediction under assignment bias.
method Information-theoretic approach learning a stochastic representation Z to minimize mutual information with outcomes.
result The method performs favorably in likelihood, counterfactual error, and policy evaluation compared to adversarial baselines.
In this paper we consider the problem of semi-supervised learning with deep Convolutional Neural Networks (ConvNets). Semi-supervised learning is motivated on the observation that unlabeled data is cheap and can be used to improve the accuracy of classifiers. In this paper we propose an unsupervised regularization term…
Improves dialogue response model interpretability using attention and regularization.
problem Improving interpretability of dual encoder models for dialogue response suggestions.
method Integrates attention mechanism and novel regularization loss to emphasize important words.
result Improves model accuracy and interpretability compared to existing methods.
Learning to cooperate is crucially important in multi-agent environments. The key is to understand the mutual interplay between agents. However, multi-agent environments are highly dynamic, where agents keep moving and their neighbors change quickly. This makes it hard to learn abstract representations of mutual interp…
New methods stabilize EEG classification performance across subjects.
problem Large performance drop in EEG classification models on unseen subjects.
method Regularization techniques using divergence estimation.
result Significant increase in balanced accuracy on test subjects.
MIND estimates mutual information from ordinal data without full distributional knowledge.
problem Estimating mutual information from ordinal data with limited data.
method Copula-based maximum-entropy estimation of copula entropies.
result MIND estimator is consistent, data-efficient, and unbounded for any sample size.
Selective regression allows abstention to improve fairness criteria.
problem Selective regression can exacerbate disparities between subgroups.
method Proposes new fairness criteria and two approaches to mitigate performance disparity.
result Proposed fairness criteria ensures performance improvement for every subgroup with reduced coverage.
In reinforcement learning (RL), temporal abstraction still remains as an important and unsolved problem. The options framework provided clues to temporal abstraction in the RL, and the option-critic architecture elegantly solved the two problems of finding options and learning RL agents in an end-to-end manner. However…
Study optimizes fairness in predictive models by balancing utility and separation.
problem Balancing fairness and utility in predictive models.
method Information-theoretic approach using conditional mutual information (CMI).
result Reduces separation violations while maintaining or improving utility.