GAttNHP predicts future events in temporal knowledge graphs by encoding long-range dependencies and handling mutual excitation.
problem Forecasting future events in temporal knowledge graphs due to long-range dependencies, mutual excitation, and heavy-tailed inter-arrival times.
method GAttNHP uses a self-attention encoder, semantic soft-grouping, and NCQ regression to address these issues.
result GAttNHP improves entity and time prediction on six benchmark TKG datasets compared to state-of-the-art baselines.
This paper improves DNN generalization by accurately estimating mutual information.
problem Intractability of estimating mutual information in DNNs.
method Introduces a probabilistic representation of DNNs to accurately estimate mutual information.
result Derives a tighter generalization bound than previous relaxations.
Attention improves edge prediction in e-commerce graphs.
problem Predicting edges in graphs from node attributes in e-commerce.
method Used attention mechanism in simple feedforward networks, derived analytically tractable model AttEST.
result Attention network outperforms LSTM architectures by over 20% on F-1 score.
Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …
We consider a framework involving behavioral economics and machine learning. Rationally inattentive Bayesian agents make decisions based on their posterior distribution, utility function and information acquisition cost Renyi divergence which generalizes Shannon mutual information). By observing these decisions, how ca…
Improves dialogue response model interpretability using attention and regularization.
problem Improving interpretability of dual encoder models for dialogue response suggestions.
method Integrates attention mechanism and novel regularization loss to emphasize important words.
result Improves model accuracy and interpretability compared to existing methods.
This paper introduces GEMINI, a new mutual information metric for unsupervised neural network training.
problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised MI by changing its core distance, introducing GEMINIs that do not require regularizations and can automatically select the number of clusters.
result GEMINIs can automatically select the number of clusters without requiring a priori knowledge of the number of clusters.
We investigate connections between information-theoretic and estimation-theoretic quantities in vector Poisson channel models. In particular, we generalize the gradient of mutual information with respect to key system parameters from the scalar to the vector Poisson channel model. We also propose, as another contributi…
This paper introduces GEMINI, a new metric for unsupervised neural network training that avoids the need for regularizations.
problem The mutual information (MI) as a clustering objective does not lead to satisfactory clusters.
method The authors generalised the mutual information by changing its core distance, introducing the Generalised Mutual Information (GEMINI).
result Some GEMINIs do not require regularizations when training and can automatically select the number of clusters.
Due to the advantage of achieving a better performance under weak regularization, elastic net has attracted wide attention in statistics, machine learning, bioinformatics, and other fields. In particular, a variation of the elastic net, adaptive elastic net (AEN), integrates the adaptive grouping effect. In this paper,…
AMI framework improves text generation by optimizing mutual information between source and target.
problem Previous MI approaches ignored the backward network, leading to loose variational bounds.
method AMI is a saddle point optimization framework that iteratively promotes and demotes generated instances.
result AMI significantly outperforms baselines on various text generation tasks.
Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As a…
Model learns image-word associations from captions using contrastive learning.
problem Phrase grounding, associating image regions to caption words.
method Optimizing word-region attention to maximize mutual information, using language model guided word substitutions for negatives.
result Model achieves 76.7% accuracy on Flickr30K Entities benchmark, a 5.7% gain from weak supervision.
Deep learning model classifies concurrent human interactions from WiFi data with high accuracy.
problem Classifying concurrent human interactions from WiFi data with high accuracy.
method Attention-BiGRU deep learning model using Multiple Input Multiple Output radio link.
result Maximum benchmark accuracy of 94% for a single subject-pair, 88% for ten subject pairs.
Predicting fine-grained interests of users with temporal behavior is important to personalization and information filtering applications. However, existing interest prediction methods are incapable of capturing the subtle degreed user interests towards particular items, and the internal time-varying drifting attention …
Activists align with large fund preferences for success.
problem Aligning with large fund preferences increases activist success.
method Analyzed previous proxy voting behavior to estimate preferences and correlated them with activist success.
result Campaigns with higher alignment receive more votes and are more successful.
We study the problem of using i.i.d. samples from an unknown multivariate probability distribution p to estimate the mutual information of p. This problem has recently received attention in two settings: (1) where p is assumed to be Gaussian and (2) where p is assumed only to lie in a large nonparametric smooth…
Study reveals self-attention's role in learning and generalizing interactions.
problem Understanding self-attention's theoretical role in neural architectures.
method Interacting entities analysis, including multi-agent RL and genetic sequences.
result Self-attention efficiently represents, learns, and generalizes pairwise interactions.
Black-box neural network models are widely used in industry and science, yet are hard to understand and interpret. Recently, the attention mechanism was introduced, offering insights into the inner workings of neural language models. This paper explores the use of attention-based neural networks mechanism for estimatin…
Collaborative filtering (CF) has been successfully employed by many modern recommender systems. Conventional CF-based methods use the user-item interaction data as the sole information source to recommend items to users. However, CF-based methods are known for suffering from cold start problems and data sparsity proble…
Paper proposes MIM-DRCFR to learn disentangled factors for better treatment effect estimation.
problem Learning disentangled factors precisely for individual-level treatment effect estimation.
method Multi-task learning framework with MI minimization criteria.
result MIM-DRCFR outperforms state-of-the-art methods in treatment effect estimation.
With the huge influx of various data nowadays, extracting knowledge from them has become an interesting but tedious task among data scientists, particularly when the data come in heterogeneous form and have missing information. Many data completion techniques had been introduced, especially in the advent of kernel meth…
Transformers learn causal structure through gradient descent on self-attention mechanisms.
problem Understanding how transformers learn causal structure during training.
method In-context learning task and simplified two-layer transformer model.
result Gradient descent on a simplified transformer learns to encode latent causal graphs.
New insights link RLHF and contrastive learning for better model alignment.
problem Aligning large language models with human values.
method Interpreting RLHF and DPO as contrastive learning methods based on mutual information.
result Proposed Mutual Information Optimization (MIO) improves model performance.
Deep neural networks are composed of layers of parametrised linear operations intertwined with non linear activations. In basic models, such as the multi-layer perceptron, a linear layer operates on a simple input vector embedding of the instance being processed, and produces an output vector embedding by straight mult…
Flow prediction (e.g., crowd flow, traffic flow) with features of spatial-temporal is increasingly investigated in AI research field. It is very challenging due to the complicated spatial dependencies between different locations and dynamic temporal dependencies among different time intervals. Although measurements of …
In order to successfully model Long Distance Dependencies (LDDs) it is necessary to understand the full-range of the characteristics of the LDDs exhibited in a target dataset. In this paper, we use Strictly k-Piecewise languages to generate datasets with various properties. We then compute the characteristics of the LD…
GOAT learns multiple node representations from graph structure alone.
problem Context-free graph representation learning limits model performance.
method Inspired by gossip and mutual attention, GOAT learns multiple node representations.
result GOAT outperforms 12 SOTA baselines on link prediction and clustering tasks.
Graph clustering is a fundamental task which discovers communities or groups in networks. Recent studies have mostly focused on developing deep learning approaches to learn a compact graph embedding, upon which classic clustering methods like k-means or spectral clustering algorithms are applied. These two-step framewo…
Model improves information transfer from visual streams.
problem Challenges in unsupervised learning from continuous visual data.
method Inspired by physics, maximizes mutual information through temporal process.
result Focus of attention enhances information transfer from input stream.
New neural network approach using mutual information.
problem Training neural networks for imbalanced datasets.
method Converts neural network classifiers to mutual information evaluators.
result New form of softmax leads to better classification accuracy, especially for imbalanced datasets.
Paper describes profiles of multivariate normal distributions and novel estimators for mutual information.
problem Estimating mutual information for complex distributions.
method Analytical description of profiles, introduction of Bend and Mix Models, Monte Carlo estimation.
result Bend and Mix Models accurately estimate mutual information profiles and provide Bayesian estimates.
Paper benchmarks mutual info estimators on diverse distributions.
problem Evaluating mutual information estimators on complex, real-world distributions.
method Constructs a diverse family of known-ground truth distributions, proposes a benchmark platform.
result Highlights differences in classical and neural estimators' performance across various conditions.
Neural estimator improves mutual information estimation in high dimensions.
problem Estimating mutual information in high dimensions is challenging.
method Parametrizing conditional densities with normalizing flows and using block autoregressive structure.
result Improved mutual information estimation on benchmark tasks.
Improved bounds on learning algorithms' performance using conditional mutual information.
problem Bounding the generalization error of learning algorithms.
method Introducing conditional mutual information and disintegrated mutual information to tighten bounds.
result New bounds are tighter than previous ones, especially for noisy, iterative algorithms.
Deep learning models can be understood through information theoretic analysis.
problem Understanding the learned representations of neural networks in security contexts.
method Information theoretic analysis of neural network representations using mutual information and homeomorphism.
result Mutual information remains invariant under homeomorphism, suggesting that neural networks require meaningful feature engineering.
We propose three measures of mutual dependence between multiple random vectors. All the measures are zero if and only if the random vectors are mutually independent. The first measure generalizes distance covariance from pairwise dependence to mutual dependence, while the other two measures are sums of squared distance…
The discovery of non-linear causal relationship under additive non-Gaussian noise models has attracted considerable attention recently because of their high flexibility. In this paper, we propose a novel causal inference algorithm called least-squares independence regression (LSIR). LSIR learns the additive noise model…
The paper argues that normalized mutual information is biased in clustering and community detection.
problem Bias in normalized mutual information for clustering and community detection.
method Introducing a modified version of mutual information to correct for information content and spurious dependence.
result The modified mutual information leads to different conclusions about which algorithms are best for community detection.
Synthetic construction of 3D complex bases.
problem Creating a complete set of unbiased bases in 3D complex space.
method Synthetic construction using complex projective trigonometry.
result Synthetic construction of mutually unbiased bases in C^3.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
This study analyzes mutual influence on investment strategies of financial market agents.
problem Mutual influence among agents in financial markets and its impact on investment strategies.
method Formulated optimal investment differential game problem, derived analytical solutions, proposed fast algorithm, and theoretically analyzed mutual influence.
result Agents' optimal strategies converge to the asymptotic strategy when mutual influence is strong and approaches infinity.
We find the maximum mutual information for neural networks and its key determinants.
problem Understanding the maximum mutual information in neural architectures.
method Derived closed-form expression for maximum mutual information across neural network families.
result Maximum mutual information stems from a generalized formula and is influenced by network width and statistical invariances.
Class-conditional extensions of generative adversarial networks (GANs), such as auxiliary classifier GAN (AC-GAN) and conditional GAN (cGAN), have garnered attention owing to their ability to decompose representations into class labels and other factors and to boost the training stability. However, a limitation is that…
We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn differe…
A new method estimates mutual information using neural classifiers.
problem Estimating mutual information for high-dimensional data is challenging.
method Trains a classifier to estimate joint distribution probability.
result Demonstrates high accuracy and reduces variance compared to variational methods.
New estimator improves mutual information estimation.
problem Estimating mutual information in data science and machine learning.
method Proposes a new estimator that uses a preliminary estimate of the data distribution.
result A preliminary estimate helps in estimating mutual information more accurately.
Study examines Indian equity mutual funds' investment style and risk-shifting.
problem Understanding how Indian equity mutual funds' investment styles affect their returns.
method Estimating size and style beta coefficients, identifying breakpoints, analyzing investment styles, and assessing risk-shifting intensity.
result Funds can enhance returns by shifting to high-return styles like Small Value and Small Blend.