A new method, REC, compresses images by encoding their latent representations efficiently.
problem Efficiently compressing single images with latent representations.
method Relative Entropy Coding (REC) that directly encodes latent representations with codelength close to relative entropy.
result REC is more efficient for single image compression compared to previous methods and is competitive for lossy compression.
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the ent…
We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encourage the encoder and decoder to learn consistent factorizations of the same underlying distribution; 2…
New insights into encoder-decoder structures using information measures.
problem Understanding the role of encoder-decoder design in machine learning.
method Using information sufficiency and mutual information loss concepts.
result Characterizes the expressiveness loss in encoder-decoder designs.
New bounds for quantum circuits depend on how data is encoded.
problem Lack of explicit dependence on data encoding in generalization bounds for PQCs.
method Derived generalization bounds that depend on data encoding strategies using Rademacher complexity and metric entropy.
result Optimal data-encoding strategies can be selected via structural risk minimization.
This work shows that supervised contrastive learning achieves similar results to cross-entropy but requires more iterations.
problem The question of whether there are fundamental differences in representation geometry between supervised contrastive learning and cross-entropy.
method The authors prove that both losses attain their minimum when representations of each class collapse to the vertices of a regular simplex, and they empirically validate this finding.
result Supervised contrastive learning requires more iterations to reach a close-to-optimal state compared to cross-entropy, indicating different optimization behavior.
The condition number predicts efficient information encoding in neural units, aiding model fine-tuning.
problem Efficient information encoding in neural units for various tasks and input modalities.
method Linking the condition number to the log-volume scaling factor and entropy of the output distribution.
result High condition number indicates efficient encoding, reducing overall information transfer.
ED-VAE improves VAEs by explicitly including entropy components in ELBO.
problem Limitations of traditional VAEs with ELBO in generating high-quality samples and interpreting latent spaces.
method Introduces ED-VAE, a re-formulation of ELBO that includes entropy and cross-entropy components.
result Significantly enhances model flexibility and improves interpretability and generative performance.
New method uses entropy dissipation to prove isoperimetric inequalities.
problem Proving isoperimetric inequalities in geometric settings.
method Information-theoretic approach based on entropy dissipation under heat flow.
result New proof of Euclidean isoperimetric inequality with sharp constant.
While the channel capacity reflects a theoretical upper bound on the achievable information transmission rate in the limit of infinitely many bits, it does not characterise the information transfer of a given encoding routine with finitely many bits. In this note, we characterise the quality of a code (i. e. a given en…
New measures for causal entropy and information gain studied.
problem Quantifying causal relationships in machine learning.
method Formal study of causal entropy and information gain.
result Established fundamental properties and relationships.
In this work, we propose an end-to-end block-based auto-encoder system for image compression. We introduce novel contributions to neural-network based image compression, mainly in achieving binarization simulation, variable bit rates with multiple networks, entropy-friendly representations, inference-stage code optimiz…
A new method learns graph compression from data.
problem Graphs lack ordering, making conventional compression algorithms ineffective.
method Partition and Code framework: decompose, learn, encode.
result PnC achieves compression gains that grow with graph size.
We propose a permutation-invariant loss function designed for the neural networks reconstructing a set of elements without considering the order within its vector representation. Unlike popular approaches for encoding and decoding a set, our work does not rely on a carefully engineered network topology nor by any addit…
Data containing human or social attributes may over- or under-represent groups with respect to salient social attributes such as gender or race, which can lead to biases in downstream applications. This paper presents an algorithmic framework that can be used as a data preprocessing method towards mitigating such bias.…
A new method for feature fusion in U-Net decoders using difference-based gating.
problem Precise fusion of high-level semantics and low-level details in U-Net decoder reconstruction.
method Proposes two difference-based gating approaches: Feature-difference gating (FDG) and Entropy-difference gating (EDG).
result Both FDG and EDG methods outperform existing attention-based fusion methods, with EDG showing superior performance.
Symplectic homology matches dual capacities for convex domains.
problem Understanding symplectic capacities and Reeb flows on convex domains.
method Isomorphic filtered symplectic homology to dual singular homology.
result Gutt-Hutchings capacities match spectral invariants for convex domains.
The price impact for a single trade is estimated by the immediate response on an event time scale, i.e., the immediate change of midpoint prices before and after a trade. We work out the price impacts across a correlated financial market. We quantify the asymmetries of the distributions and of the market structures of …
TSSC images enhance chaotic signal classification using ConvNets.
problem Classifying chaotic signals accurately and robustly.
method Triad State Space Construction (TSSC) for image encoding, Convolutional Neural Network (ConvNet) for classification.
result TSSC-ConvNet achieves high accuracy and robustness in chaotic signal classification.
ELBO of VAEs converges to a sum of three entropies.
problem Understanding the convergence of ELBO in VAEs.
method Analytical derivation of ELBO convergence for standard Gaussian VAEs.
result ELBO converges to a sum of three entropies at stationary points.
ALIEN improves uncertainty estimation of language models by refining entropy-based methods.
problem Overconfidence in uncertainty estimation for language models, especially for difficult inputs.
method ALIEN refines entropy-based uncertainty by aligning it with prediction reliability, using a lightweight uncertainty head.
result ALIEN consistently outperforms strong baselines in detecting incorrect predictions and achieving the lowest calibration error.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
We analyze the problem of learning a single user's preferences in an active learning setting, sequentially and adaptively querying the user over a finite time horizon. Learning is conducted via choice-based queries, where the user selects her preferred option among a small subset of offered alternatives. These queries …
In this manuscript we propose two objective terms for neural image compression: a compression objective and a cycle loss. These terms are applied on the encoder output of an autoencoder and are used in combination with reconstruction losses. The compression objective encourages sparsity and low entropy in the activatio…
We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn differe…
Unified approach to structured prediction combining entropy regularization and neuro-symbolic logic.
problem Structured prediction challenges due to large output spaces and insufficient labeled data.
method Neuro-symbolic entropy regularization loss that restricts entropy regularization to valid structures.
result Models predict more accurately and are more likely to be valid.
Mutual information bounds generalization error in variational classifiers.
problem Controlling overfitting in variational classifiers.
method Derive bounds on generalization error using mutual information.
result Mutual information bounds the generalization error in variational classifiers.
A grand challenge in representation learning is to learn the different explanatory factors of variation behind the high dimen- sional data. Encoder models are often determined to optimize performance on training data when the real objective is to generalize well to unseen data. Although there is enough numerical eviden…
Tensor networks and RNNs are equivalent, improving wave function encoding.
problem Efficiently encoding quantum states in neural networks.
method Generalized RNN architecture for tensor networks, supporting polynomial time wave function evaluation.
result Tensorial RNNs can encode quantum states with lower bond dimensions and higher accuracy.
Cubic predicts stock market indices by fusing stock latent embeddings and converting to binary classification.
problem Challenges in predicting stock market indices due to isolated time series treatment and simple regression.
method Fusion of stock latent embeddings, binary encoding classification, and confidence-guided prediction.
result Cubic outperforms state-of-the-art baselines in stock index prediction tasks.
Two new metrics assess LLM faithfulness and entropy, improving model reliability.
problem Evaluating the accuracy of LLMs in generating coherent responses.
method Proposes SF and SEP metrics based on information theory and thermodynamics.
result High SF and SEP scores indicate more faithful LLM responses.
This work enhances collaborative inference privacy by minimizing conditional entropy and boosting robustness against model inversion attacks.
problem Privacy leakage in collaborative inference systems via model inversion attacks.
method Theoretical proof and derivation of a differentiable measure for bounding conditional entropy, followed by a CEM algorithm to maximize it.
result Theoretical proof and experimental validation show that CEM consistently boosts inversion robustness without compromising feature utility or efficiency.
Proposes HypCSE for enhanced hierarchical clustering.
problem Challenges in existing hierarchical clustering methods.
method Hyperbolic Continuous Structural Entropy (HypCSE) neural networks.
result Superior performance on seven datasets.
The study examines how language models learn to represent the world, identifying conditions for ecological veridicality.
problem Understanding when language models learn to represent the world accurately and how this learning process can fail.
method Analyzes the Bayes-optimal next-token cross-entropy decomposition and the role of training ecology in shaping model representations.
result The minimum-complexity zero-excess solution is the quotient partition by training equivalence, and this solution is not preserved in in-context learning or per-task adaptation.
EPSTE: A geometric token and deep learning approach to estimating transfer entropy in neuroimaging time series
problem Inferring directed interactions between neural systems from EEG and MEG
method Reframing TE estimation as a learnable problem operating on structured symbolic representations
result EPSTE achieves near-perfect recovery of ground-truth directed structure and significantly lower absolute error than the baseline
We develop a complexity measure for large-scale economic systems based on Shannon's concept of entropy. By adopting Leontief's perspective of the production process as a circular flow, we formulate the process as a Markov chain. Then we derive a measure of economic complexity as the average number of bits required to e…
New method recovers curvature from heat diffusion data.
problem Recovering Riemannian curvature from heat diffusion.
method Information-theoretic approach using relative entropy.
result Local curvature determined by heat diffusion.
In many settings (e.g., robotics) demonstrations provide a natural way to specify tasks; however, most methods for learning from demonstrations either do not provide guarantees that the artifacts learned for the tasks, such as rewards or policies, can be safely composed and/or do not explicitly capture history dependen…
In information theory, Fisher information and Shannon information (entropy) are respectively used to quantify the uncertainty associated with the distribution modeling and the uncertainty in specifying the outcome of given variables. These two quantities are complementary and are jointly applied to information behavior…
This paper proposes a new method for efficient data compression using Bayesian neural networks.
problem Efficient compression of data represented as functions mapping coordinates to signal values.
method Overfitting variational Bayesian neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding.
result Our method achieves strong performance on image and audio compression while retaining simplicity.
The paper proposes a method to learn structured representations from unlabeled data using mutual information maximization.
problem Learning structured representations from unlabeled data.
method Adversarial maximization of mutual information between a structured latent variable and a target variable.
result The proposed method outperforms current baselines in document hashing and yields highly compressed interpretable representations.
This paper shows cross-entropy can recover latent structures in supervised learning.
problem Understanding why supervised learning works well and how models learn interpretable factors of variation.
method Extending identifiability results to parametric instance discrimination, proving cross-entropy minimization can recover latent structures up to linear transformations.
result Models trained with cross-entropy can learn representations of ground-truth factors of variation up to a linear transformation.
COIN++ compresses multiple data types efficiently.
problem Handling diverse data modalities in neural compression.
method Implicit neural representations and modulations quantization.
result Significant compression gains with reduced encoding time.
New method recovers clean data from corrupted samples.
problem Recovering clean data from corrupted samples with uncertainty.
method Probabilistic Tomographic Auto-Encoder method that derives reduced entropy condition approximate inference.
result Superior performance in imputation and de-noising compared to existing methods.
Spectral flow connects manifold geometry to rigidity criteria.
problem Tackling rigidity of simply-connected closed manifolds.
method Spectral deformation flow and invariant-based approach.
result Spherical profile is the unique manifold-compatible asymptotic realization.
DeepDIVE disentangles input into marginal and conditional distributions for multi-task learning.
problem Challenges in multi-task learning due to conflicting objectives.
method Inspired by probability theory, DeepDIVE uses a variational autoencoder with disentangled features and cross-attention mechanism.
result DeepDIVE disentangles input and improves forecast accuracy compared to baseline models.
We propose a novel node embedding of directed graphs to statistical manifolds, which is based on a global minimization of pairwise relative entropy and graph geodesics in a non-linear way. Each node is encoded with a probability density function over a measurable space. Furthermore, we analyze the connection between th…
This work explains how linear representations in large language models arise from training objectives and gradient descent.
problem Understanding the origins of linear representations in large language models.
method A latent variable model to abstract and formalize concept dynamics, combined with analysis of the softmax cross-entropy objective and gradient descent.
result Linear representations emerge when learning from data matching the latent variable model, and this simple structure suffices to yield linear representations.