The paper sets limits on prediction accuracy and generalization.
problem Fundamental limits of prediction accuracy and generalization.
method Combining entropic analysis and innovations approach.
result Conditions for achieving prediction error bounds.
Study applies market microstructure to Cuban informal currency market, finding market makers improve liquidity.
problem Understanding dynamics of informal currency markets.
method Modeling bid/ask intentions using Limit Order Book, applying Avellaneda-Stoikov model with Market Maker.
result Market Maker improves market quality and bid/ask dynamics.
The paper uses information theory to find limits of feedback control systems.
problem Fundamental performance limitations of feedback control systems.
method Utilizes information theory to derive Lp bounds on control error. result Bounds are characterized by the conditional entropy of the disturbance.
Paper models limit order book with informed traders and market makers.
problem Modeling the limit order book with heterogeneous market participants.
method Agent-based model with four types of participants: informed traders, noise traders, informed market makers, and noise market makers. Based on Glosten-Milgrom and Huang-Rosenbaum-Saliba approaches.
result Derived the static limit order book characteristics and compared them with existing models.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Study sets limits for detecting a subhypergraph in uniform hypergraphs.
problem Recovering a subhypergraph from a uniform hypergraph with different edge probabilities.
method Information-theoretic analysis for weak and exact recovery.
result Sharp conditions for weak or exact recovery of the subhypergraph.
A limit order book provides information on available limit order prices and their volumes. Based on these quantities, we give an empirical result on the relationship between the bid-ask liquidity balance and trade sign and we show that liquidity balance on best bid/best ask is quite informative for predicting the futur…
New bounds show limitations of sample-wise information-theoretic generalization.
problem Limitations of sample-wise information-theoretic generalization bounds.
method Analysis of existing bounds and derivation of new bounds.
result No sample-wise information-theoretic bounds exist for expected squared generalization gap.
HLOB predicts mid-price changes in L.O.Bs using deep learning.
problem Forecasting mid-price changes in Limit Order Books.
method HLOB uses a deep learning model with an Information Filtering Network and Homological Convolutional Neural Networks.
result HLOB outperforms state-of-the-art models in real-world datasets.
Study insider trading benefits in a market with high transaction costs.
problem Super--replication of European contingent claims in illiquid markets.
method Model insider information and quadratic transaction costs, analyze scaling limits.
result Scaling limit gives the value of insider information.
Study utility indifference pricing with delayed investment information in a Bachelier model.
problem Investment decisions based on delayed information in a Bachelier model.
method Developed discrete-time duality and used techniques from [7] to compute scaling limits.
result Utility indifference prices scaling limit for vanishing delay with quadratic penalty.
Network embedding helps predict speed limits on incomplete Danish road network.
problem Incomplete speed limit data on Danish roads limits machine learning applications.
method Applied node2vec network embedding to Danish road network.
result Network embedding can derive useful features for predicting speed limits.
Study a market with uncertain informed traders, finding price impact depends on both asset value and informed trader count distribution.
problem Uncertain participation of informed traders in a market with limit orders.
method Characterized equilibrium by a fixed point integral equation, analyzed large order asymptotics, solved numerically.
result Equilibrium price impact depends on both asset value and distribution of informed traders, not just expected number of informed traders.
This paper uses information theory to improve risk modeling in big data.
problem Insufficient application of information theory in actuarial science.
method Explores information theory to uncover performance limits of insurance big data systems.
result Guidance for risk modeling and actuarial pricing systems.
In this paper, we study the information-theoretic limits of learning the structure of Bayesian networks (BNs), on discrete as well as continuous random variables, from a finite number of samples. We show that the minimum number of samples required by any procedure to recover the correct structure grows as Ω(m) and $Ω…
Study neural communication systems with bandwidth-limited channels.
problem Reliable message transmission despite noisy channels.
method Jointly model compression and error correction with neural networks; introduce prior for missing information; use auxiliary latent variables.
result Joint neural communication systems outperform separate models under expected information loss.
In network embedding, random walks play a fundamental role in preserving network structures. However, random walk based embedding methods have two limitations. First, random walk methods are fragile when the sampling frequency or the number of node sequences changes. Second, in disequilibrium networks such as highly bi…
Study on limits of LLM-based multi-agent planning reliability.
problem Reliability limits of LLM-based multi-agent planning.
method Modeling LLM-based multi-agent architecture as a decision network, showing dominance by centralized Bayes decision maker.
result Optimizing multi-agent directed acyclic graphs under communication budget is equivalent to choosing a constrained experiment.
OMGD algorithm optimizes online convex optimization with switching costs and delayed gradients.
problem Optimizing online convex optimization with switching costs and delayed gradients.
method Proposed an online multiple gradient descent (OMGD) algorithm for quadratic and linear switching costs.
result OMGD achieves optimal dynamic regret in the limited information setting.
A distinctive property of human and animal intelligence is the ability to form abstractions by neglecting irrelevant information which allows to separate structure from noise. From an information theoretic point of view abstractions are desirable because they allow for very efficient information processing. In artifici…
Harmonization schemes limit accuracy due to domain information.
problem Harmonization schemes lead to inaccurate predictions due to domain information.
method Analysis of mutual information and real label value informativeness.
result Accuracy is limited by the domain with least information.
New method uses KL-divergence to create non-informative priors for multivariate Gaussian.
problem Handling hyperparameters for non-informative limits in multivariate Gaussian conjugate priors.
method Using scaled KL-divergence between multivariate Gaussians to construct Wishart and normal-Wishart conjugate priors.
result Forming non-informative priors without violating Wishart shape parameter restrictions.
Study on Privileged ERM showing limitations and providing capacity analysis.
problem Improving classification accuracy with privileged information.
method Theoretical analysis of Privileged ERM using VC dimension and generalization bounds.
result Worst-case guarantees for Privileged ERM cannot improve over standard ERM unless privileged information capacity is similar or smaller.
Develops a framework for decision-making abstractions under computational limitations.
problem Decision-making by agents with limited computational resources.
method Information-theoretic signal compression and optimization problem formulation.
result Generates a hierarchy of abstractions for a non-trivial environment.
Study shows how learning and analytical models affect reneging and jockeying in a dual M/M/1 system.
problem How do learning and analytical models affect reneging and jockeying in a dual M/M/1 system?
method Analytical and online trained actor-critic models were used to study reneging and jockeying in a dual M/M/1 system.
result Both analytical and online trained actor-critic models yield the same asymptotic limits for reneging and jockeying, but differ in practical sizes.
Financial markets, with their vast range of different investment opportunities, can be seen as a system of many different simultaneous games with diverse and often unknown levels of risk and reward. We introduce generalizations to the classic Kelly investment game [Kelly (1956)] that incorporates these features, and us…
New method detects information leakage using approximate Bayes predictor.
problem Unintentional exposure of sensitive information via observable data.
method Statistical learning theory and information theory framework, approximating Bayes predictor's log-loss and accuracy.
result MI can be accurately estimated to detect ILs, outperforming state-of-the-art baselines.
Physics-informed methods infer spatial dynamics from static snapshots, but limits exist.
problem Inferring spatial dynamics from static molecular patterns.
method Combining flexible representations with mechanistic constraints, analyzing structural identifiability, and adapting physics-informed schemes.
result Static spatial patterns can identify spatially varying dynamics, but limits exist due to modeling choices.
Global graph structure improves GNN performance.
problem Limited graph structure in GNNs leads to indistinguishable node embeddings.
method Empirically tested the impact of global graph information on GNN performance.
result Global information can significantly improve GNN performance by more than 5%.
The recent trend for acquiring big data assumes that possessing quantitatively more and qualitatively finer data necessarily provides an advantage that may be critical in competitive situations. Using a model complex adaptive system where agents compete for a limited resource using information coarse-grained to differe…
Dynamic acquisition of features improves predictions with limited data.
problem Limited or uncertain data requires additional relevant information for accurate assessments.
method Proposes models that dynamically acquire new features using conditional mutual information and arbitrary conditional flow.
result Demonstrates superior performance over baselines in multiple settings.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
This paper investigates a multi-terminal source coding problem under a logarithmic loss fidelity which does not necessarily lead to an additive distortion measure. The problem is motivated by an extension of the Information Bottleneck method to a multi-source scenario where several encoders have to build cooperatively …
Information-theoretic bounded rationality describes utility-optimizing decision-makers whose limited information-processing capabilities are formalized by information constraints. One of the consequences of bounded rationality is that resource-limited decision-makers can join together to solve decision-making problems …
Variational inference with a factorized Gaussian posterior estimate is a widely used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information …
Modeling trading behavior with information signals and limit order books, showing market impact and equilibrium properties.
problem Analyzing the impact of information signals on trading behavior and market equilibrium in limit order books.
method Static equilibrium model with profit-maximizing investors and competitive dealers, using iterative algorithms and asymptotic analysis.
result The market impact of large trades follows a power law with fat tails and a logarithmic law with lighter tails, and the order book flattens as noise trading increases.
Online boosting for multilabel ranking with limited feedback.
problem Multilabel ranking with top-k feedback.
method Surrogate loss function and unbiased estimator for weak learners.
result Adapted full information multilabel ranking algorithms to top-k feedback setting with theoretical and experimental support.
Subjective expected utility theory assumes that decision-makers possess unlimited computational resources to reason about their choices; however, virtually all decisions in everyday life are made under resource constraints - i.e. decision-makers are bounded in their rationality. Here we experimentally tested the predic…
Shannon's mathematical theory of communication defines fundamental limits on how much information can be transmitted between the different components of any man-made or biological system. This paper is an informal but rigorous introduction to the main ideas implicit in Shannon's theory. An annotated reading list is pro…
The Information Plane theory predicts autoencoders do not compress input information.
problem Understanding the training dynamics of hidden layers in autoencoders.
method Derive a theoretical convergence for the Information Plane of autoencoders using a Gram-matrix based mutual information estimator.
result Ideal autoencoders with a large bottleneck layer size do not compress input information, while a small size causes compression only in the encoder layers.
Paper revisits Deep Variational Information Bottleneck and proposes a new optimization approach.
problem Limitations of Deep Variational Information Bottleneck in optimizing mutual information.
method Proposes a new optimization approach by circumventing the limitation of requiring both Markov chains during optimisation.
result Shows how to optimise a lower bound for mutual information, circumventing the limitation of requiring both Markov chains.
To model modern large-scale datasets, we need efficient algorithms to infer a set of P unknown model parameters from N noisy measurements. What are fundamental limits on the accuracy of parameter inference, given finite signal-to-noise ratios, limited measurements, prior information, and computational tractability …
Two parallel samplers enhance image quality in limited denoising steps.
problem Limited denoising steps in diffusion models reduce image quality.
method Two parallel samplers denoise at successive times, integrating their information.
result Two parallel samplers improve image quality compared to a single sampler.
New research on limits of transfer learning, proving key selection and dependence requirements.
problem Insufficient theoretical foundation for transfer learning.
method Proved novel results on transfer learning, emphasizing selection of information and dependence between domains.
result Upper bound on improvement possible with transfer learning, highlighting the need for careful selection.
Current neural network-based classifiers are susceptible to adversarial examples even in the black-box setting, where the attacker only has query access to the model. In practice, the threat model for real-world systems is often more restrictive than the typical black-box model where the adversary can observe the full …
Proposes a new framework for resource-limited recommendation.
problem Resource constraints affect user choices in recommendation tasks.
method Interest-behavior multiplicative network with MRRNNs and resource-limited branch.
result Framework effectively predicts user interactions considering resource limitations.
Neural network learns low-dimensional polynomials with SGD near information-theoretic limit.
problem Learning a single-index target function with gradient descent.
method Two-layer neural network optimized by SGD on squared loss.
result Sample and runtime complexity of n≃T=Θ(d⋅polylogd) for polynomial single-index models, matching information theoretic limit up to polylogarithmic factors. Adversarially-trained models transfer better in limited data scenarios.
problem Improving transfer learning performance with limited data.
method Adversarially-train deep nets, freeze early layers, fine-tune last layers, observe shape vs texture bias.
result Adversarially-trained models transfer better than non-adversarially-trained models, especially with limited data.