arXiv research

New bounds show limitations of sample-wise information-theoretic generalization.

New bounds improve generalization in learning scenarios.

Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.

Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…

The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.

Proposes a new information-theoretic framework for analyzing deep neural networks.

Information-theoretic Bayesian regret bounds of Russo and Van Roy capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among o…

Novel bounds for SGLD show generalization error decreases with more samples.

We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling. By making a connection between information-theoretic quantities and confidence bounds, we obtain results that relate the per-period performance of the agent with its information gain about the enviro…

We analyze the necessary number of samples for sparse vector recovery in a noisy linear prediction setup. This model includes problems such as linear regression and classification. We focus on structured graph models. In particular, we prove that sufficient number of samples for the weighted graph model proposed by Heg…

Unified notation simplifies information-theoretic concepts in machine learning.

Neural networks learn complex functions efficiently near information-theoretic limits.

Information theoretical measures, such as entropy, mutual information, and various divergences, exhibit robust characteristics in image registration applications. However, the estimation of these quantities is computationally intensive in high dimensions. On the other hand, consistent estimation from pairwise distances…

This work tackles online memory selection in continual learning using information theory.

We consider the problem of sampling from a strongly log-concave density in $\mathbb{R}^d$, and prove an information theoretic lower bound on the number of stochastic gradient queries of the log density needed. Several popular sampling algorithms (including many Markov chain Monte Carlo methods) operate by using stochas…

In this paper, we study the information-theoretic limits of learning the structure of Bayesian networks (BNs), on discrete as well as continuous random variables, from a finite number of samples. We show that the minimum number of samples required by any procedure to recover the correct structure grows as $Ω(m)$ and $Ω…

BITS for GAPS uses Bayesian methods to improve surrogate model accuracy in complex systems.

Unified framework improves meta-learning generalization bounds.

Meta-learning bound uses conditional mutual information.

A new method clusters intersecting lines using hypergraphs.

The paper extends Thompson Sampling to infinite action spaces using information theory.

The information-theoretic analysis by Russo and Van Roy (2014) in combination with minimax duality has proved a powerful tool for the analysis of online learning algorithms in full and partial information settings. In most applications there is a tantalising similarity to the classical analysis based on mirror descent.…

New confidence intervals improve treatment effect estimation in randomized experiments.

Study on sparse recovery with mixed-quality data, establishing sample-size conditions.

Improved Thompson Sampling for logistic bandits with information-theoretic analysis.

This paper tightens information-theoretic bounds on generalization errors.

We consider the problem of model selection in Gaussian Markov fields in the sample deficient scenario. The benchmark information-theoretic results in the case of d-regular graphs require the number of samples to be at least proportional to the logarithm of the number of vertices to allow consistent graph recovery. When…

The paper tightens the regret rate for linear bandit problems.

Bounded rationality investigates utility-optimizing decision-makers with limited information-processing power. In particular, information theoretic bounded rationality models formalize resource constraints abstractly in terms of relative Shannon information, namely the Kullback-Leibler Divergence between the agents' pr…

Information theoretic measures (e.g. the Kullback Liebler divergence and Shannon mutual information) have been used for exploring possibly nonlinear multivariate dependencies in high dimension. If these dependencies are assumed to follow a Markov factor graph model, this exploration process is called structure discover…

Thompson Sampling tackles noisy context in stochastic bandits.

Ensemble sampling approximates Thompson sampling for complex models.

Neural network learns low-dimensional polynomials with SGD near information-theoretic limit.

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using mu…

DPPS uses DP priors for Bayesian non-parametric multi-arm bandits.

In this paper, we present and illustrate some new tools for rigorously analyzing training data selection methods. These tools focus on the information theoretic losses that occur when sampling data. We use this framework to prove that two methods, Facility Location Selection and Transductive Experimental Design, reduce…

The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.

Sharp concentration bounds for i.i.d. variables.

LinFACT identifies all ε-best arms in linear bandits with near-optimal efficiency.

New algorithm for nonnegative tensor completion with linear convergence rate.

New approach uses contrastive learning for better wireless power control.

Despite the availability of ever more data enabled through modern sensor and computer technology, it still remains an open problem to learn dynamical systems in a sample-efficient way. We propose active learning strategies that leverage information-theoretical properties arising naturally during Gaussian process regres…

Gaussian processes struggle with compositional functions, but deep Gaussian processes can outperform.

We study the information-theoretic lower bound of the sample complexity of the correct recovery of diffusion network structures. We introduce a discrete-time diffusion model based on the Independent Cascade model for which we obtain a lower bound of order $Ω(k \log p)$, for directed graphs of $p$ nodes, and at most $k$…

New bounds on IDS for RL show how to balance computation and learning efficiency.

Multi-view clustering has received much attention recently. Most of the existing multi-view clustering methods only focus on one-sided clustering. As the co-occurring data elements involve the counts of sample-feature co-occurrences, it is more efficient to conduct two-sided clustering along the samples and features si…

A new method for forgetting data from trained models using information theory.

Proposes ITISC for clustering with minimized worst-case expected distortions.