arXiv research

Paper analyzes ECE bias and provides bounds for its estimation.

We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling. By making a connection between information-theoretic quantities and confidence bounds, we obtain results that relate the per-period performance of the agent with its information gain about the enviro…

New analysis improves generalization bounds for meta-learning.

The information-theoretic analysis by Russo and Van Roy (2014) in combination with minimax duality has proved a powerful tool for the analysis of online learning algorithms in full and partial information settings. In most applications there is a tantalising similarity to the classical analysis based on mirror descent.…

New method explains sensitivity of test data uncertainty in Bayesian inference.

Optimizes SGLD noise structure for better generalization bounds.

Information-theoretic Bayesian regret bounds of Russo and Van Roy capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among o…

New research shows existing information-theoretic methods can't establish minimax rates for gradient descent in stochastic convex optimization.

We provide the first information theoretic tight analysis for inference of latent community structure given a sparse graph along with high dimensional node covariates, correlated with the same latent communities. Our work bridges recent theoretical breakthroughs in the detection of latent community structure without no…

We study the problem of robust mean estimation and introduce a novel Hamming distance-based measure of distribution shift for coordinate-level corruptions. We show that this measure yields adversary models that capture more realistic corruptions than those used in prior works, and present an information-theoretic analy…

In this paper, we derive generic bounds on the maximum deviations in prediction errors for sequential prediction via an information-theoretic approach. The fundamental bounds are shown to depend only on the conditional entropy of the data point to be predicted given the previous data points. In the asymptotic case, the…

The paper extends Thompson Sampling to infinite action spaces using information theory.

Proof of Gaussian ML estimator consistency in linear auto-regressive models.

The high pay packages of U.S. CEOs have raised serious concerns about what would constitute a fair pay.

The paper explores the information-theoretic nature of excess risk in machine learning.

Exact pairwise ranking is achievable but not possible under noisy comparisons.

Novel bounds for SGLD show generalization error decreases with more samples.

Unified framework improves meta-learning generalization bounds.

We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit an…

We study optimal estimation for sparse principal component analysis when the number of non-zero elements is small but on the same order as the dimension of the data. We employ approximate message passing (AMP) algorithm and its state evolution to analyze what is the information theoretically minimal mean-squared error …

SGD generalization bounds derived from information theory.

New method uses entropy dissipation to prove isoperimetric inequalities.

We prove a new minimax theorem connecting the worst-case Bayesian regret and minimax regret under partial monitoring with no assumptions on the space of signals or decisions of the adversary. We then generalise the information-theoretic tools of Russo and Van Roy (2016) for proving Bayesian regret bounds and combine th…

Improved Thompson Sampling for logistic bandits with information-theoretic analysis.

Proposes a new information-theoretic framework for analyzing deep neural networks.

In this paper, we present and illustrate some new tools for rigorously analyzing training data selection methods. These tools focus on the information theoretic losses that occur when sampling data. We use this framework to prove that two methods, Facility Location Selection and Transductive Experimental Design, reduce…

Study OOD generalization in meta-reinforcement learning using information theory.

Novel kernelized Renyi's entropy improves deep learning generalization bounds.

Unified framework for removing unwanted information from machine learning models.

New bounds improve generalization in learning scenarios.

scICML integrates multi-omics data from single cells using co-clustering.

New bounds show limitations of sample-wise information-theoretic generalization.

New TDA approach using Finsler metrics.

Acquiring abilities in the absence of a task-oriented reward function is at the frontier of reinforcement learning research. This problem has been studied through the lens of empowerment, which draws a connection between option discovery and information theory. Information-theoretic skill discovery methods have garnere…

New framework connects online learning to statistical learning for better generalization bounds.

Thompson Sampling tackles noisy context in stochastic bandits.

Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…

Ensemble sampling approximates Thompson sampling for complex models.

Study accelerates NAS research with a large dataset of ZC proxies.

The paper develops a theory for identifying the best arm in non-parametric multi-armed bandits with a fixed budget.

This work analyzes VQ-VAEs using information theory, focusing on latent variables and their impact on generalization and data generation.

In this work, we improve upon the stepwise analysis of noisy iterative learning algorithms initiated by Pensia, Jog, and Loh (2018) and recently extended by Bu, Zou, and Veeravalli (2019). Our main contributions are significantly improved mutual information bounds for Stochastic Gradient Langevin Dynamics via data-depe…

DPPS uses DP priors for Bayesian non-parametric multi-arm bandits.

In this paper we consider an information theoretic approach for the accounting classification process. We propose a matrix formalism and an algorithm for calculations of information theoretic measures associated to accounting classification. The formalism may be useful for further generalizations and computer-based imp…

Gaussian process (GP) models provide a powerful tool for prediction but are computationally prohibitive using large data sets. In such scenarios, one has to resort to approximate methods. We derive an approximation based on a composite likelihood approach using a general belief updating framework, which leads to a recu…

Lower bounds on Bayes risk for realizable models derived using information theory.

Objective: A variety of pattern analysis techniques for model training in brain interfaces exploit neural feature dimensionality reduction based on feature ranking and selection heuristics. In the light of broad evidence demonstrating the potential sub-optimality of ranking based feature selection by any criterion, we …

The paper tightens the regret rate for linear bandit problems.