The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
The paper sets lower bounds for sampling non-log-concave distributions using Fisher information.
problem Understanding the complexity of sampling non-log-concave distributions.
method Proves two lower bounds using Fisher information in the context of sampling.
result Lower bounds on the complexity of sampling non-log-concave distributions, ruling out high-accuracy algorithms.
Lower bounds on Bayes risk for realizable models derived using information theory.
problem Deriving lower bounds on Bayes risk for realizable machine learning models.
method Information-theoretic analysis using rate-distortion theory and mutual information.
result Lower bounds on Bayes risk for realizable models, matching known bounds up to logarithmic factors.
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confide…
Optimizes experimental designs for intractable models using mutual information bounds.
problem Finding optimal experimental designs for models with intractable data-generating distributions.
method Maximizes mutual information lower bounds parametrized by neural networks, updating network parameters and designs simultaneously.
result Framework enables experimental design for various tasks including parameter estimation and model discrimination.
We consider a sequential learning problem with Gaussian payoffs and side information: after selecting an action i, the learner receives information about the payoff of every action j in the form of Gaussian observations whose mean is the same as the mean payoff, but the variance depends on the pair (i,j) (and may…
Unified framework for expert selection with bandit and lower-bound feedback.
problem Selecting the best expert in scenarios with bandit feedback and lower-bound information.
method Introduces a new feedback model combining bandit and lower-bound information, proving optimal regret bounds for modified Exp3 algorithms.
result Optimal regret bounds for modified Exp3 algorithms, generalizing both bandit and full-information settings.
Optimizes arm selection with side information in Gaussian bandits.
problem Optimizing arm selection with side information in Gaussian bandits.
method Constructs an LP-based asymptotic instance-dependent lower bound on the regret and develops the first known asymptotically optimal algorithm.
result First known asymptotically optimal algorithm for Gaussian bandits with side information.
Paper proves tight lower bounds for online multicalibration, separating it from marginal calibration.
problem Proving lower bounds for online multicalibration in relation to marginal calibration.
method Information-theoretic approach, constructing group families from orthonormal bases.
result Establishes tight lower bounds for online multicalibration, matching upper bounds up to logarithmic factors.
We consider the problem of sampling from a strongly log-concave density in Rd, and prove an information theoretic lower bound on the number of stochastic gradient queries of the log density needed. Several popular sampling algorithms (including many Markov chain Monte Carlo methods) operate by using stochas…
We study the information-theoretic lower bound of the sample complexity of the correct recovery of diffusion network structures. We introduce a discrete-time diffusion model based on the Independent Cascade model for which we obtain a lower bound of order Ω(klogp), for directed graphs of p nodes, and at most k…
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
This paper sets a lower bound for sample complexity in inverse reinforcement learning.
problem Finding a reward function that generates a desired optimal policy in MDPs.
method Information-theoretic lower bound using geometric construction and Fano's inequality.
result An O(nlogn) sample complexity lower bound for IRL problems. Paper proves first non-trivial PTF testing lower bounds for NGCA.
problem Proving lower bounds against PTF tests is challenging.
method Developed tools to prove PTF testing lower bounds for NGCA.
result First non-trivial PTF testing lower bounds for NGCA.
A new method for multi-objective Bayesian optimization using entropy search and variational lower bound maximization.
problem Efficiently optimizing multiple objectives in continuous domains.
method Approximates the Pareto-frontier using a mixture distribution and optimizes the balance through variational lower bound maximization.
result Demonstrated effectiveness especially with many objective functions.
The Fisher information approximation (FIA) is an implementation of the minimum description length principle for model selection. Unlike information criteria such as AIC or BIC, it has the advantage of taking the functional form of a model into account. Unfortunately, FIA can be misleading in finite samples, resulting i…
We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Information Lower BOund). It offers a data-independent measure of the complexity of the learned latent variable description, giving the log of the…
In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective. This generalized functional encompasses most moder…
We consider the problem of estimating the underlying graph associated with a Markov random field, with the added twist that the decoding algorithm can iteratively choose which subsets of nodes to sample based on the previous samples, resulting in an active learning setting. Considering both Ising and Gaussian models, w…
Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using mu…
Efficient algorithm for CLSBM reduces misclassification rate.
problem Reducing misclassification in community detection for CLSBM.
method Spectral-based algorithm for CLSBM, with theoretical misclassification bounds.
result Upper bound on misclassification rate of efficient algorithm.
New algorithm finds approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex stochastic optimization.
method Design of an algorithm using O(ε−3) stochastic gradient and Hessian-vector products. result Optimal rate of O(ε−3) for finding ε-approximate stationary points, matching lower bounds. Sum-of-Squares lower bound shows NGCA requires more samples than known algorithms.
problem Finding a non-Gaussian direction in a high-dimensional dataset.
method Sum-of-Squares (SoS) framework to prove lower bounds.
result First super-constant degree SoS lower bound for NGCA.
Paper establishes lower bounds and optimal algorithms for deployment-efficient RL.
problem Deployment efficiency in reinforcement learning.
method Optimization with constraints, lower bounds, algorithms.
result Established optimal algorithms for deployment-efficient RL.
Contextual information helps identify the best arm more efficiently.
problem Best arm identification with contextual covariate information.
method Proposed a context-aware version of the 'Track-and-Stop' strategy.
result Expected number of arm draws matches lower bound asymptotically.
Improved robust regression with clean covariates achieves better rates than Huber's model.
problem Robust regression under adaptive contamination of responses with clean covariates.
method Exploiting clean covariates to construct an estimator achieving better rates than Huber's model.
result Improved estimation rate even with constant contamination, achieving consistency.
Consider the problem of learning the drift coefficient of a stochastic differential equation from a sample path. In this paper, we assume that the drift is parametrized by a high dimensional vector. We address the question of how long the system needs to be observed in order to learn this vector of parameters. We prove…
Lower bound shows super-polynomial gap for estimating truncated Gaussian means.
problem Estimating mean of truncated Gaussian distribution with limited samples.
method Statistical Query (SQ) lower bounds for learning.
result Super-polynomial information-computation gap for the task.
Paper studies signal detection in noisy environments with limited communication.
problem Signal detection in Gaussian noise with 1-bit communication constraints.
method Derives lower bounds and exhibits optimal testing strategies.
result Optimal distributed testing strategies attain the derived lower bound.
New algorithm bounds regret in mediator feedback bandit problems.
problem Mediator feedback bandit game with policy sets.
method Adopting EXP4 algorithm, new regret bounds based on policy set capacity.
result Nearly-matching lower bounds for policy set families.
Combining the Information Bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proved successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the Deep Variational Information Bottleneck and the assumptions neede…
New Variational InfoMax objective improves neural network performance.
problem Optimizing neural networks using Bayesian Inference and Information Bottleneck.
method Derive Variational InfoMax (VIM) objective that maximizes InfoMax directly.
result VIM improves model performance in accuracy, robustness, and representation quality.
Paper studies statistical-computational trade-offs in tensor PCA and related problems.
problem Statistical-computational gap in tensor PCA estimation.
method Derives computational lower bounds using communication complexity.
result Lower bounds specify trade-off among passes, sample size, and memory.
New algorithm reduces clustering cost in bandit feedback.
problem Discovering hidden groupings in stochastic bandit feedback.
method ACB algorithm for active clustering with non-asymptotic lower bound.
result ACB algorithm's budget matches lower bound in most regimes.
We prove upper and lower bounds for the eigenvalues of the Dirac operator and the Laplace operator on 2-dimensional tori. In particluar we give a lower bound for the first eigenvalue of the Dirac operator for non-trivial spin structures. It is the only explicit estimate for eigenvalues of the Dirac operator known so fa…
Study minimax regret in sequential probability assignment with and without side information.
problem Minimax regret analysis in sequential probability assignment.
method Upper and lower bounds on minimax regret using square-root entropy.
result Lower bound matches upper bound for Donsker classes, up to log factors.
We estimate from below the isoperimetric profile of $S^2 \times \re^2$ and use this information to obtain lower bounds for the Yamabe constant of $S^2 \times \re^2$. This provides a lower bound for the Yamabe invariants of products S2×M2 for any closed Riemann surface M. Explicitly we show that $Y(S^2 \tim…
LinFACT identifies all ε-best arms in linear bandits with near-optimal efficiency.
problem Efficiently identifying multiple optimal candidates in high trial-and-error cost tasks.
method LinFACT algorithm designed for linear bandits, with information-theoretic lower bound and upper bound derivation integration.
result LinFACT achieves instance optimality, matching lower bound up to a logarithmic factor.
AMI framework improves text generation by optimizing mutual information between source and target.
problem Previous MI approaches ignored the backward network, leading to loose variational bounds.
method AMI is a saddle point optimization framework that iteratively promotes and demotes generated instances.
result AMI significantly outperforms baselines on various text generation tasks.
Variational Auto-encoders (VAEs) have been very successful as methods for forming compressed latent representations of complex, often high-dimensional, data. In this paper, we derive an alternative variational lower bound from the one common in VAEs, which aims to minimize aggregate information loss. Using our lower bo…
In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically doe…
The paper generalizes Bayesian Cramér-Rao inequality using information geometry of relative α-entropy.
problem Establishing a lower bound for the variance of an unbiased estimator for the α-escort distribution.
method Proposes a general Riemannian metric based on relative α-entropy to derive a generalized Bayesian Cramér-Rao inequality.
result Establishes a lower bound for the variance of an unbiased estimator for the α-escort distribution.
The paper develops a theory for identifying the best arm in non-parametric multi-armed bandits with a fixed budget.
problem Identifying the best arm in non-parametric multi-armed bandits with a limited number of trials.
method The paper proposes upper and lower bounds on the average log-probability of misidentification using information-theoretic quantities and a refined analysis of the successive-rejects strategy.
result The paper provides new upper and lower bounds on the average log-probability of misidentification, which generalize existing bounds.
We study the K-armed dueling bandit problem, a variation of the standard stochastic bandit problem where the feedback is limited to relative comparisons of a pair of arms. We introduce a tight asymptotic regret lower bound that is based on the information divergence. An algorithm that is inspired by the Deterministic…
Optimal best-arm identification with known number of optimal arms.
problem Identifying the best arm in a multi-armed bandit with multiple optimal arms under fixed confidence.
method Deriving a new information-theoretic lower bound and proposing a modified stopping rule.
result Achieving asymptotic instance-optimality with a new lower bound and new stopping rule.
Measuring Mutual Information (MI) between high-dimensional, continuous, random variables from observed samples has wide theoretical and practical applications. Recent work, MINE (Belghazi et al. 2018), focused on estimating tight variational lower bounds of MI using neural networks, but assumed unlimited supply of samp…
New algorithm for multi-player bandits without needing lower bounds or scaling inversely.
problem Multi-player bandits without collision sensing information.
method Proposes a novel algorithm that circumvents two problems of existing algorithms.
result Proves a theoretical regret upper bound and shows superior performance in practice.
Lower bounds for higher-order methods in non-convex optimization.
problem Proving lower bounds for higher-order methods in smooth non-convex finite-sum optimization.
method Analyzing deterministic and randomized algorithms, proposing a new smoothness assumption.
result Proves optimal lower bounds for simulating pth-order regularized methods on the whole function.