arXiv research

The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.

Lower bounds show many sampling algorithms need many gradient queries.

We study the information-theoretic lower bound of the sample complexity of the correct recovery of diffusion network structures. We introduce a discrete-time diffusion model based on the Independent Cascade model for which we obtain a lower bound of order $Ω(k \log p)$, for directed graphs of $p$ nodes, and at most $k$…

Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.

Lower bounds on Bayes risk for realizable models derived using information theory.

The paper develops a theory for identifying the best arm in non-parametric multi-armed bandits with a fixed budget.

This work optimizes quantization of linear models to reduce memory usage.

Optimal best-arm identification with known number of optimal arms.

Improved robust regression with clean covariates achieves better rates than Huber's model.

LinFACT identifies all ε-best arms in linear bandits with near-optimal efficiency.

Gaussian processes struggle with compositional functions, but deep Gaussian processes can outperform.

We consider the problem of learning a dictionary matrix from a number of observed signals, which are assumed to be generated via a linear model with a common underlying dictionary. In particular, we derive lower bounds on the minimum achievable worst case mean squared error (MSE), regardless of computational complexity…

Feature selection is one of the most fundamental problems in machine learning. An extensive body of work on information-theoretic feature selection exists which is based on maximizing mutual information between subsets of features and class labels. Practical methods are forced to rely on approximations due to the diffi…

We analyze the necessary number of samples for sparse vector recovery in a noisy linear prediction setup. This model includes problems such as linear regression and classification. We focus on structured graph models. In particular, we prove that sufficient number of samples for the weighted graph model proposed by Heg…

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using mu…

Paper proves tight lower bounds for online multicalibration, separating it from marginal calibration.

In statistical inference problems, we wish to obtain lower bounds on the minimax risk, that is to bound the performance of any possible estimator. A standard technique to obtain risk lower bounds involves the use of Fano's inequality. In an information-theoretic setting, it is known that Fano's inequality typically doe…

Paper proves KRR saturation effect for smooth functions.

This work sets lower bounds on the number of score queries needed for diffusion sampling.

In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this functional as a variational lower bound of a predictive information bottleneck objective. This generalized functional encompasses most moder…

Optimizes sensor usage for detecting abrupt changes in sensor data.

This paper sets a lower bound for sample complexity in inverse reinforcement learning.

A new mutual information lower bound for multimodal regression active learning.

The paper analyzes the efficiency of gradient estimation methods in noisy function evaluations.

Exact pairwise ranking is achievable but not possible under noisy comparisons.

Improved regret bounds for bandits with fixed expert advice using information theory.

Study on statistical estimation over Gaussian MAC, comparing analog and digital schemes.

A new method clusters intersecting lines using hypergraphs.

We consider in this paper the problem of noisy 1-bit matrix completion under a general non-uniform sampling distribution using the max-norm as a convex relaxation for the rank. A max-norm constrained maximum likelihood estimate is introduced and studied. The rate of convergence for the estimate is obtained. Information…

New bounds improve generalization in learning scenarios.

We consider the problem of dictionary learning under the assumption that the observed signals can be represented as sparse linear combinations of the columns of a single large dictionary matrix. In particular, we analyze the minimax risk of the dictionary learning problem which governs the mean squared error (MSE) perf…

New bounds show limitations of sample-wise information-theoretic generalization.

Study optimal control in unknown nonlinear systems with near-optimal regret bound.

Despite great popularity of applying softmax to map the non-normalised outputs of a neural network to a probability distribution over predicting classes, this normalised exponential transformation still seems to be artificial. A theoretic framework that incorporates softmax as an intrinsic component is still lacking. I…

Consider the problem of learning the drift coefficient of a stochastic differential equation from a sample path. In this paper, we assume that the drift is parametrized by a high dimensional vector. We address the question of how long the system needs to be observed in order to learn this vector of parameters. We prove…

The paper explores high-dimensional learning in finance, proving key aspects and setting lower bounds.

Unified framework for lower bounds in interactive decision making.

New algorithm recovers sparse binary vectors from generalized linear measurements efficiently.

Unified notation simplifies information-theoretic concepts in machine learning.

Paper proves first non-trivial PTF testing lower bounds for NGCA.

We consider the mixed regression problem with two components, under adversarial and stochastic noise. We give a convex optimization formulation that provably recovers the true solution, and provide upper bounds on the recovery errors for both arbitrary noise and stochastic noise settings. We also give matching minimax …

Study on estimating Gaussian mean with missing data in high dimensions.

We characterize the communication complexity of the following distributed estimation problem. Alice and Bob observe infinitely many iid copies of $ρ$-correlated unit-variance (Gaussian or $\pm1$ binary) random variables, with unknown $ρ\in[-1,1]$. By interactively exchanging $k$ bits, Bob wants to produce an estimate $…

Paper establishes lower bounds and optimal algorithms for deployment-efficient RL.

Relative to the large literature on upper bounds on complexity of convex optimization, lesser attention has been paid to the fundamental hardness of these problems. Given the extensive use of convex optimization in machine learning and statistics, gaining an understanding of these complexity-theoretic issues is importa…

New algorithms achieve optimal robustness in stochastic convex optimization under contamination.

We study the linear contextual bandit problem with finite action sets. When the problem dimension is $d$, the time horizon is $T$, and there are $n \leq 2^{d/2}$ candidate actions per time period, we (1) show that the minimax expected regret is $Ω(\sqrt{dT (\log T) (\log n)})$ for every algorithm, and (2) introduce a V…

Lower bound shows super-polynomial gap for estimating truncated Gaussian means.