arXiv research

Eluder dimension and information gain are equivalent for reproducing kernel Hilbert spaces.

Neural estimator improves mutual information estimation in high dimensions.

The article examines entropy-information inequalities for continuous-time Markov chains under curvature-dimension conditions.

Introduces relative information gain for improving Gaussian process regression rates.

Estimates expected information gain using density approximations and dimension reduction.

Estimates latent dimensionality for prediction tasks using mutual information.

Optimal persuasion involves projecting state vectors onto lower-dimensional 'optimal information manifolds'.

Information about intrinsic dimension is crucial to perform dimensionality reduction, compress information, design efficient algorithms, and do statistical adaptation. In this paper we propose an estimator for the intrinsic dimension of a data set. The estimator is based on binary neighbourhood information about the ob…

Information concentration of probability measures have important implications in learning theory. Recently, it is discovered that the information content of a log-concave distribution concentrates around their differential entropy, albeit with an unpleasant dependence on the ambient dimension. In this work, we prove th…

Unified bounds linking compressibility, fractal dimensions, and mutual information.

Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.

We introduce a notion of "effective dimension" of a statistical model based on the number of cubes of size $1/\sqrt{n}$ needed to cover the model space when endowed with the Fisher Information Matrix as metric, $n$ being the number of observations. The number of observations fixes a natural scale or resolution. The eff…

Bayesian optimization improves with nonstationary covariance functions.

This paper improves sample efficiency in noisy inductive matrix completion with side-information.

This study quantifies the scalability of k-Sliced Mutual Information (k-SMI) with dimension.

Two novel methods estimate multiple FDR directions for binary categorical responses.

LMI approximates mutual information in high dimensions using learned low-dimensional representations.

Study improves sampling from non-log-concave distributions using Fisher information.

This paper analyzes Barlow Twins' representation efficiency using information-geometric methods.

New method solves high-dimensional PDEs fast using physics-informed neural networks.

Unified physics-informed learning method improves generalization performance.

Contrastive learning adapts to data intrinsic dimensions, learning low-dimensional representations.

This study analyzes VAEs using ID and II, revealing a transition in behaviour and distinct training phases.

To certain types of generic distributions (subbundles in a tangent bundle) one can associate canonical Cartan connections. Many of these constructions fall into the class of parabolic geometries. The aim of this article is to show how strong restrictions on the possibles sizes of automorphism groups of such distributio…

New method improves counterfactual distribution learning for high-dimensional outcomes.

New bounds explain modern machine learning algorithms' generalization.

This paper reviews and compares supervised linear dimension-reduction techniques.

Study on Privileged ERM showing limitations and providing capacity analysis.

How many bits of information are required to PAC learn a class of hypotheses of VC dimension $d$? The mathematical setting we follow is that of Bassily et al. (2018), where the value of interest is the mutual information $\mathrm{I}(S;A(S))$ between the input sample $S$ and the hypothesis outputted by the learning algo…

Study online multiclass classification under bandit feedback, extending previous results.

In the field of machine learning, it is still a critical issue to identify and supervise the learned representation without manually intervening or intuition assistance to extract useful knowledge or serve for the downstream tasks. In this work, we focus on supervising the influential factors extracted by the variation…

Study provides bounds for estimating intrinsic dimension using Gaussian kernels.

NGD models have higher effective dimension than SGD models.

New method corrects Laplace/BIC errors in singular models, revealing effective dimension.

New method extracts factors of variation from data without much supervision.

A typical goal of supervised dimension reduction is to find a low-dimensional subspace of the input space such that the projected input variables preserve maximal information about the output variables. The dependence maximization approach solves the supervised dimension reduction problem through maximizing a statistic…

The study reveals optimal early stopping behaviors in deep learning models.

Proposes CLUB for reliable MI minimization in high dimensions.

Lazy, perfectly informed investors trade infrequently due to costs.

Dimension Estimation (DE) and Dimension Reduction (DR) are two closely related topics, but with quite different goals. In DE, one attempts to estimate the intrinsic dimensionality or number of latent variables in a set of measurements of a random vector. However, in DR, one attempts to project a random vector, either l…

A new method reduces both input and output dimensions for better goal-oriented analysis.

Our goal in this paper is to develop an effective estimator of fractal dimension. We survey existing ideas in dimension estimation, with a focus on the currently popular method of Grassberger and Procaccia for the estimation of correlation dimension. There are two major difficulties in estimation based on this method. …

Study minimax regret in sequential probability assignment with and without side information.

Study identifies key parameters and input dimensions making LLMs and VLMs brittle.

Paper proposes conditional multidimensional scaling for better data reduction.

Study nearest-neighbor radii under dependent sampling, finding they remain informative.

We study the problem of using i.i.d. samples from an unknown multivariate probability distribution $p$ to estimate the mutual information of $p$. This problem has recently received attention in two settings: (1) where $p$ is assumed to be Gaussian and (2) where $p$ is assumed only to lie in a large nonparametric smooth…

SCBMs model causal effects using low-dimensional bottlenecks.