A new framework for information theory considers computational constraints.
problem Understanding information in complex systems with computational limitations.
method Variational extension of Shannon's information theory with computational constraints.
result Predictive V \mathcal{V} V -information can be created through computation and reliably estimated from data. The paper sets limits on prediction accuracy and generalization.
problem Fundamental limits of prediction accuracy and generalization.
method Combining entropic analysis and innovations approach.
result Conditions for achieving prediction error bounds.
Combines Bézier curves with Gaussian processes for better sequential data modeling.
problem Limited expressiveness of MDNs in probabilistic modeling of sequential data.
method Integrates Gaussian processes with probabilistic Bézier curves for full Bayesian inference.
result Improves expressiveness of MDNs by enabling full Bayesian inference.
Physics-informed GP regression solves eigenvalue problems by identifying non-trivial eigenspaces.
problem Solving eigenvalue problems of linear operators with trivial solutions.
method Constructing a transfer function-type indicator using physics-informed Gaussian Process posterior.
result The posterior covariance is non-trivial only for eigenvalues of the operator, indicating non-trivial eigenspaces.
The paper decomposes probabilistic scores into reliability, uncertainty, and information loss.
problem Understanding the reliability and uncertainty of probabilistic predictions.
method Developed decomposition identities for proper losses, quantifying reliability, residual uncertainty, and information gain.
result A three-term identity for classification scores, revealing miscalibration, grouping term, and feature-level uncertainty.
Optimal online learning algorithms for label-efficient prediction and bandits.
problem Efficient prediction in online learning with limited information.
method Optimistic online mirror descent with second order corrections and hybrid regularizers.
result Improved regret bounds for label-efficient prediction and bandits.
Study learns optimal strategies in imperfect information games with self-play.
problem Learning optimal strategies in imperfect information games.
method Proposes Follow the Regularized Leader (FTRL) algorithms for imperfect information games.
result Proposes two FTRL algorithms: Balanced FTRL and Adaptive FTRL.
The paper addresses fairness in online learning by extending auditing schemes and presenting efficient algorithms.
problem Ensuring fairness in online learning while maximizing predictive accuracy.
method Extending auditing schemes to handle multiple auditors and presenting oracle-efficient algorithms.
result Presented algorithms achieve upper bounds on regret and fairness violations, improving on existing bounds.
A new Helmholtzian operator from point clouds for flow analysis.
problem Analyzing flows and vector fields on manifolds from point cloud data.
method Estimation of manifold Helmholtzian from point cloud data using weighted 1-Laplacian.
result The Helmholtzian operator L 1 \mathcal L_1 L 1 effectively smooths, predicts, and extracts features from flows on manifolds. LeanML reduces machine learning project waste by estimating best performance without training models.
problem Avoidable wastes in machine learning projects.
method Lean design pattern based on mutual information and performance metrics.
result Estimating best performance without training models is faster and cheaper.
Stokes' theorem's boundary maximizes entropy.
problem Characterizing the boundary of a manifold using entropy.
method Maximizing entropy for codimension-1 submanifolds satisfying Stokes' theorem.
result The boundary of a manifold maximizes the entropy functional.
This paper introduces a new Lagrangian for the Information Bottleneck problem to simplify optimization.
problem Optimizing compressed representations for predicting Y Y Y while limiting information about X X X . method Introduces a general family of Lagrangians to explore the Information Bottleneck curve.
result Solves the original constrained optimization problem with a single optimization.
Bayesian framework calibrates imperfect models using physics-informed priors and Hamiltonian Monte Carlo.
problem Quantifying uncertainty in imperfect computer models described by differential equations.
method Physics-informed Gaussian process priors, discrepancy function, Hamiltonian Monte Carlo, data approximations.
result Framework accurately recovers true parameters and produces accurate predictions.
We give an online algorithm and prove novel mistake and regret bounds for online binary matrix completion with side information. The mistake bounds we prove are of the form O ~ ( D / γ 2 ) \tilde{O}(D/γ^2) O ~ ( D / γ 2 ) . The term 1 / γ 2 1/γ^2 1/ γ 2 is analogous to the usual margin term in SVM (perceptron) bounds. More specifically, if we assume that there i…
How many bits of information are required to PAC learn a class of hypotheses of VC dimension d d d ? The mathematical setting we follow is that of Bassily et al. (2018), where the value of interest is the mutual information I ( S ; A ( S ) ) \mathrm{I}(S;A(S)) I ( S ; A ( S )) between the input sample S S S and the hypothesis outputted by the learning algo…
The paper connects Bergman geometry with information geometry.
problem Exploring the Bergman geometry of complex domains.
method Introducing a mapping Φ and using Fisher information metrics.
result Established a new statistical curvature formula for the Bergman metric.
This work uses QPGPs to improve ILC performance in repetitive tasks.
problem Performance degradation in repetitive motion tasks due to environmental changes and robot wear.
method Incorporates Quasi-Periodic Gaussian Processes into a predictive ILC framework.
result The proposed approach achieves faster convergence and robustness under disturbances.
Unified CMI bounds for meta learning, bridging information-theoretic and classical learning theory.
problem Lack of unified generalization bounds for meta learning.
method Evaluated CMI (e-CMI) framework for meta learning.
result Unified generalization bounds for meta learning in terms of e-CMI.
Jordan algebras in information geometry linked to metrics on probability distributions.
problem Understanding Jordan algebras in information geometry.
method Inspired by Kirillov's coadjoint orbits, a pseudo-Riemannian metric is constructed on Jordan algebra leaves.
result Not all points in the dual space lie on a leaf, and the metric structure depends on the cone of positive functionals.
This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.
problem Understanding why unsupervised pretraining helps in machine learning tasks.
method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.
result Proves an excess risk of i l d e O ( C Φ / m + C Ψ / n ) ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n}) i l d e O ( C Φ / m + C Ψ / n ) for downstream tasks under mild conditions. PASTIS method selects simple models from noisy data.
problem Selecting correct models from large candidate libraries.
method PASTIS (Parsimonious Stochastic Inference) using extreme value theory.
result PASTIS outperforms other methods in model identification and predictive capability.
Assume that M ( T ) M(\mathcal{T}) M ( T ) is a rational homology sphere plumbed 3-manifold associated with a connected negative definite graph T \mathcal{T} T . We consider the combinatorial multivariable Poincaré series associated with T \mathcal{T} T and its counting functions, which encode rich topological information. Using the `per…
Improved mean estimation for symmetric distributions with finite-sample guarantees.
problem Estimating the mean of a symmetric distribution from samples.
method Using Fisher information rate for finite-sample guarantees.
result Finite-sample convergence close to subgaussian with variance 1/(n * I_r), where I_r is r-smoothed Fisher information.
Identifies minimal training subset to flip a prediction.
problem Flipping predictions in machine learning models.
method Extended influence function for relabeling minimal subset.
result Relabeling fewer than 2% of training points can flip a prediction.
The space of all probability measures having positive density function on a connected compact smooth manifold M M M , denoted by P ( M ) \mathcal{P}(M) P ( M ) , carries the Fisher information metric G G G . We define the geometric mean of probability measures by the aid of which we investigate information geometry of P ( M ) \mathcal{P}(M) P ( M ) , equ…
New algorithm reduces regret and constraint violation in online convex optimization with predictions.
problem Online convex optimization with time-varying constraints and predictions.
method Primal-dual algorithm combining Follow-The-Regularized-Leader with adaptive steps.
result Achieves O ( T 3 − β 4 ) \mathcal O(T^{\frac{3-β}{4}}) O ( T 4 3 − β ) regret and O ( T 1 + β 2 ) \mathcal O(T^{\frac{1+β}{2}}) O ( T 2 1 + β ) constraint violation bounds. Paper justifies ideal point forecasts as measurable, clarifying conditions for their existence.
problem Justifying ideal point forecasts as measurable random variables.
method Clarifying and establishing measurability conditions for a wide class of functionals.
result Ideal point forecasts are shown to be measurable, providing theoretical justification.
Bayesian active learning method improved for censored regression data.
problem Challenges in estimating BALD for censored regression data.
method Derived entropy and mutual information for censored distributions, developed C \mathcal{C} C -BALD objective, proposed novel modelling approach. result Demonstrated C \mathcal{C} C -BALD outperforms other methods in censored regression. Gaussian processes (GPs) provide a powerful framework for extrapolation, interpolation, and noise removal in regression and classification. This paper considers constraining GPs to arbitrarily-shaped domains with boundary conditions. We solve a Fourier-like generalised harmonic feature representation of the GP prior in…
Accurately annotating large scale dataset is notoriously expensive both in time and in money. Although acquiring low-quality-annotated dataset can be much cheaper, it often badly damages the performance of trained models when using such dataset without particular treatment. Various methods have been proposed for learni…
Physics-informed DeepONets solve PDEs without paired data, predicting solutions quickly.
problem Lack of paired input-output data for solving PDEs.
method Physics-informed DeepONets use automatic differentiation to enforce physical laws as soft penalty constraints.
result Physics-informed DeepONets can solve PDEs without paired data, predicting solutions up to 3 orders of magnitude faster.
PeL separates sensory interface optimization from decision learning.
problem Optimizing sensory interfaces without task-specific information.
method Formal separation of perception and decision learning, using metrics for stability, informativeness, and geometry.
result Updates preserving invariants are orthogonal to decision gradients.
The paper characterizes the efficiency of transferring knowledge from a teacher to a student classifier over finite domains.
problem Characterizing the statistical efficiency of knowledge transfer over finite domains.
method Three progressive levels of privileged information: hard labels, teacher probabilities, and soft labels. Novel empirical loss functions used to achieve the fundamental limits.
result Achieving the fundamental limits of knowledge transfer through specific levels of privileged information and novel loss functions.
Improved Thompson Sampling for logistic bandits with information-theoretic analysis.
problem Optimizing binary reward probabilities in logistic bandit problems.
method Information-theoretic framework, focusing on the information ratio and minimax measure.
result Bound on Bayesian expected regret of O ( d / α T log ( β T / d ) ) O(d/α\sqrt{T \log(βT/d)}) O ( d / α T log ( β T / d ) ) for logistic bandits. Study of psc metrics via block diffeomorphisms and cubical sets.
problem Understanding the concordance of psc metrics.
method Constructing cubical sets and using block Dirac operators.
result Construction of a cubical Kan set and comparison map.
Paper solves learning imperfect-information games with fewer episodes.
problem Learning imperfect-information extensive-form games from bandit feedback.
method Balanced Online Mirror Descent and Balanced Counterfactual Regret Minimization algorithms.
result Achieves near-optimal sample complexity for finding approximate Nash equilibria.
New algorithms achieve optimal regret in sliding window model with limited memory.
problem Experts problem in the sliding window model with limited information.
method 2 queries, polylog(nT) memory, exponential improvement on memory.
result Achieve optimal regret of sqrt(nW)polylog(nT) with 2 queries and polylog(nT) memory.
Estimates KL divergence with fairness considerations for sub-populations.
problem Fairly estimate KL divergence between distributions considering sub-populations.
method Proposes multi-group attribution for KL divergence estimation, derived from multi-calibration.
result Shows multi-group attribution provides better KL divergence estimates conditioned on sub-populations.
Let M = ( M , O M ) \mathcal M= (M,\mathcal O_\mathcal M) M = ( M , O M ) be a smooth supermanifold with connection ∇ \nabla ∇ and Batchelor model O M ≅ Γ Λ E ∗ \mathcal O_\mathcal M\congΓ_{ΛE^\ast} O M ≅ Γ Λ E ∗ . From ( M , ∇ ) (\mathcal M,\nabla) ( M , ∇ ) we construct a connection on the total space of the vector bundle E → M E\to{M} E → M . This reduction of ∇ \nabla ∇ is well-defined independently of …
New bounds on efficiency for conformalized regression methods.
problem Efficiency of conformal prediction in regression models.
method Non-asymptotic bounds on prediction set length for conformalized quantile and median regression.
result Identifies phase transitions in convergence rates across different regimes of miscoverage level.
New method finds balanced clusters in graphs using auxiliary information.
problem Finding balanced clusters in graphs with population-level constraints.
method Proposes individual-level balancing constraint and develops spectral clustering algorithms.
result Establishes first statistical consistency result for constrained spectral clustering.
Hybrid GP/NN framework for operator learning improves performance and enables zero-shot predictions.
problem Approximating mappings between infinite-dimensional function spaces for solving PDEs.
method A hybrid GP/NN framework that approximates the bilinear form of an operator, allowing recovery of the operator.
result Improves performance of neural operators and enables zero-shot predictions.
The paper sets limits for sequential prediction and recursive algorithms using entropy analysis.
problem Fundamental limitations in sequential prediction and recursive algorithms.
method Entropic analysis to investigate underlying relationships of data and noises.
result Derives L p \mathcal{L}_{p} L p bounds quantifiable in conditional entropy. Improved bounds for continuous functions in online learning.
problem Generalizing mistake-bound model to continuous real-valued functions.
method Investigating the class of absolutely continuous functions with bounded derivative, proving bounds on prediction errors.
result Proved that for 1 < p < 2 1 < p < 2 1 < p < 2 with p = 1 + ε p = 1+ε p = 1 + ε , the bound on the worst-case sum of the p t h p^{th} p t h powers of prediction errors is $Θ(ε^{- rac{1}{2}})$ , independent of q q q . The paper proposes multicalibration to improve matching in graphs with imperfect predictors.
problem Finding the best matching in graphs with imperfect predictors.
method Introduces multicalibration as a fairness notion to ensure unbiasedness on protected sets of contexts.
result Constructing a multicalibrated predictor that outperforms standard optimal rules in matching algorithms.
Numerous control and learning problems face the situation where sequences of high-dimensional highly dependent data are available but no or little feedback is provided to the learner, which makes any inference rather challenging. To address this challenge, we formulate the following problem. Given a series of observati…
Paper explores how uncertainty quantification improves Transformer's in-context learning ability.
problem Understanding and quantifying in-context learning ability of Transformers.
method Revisit linear regression tasks with bi-objective prediction (conditional expectation and variance).
result Trained Transformers achieve near Bayes-optimum performance, suggesting use of training distribution.
NeuralChaos efficiently approximates complex stochastic processes.
problem Representing and computing square-integrable predictable processes over time.
method Introduces NeuralChaos, a neural operator architecture for R d \mathbb{R}^{d} R d -valued predictable processes. result NeuralChaos achieves best N N N -term chaoslet approximation rates and is dense in H T 2 ( R d ) \mathcal{H}^2_T(\mathbb{R}^{d}) H T 2 ( R d ) .