The paper explores game-theoretic alignment of LLMs with human preferences, finding limitations and conditions.
problem Aligning LLMs with human preferences using game theory.
method Systematic study of payoff choices in a two-player zero-sum game for desirable alignment properties.
result Impossibility of preference matching in game-theoretic LLM alignment under standard assumptions.
Survey on GNNs' power and limitations.
problem Theoretical limitations of GNNs.
method Comprehensive overview of GNNs and their variants.
result Provably powerful variants of GNNs.
New bounds show limitations of sample-wise information-theoretic generalization.
problem Limitations of sample-wise information-theoretic generalization bounds.
method Analysis of existing bounds and derivation of new bounds.
result No sample-wise information-theoretic bounds exist for expected squared generalization gap.
In this paper, we study the information-theoretic limits of learning the structure of Bayesian networks (BNs), on discrete as well as continuous random variables, from a finite number of samples. We show that the minimum number of samples required by any procedure to recover the correct structure grows as Ω ( m ) Ω(m) Ω ( m ) and $Ω…
The exchange algorithm is studied for its convergence and asymptotic variance.
problem Theoretical limitations of the exchange algorithm in sampling from doubly-intractable distributions.
method Theoretical analysis of the exchange algorithm's convergence speed and asymptotic variance.
result The exchange algorithm converges at a geometric rate and satisfies a Central Limit Theorem.
TAMD prevents degeneracy in finite mixtures, offering strong guarantees but modest practical improvements.
problem Degeneracy in maximum likelihood estimation of finite mixtures.
method Transcendental regularization with analytic barrier functions.
result Strong theoretical guarantees (identifiability, consistency, robustness) but modest practical improvements.
Study shows algorithms benefit from limited target data with many source domains.
problem Adapting to new domains with scarce labeled target data.
method New family of model selection algorithms.
result Beneficial guarantees in scenarios with limited target data.
Paper tackles offline RL with limited target samples using domain adaptation.
problem Limited samples in target dataset degrade offline RL performance.
method Proposes a framework to balance target and source datasets with theoretical guarantees.
result Establishes performance bounds and optimal weight for offline RL.
The study initiates a theoretical analysis of dynamic benchmarking models.
problem Lack of theoretical foundation and empirical studies in dynamic benchmarks.
method Examined two realizations of dynamic benchmarking: sequential and hierarchical dependency models.
result Sequential dynamic benchmarks show initial performance improvement but can stall after three rounds due to label noise.
Study sets limits for detecting a subhypergraph in uniform hypergraphs.
problem Recovering a subhypergraph from a uniform hypergraph with different edge probabilities.
method Information-theoretic analysis for weak and exact recovery.
result Sharp conditions for weak or exact recovery of the subhypergraph.
ResNets approximate log-Gaussian at initialization, improving network performance.
problem Understanding the initialization behavior of deep neural networks like ResNets.
method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.
result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.
Limited liability reduces leveraged risk in loan portfolio management models.
problem The impact of limited liability on risk in loan portfolio management models is not well understood.
method Formulated four models to analyze the effect of limited liability on risk and return in loan portfolio management.
result Including limited liability in loan portfolio management models produces better results in minimizing risk and maximizing expected return.
We solve matrix denoising with both row and column correlations, setting limits and designing optimal methods.
problem Matrix denoising with doubly heteroscedastic noise (both row and column correlations).
method Established information-theoretic and algorithmic limits, designed a novel spectral estimator with optimality guarantees.
result The novel spectral estimator achieves positive correlation with the signal and Bayes-optimal error under one-sided heteroscedasticity.
New framework connects two neural network theories, improving finite-width approximations.
problem Theoretical guarantees for neural network training in general cases.
method Developed a general framework linking mean-field and constant kernel theories.
result Discrete-time MF limit provides better approximation for finite-width nets.
Game-theoretic models predict asset prices in financial markets.
problem Understanding price formation in financial markets with limited liquidity.
method Developed game-theoretic models for many-person and mean-field games, derived analytical formulas, and numerically assessed results.
result The derived price converges to the mean-field counterpart under specific conditions.
We introduce a solvable model of randomly growing systems consisting of many independent subunits. Scaling relations and growth rate distributions in the limit of infinite subunits are analysed theoretically. Various types of scaling properties and distributions reported for growth rates of complex systems in a variety…
Paper improves deep learning models for limit order book data.
problem Deep learning models' performance depends on robust input data representation.
method Identified and modified flaws in existing representations.
result Proposed modifications lead to state-of-the-art performance.
Theoretical study shows AI models can recover from contaminated training data.
problem Data contamination in AI training can degrade model performance.
method Theoretical analysis and experiments on various data types.
result Models converge to true distribution under mild conditions, with rate dependent on real data fraction.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
We study the analytical properties of a one-side order book model in which the flows of limit and market orders are Poisson processes and the distribution of lifetimes of cancelled orders is exponential. Although simplistic, the model provides an analytical tractability that should not be overlooked. Using basic result…
New research limits how well attackers can guess if data points were in a model's training set.
problem Revealing membership of data points in machine learning models.
method Theoretical analysis of statistical limits for membership inference attacks.
result The effectiveness of membership inference attacks is limited by a constant that quantifies data distribution diversity.
Theoretical limits of deep residual networks show consistent covariance structures.
problem Understanding the limits of deep residual networks.
method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.
result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.
Neural networks learn complex functions efficiently near information-theoretic limits.
problem Understanding how neural networks learn high-dimensional features.
method Gradient descent learning of a Gaussian Multi-index model with hidden subspace.
result A standard two-layer neural network can learn the target with optimal sample and time complexity.
A new method for robust matrix completion overcomes limitations of existing approaches.
problem Robust matrix completion from corrupted entries, especially under overparameterization and ill-conditioning.
method Factorization-based iterative algorithm combining Gauss-Newton linearization and outlier removal.
result Theoretical guarantees of exact recovery for suitable assumptions.
In statistical learning for real-world large-scale data problems, one must often resort to "streaming" algorithms which operate sequentially on small batches of data. In this work, we present an analysis of the information-theoretic limits of mini-batch inference in the context of generalized linear models and low-rank…
New bounds improve generalization in learning scenarios.
problem Limitations of existing information-theoretic bounds in SCO problems.
method Sample-conditioned hypothesis stability and neighboring-hypothesis matrix.
result Sharper generalization guarantees in various learning scenarios.
Several recent papers have discussed utilizing Lipschitz constants to limit the susceptibility of neural networks to adversarial examples. We analyze recently proposed methods for computing the Lipschitz constant. We show that the Lipschitz constant may indeed enable adversarially robust neural networks. However, the m…
Paper explores statistical and computational limits of estimating low-rank Gaussian mixtures.
problem Estimating low-rank matrix-variate observations with optimal statistical and computational limits.
method Low-rank Gaussian mixture model (LrMM) and minimax lower bounds.
result Minimax optimality of maximum likelihood estimator and spectral aggregation method.
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
This work sets theoretical limits on meta-learning performance.
problem Understanding the difficulty of adapting machine learning models to real-world data distributions.
method Information-theoretic lower bounds on convergence rates for meta-learning algorithms.
result Theoretical bounds on parameter estimation error for hierarchical Bayesian models of meta-learning.
Recent advances in Reinforcement Learning, grounded on combining classical theoretical results with Deep Learning paradigm, led to breakthroughs in many artificial intelligence tasks and gave birth to Deep Reinforcement Learning (DRL) as a field of research. In this work latest DRL algorithms are reviewed with a focus …
Study reveals limits of detecting local geometry in random graphs.
problem Detecting local geometry in random graphs with hidden communities.
method Introduced model and used information-theoretic and computational limits to investigate detection.
result Detection threshold determined at d = Θ ~ ( k 2 ∨ k 6 / n 3 ) d = \widetildeΘ(k^2 \vee k^6/n^3) d = Θ ( k 2 ∨ k 6 / n 3 ) for fixed p p p . Proposes h-calibration for improving miscalibrated probability outputs of neural networks.
problem Improving reliability of probability outputs from neural networks.
method Probabilistic learning framework for calibration, including a simple yet effective post-hoc algorithm.
result Significantly better performance than traditional methods, validated by experiments.
Examines challenges and proposes new approaches in machine learning theory.
problem Challenges in machine learning as a function approximation and optimization.
method Mathematical analysis of gradient descent, fixed network limitations, and RNNs.
result New insights and mathematical approaches to improve machine learning.
The study examines how limited liability and haircut affect a bank's loan portfolio's liquidity risk.
problem Impact of limited liability and haircut on a bank's loan portfolio's liquidity risk.
method Constructed a novel loan portfolio model with limited liability and haircut constraint, analyzed at three time steps.
result Model with haircut constraint leads to lesser liquidity risk.
CGAN fails to improve deterministic sequence predictions, revealing a theoretical limitation.
problem Improving deterministic sequence predictions with CGAN.
method Developed an adversarial content loss approach.
result CGAN does not improve deterministic sequence predictions.
NTK theory fails to predict practical behavior of large-width neural networks.
problem Theoretical limits of NTK do not match practical neural network architectures.
method Empirical investigation of NTK's applicability to large-width architectures.
result Practically relevant behavior of large-width architectures differs from NTK theory.
Measures time-delay embedding for noisy, sparse data.
problem Applying Takens' embedding theorem to real-world, noisy data.
method Formulated a measure-theoretic generalization of the embedding theorem, using optimal transport.
result Reconstructed full state of dynamical systems from time-lagged partial observations robust to noise and sparsity.
We unify slice sampling and Hamiltonian Monte Carlo (HMC) sampling, demonstrating their connection via the Hamiltonian-Jacobi equation from Hamiltonian mechanics. This insight enables extension of HMC and slice sampling to a broader family of samplers, called Monomial Gamma Samplers (MGS). We provide a theoretical anal…
The paper explores fundamental limits of learning non-hallucinating generative models.
problem Hallucinations in generative models producing invalid outputs.
method Developed a theoretical framework to analyze learnability from a learning-theoretic perspective, incorporating inductive biases.
result Non-hallucinating learning is statistically impossible without additional inductive biases.
OSAMD adapts online to changing distributions with limited labels.
problem Models struggle with continual distribution shifts and expensive labeling in changing environments.
method Online Active Continual Adaptation with OSAMD, an online teacher-student structure and margin-based criterion.
result OSAMD achieves favorable dynamic regret bounds under changing environments with limited labels.
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guara…
New approach prevents model collapse in language generation.
problem Model collapse risk in large language models.
method Introduces a replay adversary to study language generation.
result Replay limits generation in certain ways but not all.
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
Study on reproducibility in optimization with bounds on limits.
problem Limits of reproducibility in noisy or error-prone optimization procedures.
method Defined a quantitative measure of reproducibility and analyzed convex optimization settings.
result Revealed a fundamental trade-off between computation and reproducibility.
Study on gradient clipping in SGD for high-dimensional problems.
problem Understanding and optimizing gradient clipping in high-dimensional machine learning models.
method Theoretical analysis of streaming SGD with gradient clipping in a least squares problem, focusing on large intrinsic dimensionality.
result Developed a deterministic equation to describe the loss evolution in clipped SGD, showing benefits under certain conditions.
Information-theoretic Bayesian optimisation techniques have demonstrated state-of-the-art performance in tackling important global optimisation problems. However, current information-theoretic approaches require many approximations in implementation, introduce often-prohibitive computational overhead and limit the choi…
Information-theoretic bounded rationality describes utility-optimizing decision-makers whose limited information-processing capabilities are formalized by information constraints. One of the consequences of bounded rationality is that resource-limited decision-makers can join together to solve decision-making problems …