We introduce a stochastic contextual bandit model where at each time step the environment chooses a distribution over a context set and samples the context from this distribution. The learner observes only the context distribution while the exact context realization remains hidden. This allows for a broad range of appl…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “context distribution”
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
Optimal algorithm for contextual bandits with unknown context distributions.
problem Designing efficient algorithms for contextual bandits with unknown context distributions.
method Cross-learning setting, novel technique for coordinating multiple epochs.
result Nearly tight regret bound of for learning to bid in first-price auctions and sleeping bandits.
Develops a robust learning method for unknown context distributions.
problem Learning from data in different, unknown contexts.
method Focuses on excess risks, constructs distribution sets with statistical coverage.
result Shows robustness in worst-case scenarios without sacrificing nominal performance.
Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data ef…
Bayesian optimization tackles uncertainty in context variables.
problem Sequential decision-making under context distributional uncertainty.
method Wasserstein Distributionally Robust Bayesian Optimization.
result Sublinear regret bounds matching state-of-the-art results.
Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layerscs.LG
The study compares feed-forward and attention layers in language models.
problem Understanding the role of feed-forward and attention layers in language models.
method Empirical and theoretical analysis in a synthetic setting.
result Feed-forward layers learn simple distributional associations, while attention layers focus on in-context reasoning.
Counterfactual data augmentations may not ensure OOD robustness if performed by a context-guessing machine.
problem Deep learning models lack out-of-distribution robustness due to reliance on spurious features.
method Theoretical analysis and demonstration of counterfactual data augmentations performed by a context-guessing machine.
result Counterfactual data augmentations by a context-guessing machine do not lead to robust OOD classifiers.
New method accounts for hidden context in preference learning for RLHF models.
problem Incomplete data with hidden context affects RLHF model outcomes.
method Distributional Preference Learning (DPL) methods estimate hidden context distributions.
result DPL methods reduce RLHF vulnerabilities by accounting for hidden context.
The paper addresses adversarial robustness in in-context learning models.
problem Adversarial distribution shifts threaten the reliability of in-context learning models.
method A distributionally robust meta-learning framework is introduced to provide worst-case performance guarantees under Wasserstein-based distribution shifts.
result Model robustness scales with the square root of its capacity and is penalized by the square of the perturbation magnitude.
Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscs.LG
Transformers learn to solve modular arithmetic tasks by in-context learning and skill composition.
problem Understanding how large language models generalize to unseen tasks in modular arithmetic.
method Pre-training on a set of modular arithmetic tasks and evaluating out-of-distribution performance.
result Transformers require two transformer blocks for out-of-distribution generalization, and deeper models exhibit transient out-of-distribution performance.
In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcs.LG
Study on neural scaling laws for solving linear systems in-context.
problem Theoretical guarantees for solving linear systems using a linear transformer architecture.
method Neural scaling laws and task diversity for in-domain and out-of-domain generalization.
result Novel notion of task diversity for necessary and sufficient condition of generalization under task shifts.
Generalization and adaptation of learned skills to novel situations is a core requirement for intelligent autonomous robots. Although contextual reinforcement learning provides a principled framework for learning and generalization of behaviors across related tasks, it generally relies on uninformed sampling of environ…
New research reveals how the pretraining distribution affects in-context learning in large language models.
problem Understanding how the pretraining distribution influences in-context learning in large language models.
method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.
result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.
Paper proposes linear transformers for efficient in-context learning without context length limitations.
problem Quadratic complexity of softmax transformers limits data processing speed.
method Investigates linear transformers under domain generalization, showing they learn mappings from context distributions to response functions.
result Linear transformers achieve in-context learning with a linear complexity in context length, offering a dimension-independent convergence rate.
Proposes a context-aware approach to deep autoencoder novelty detection.
problem Challenges of static distribution in novelty detection.
method Semi-supervised network architecture with auxiliary labels for contextual information.
result Single model achieves performance of individually trained models on various contexts.
The paper develops a new framework for detecting distributional drifts conditioned on context.
problem Detecting distributional drifts in machine learning systems when context changes.
method Develops a framework using two-sample tests for conditional distributional treatment effects.
result Demonstrates effectiveness for detecting drift in subpopulations of data.
Transformers can approximate posterior predictive distributions through in-context learning.
problem Bayesian prediction tasks, especially beyond point predictions.
method Gradient descent algorithm targeting posterior predictive mean and variance, followed by nonlinear mappings.
result Transformers can implement algorithms to approximate posterior predictive distributions.
CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching
problem Inference of tabular foundation models with large training datasets
method CRUMB (Clustered Retrieval Using Minimised-MMD Batching)
result CRUMB outperforms state-of-the-art context selection strategies on the TabArena benchmark
Paper explores how uncertainty quantification improves Transformer's in-context learning ability.
problem Understanding and quantifying in-context learning ability of Transformers.
method Revisit linear regression tasks with bi-objective prediction (conditional expectation and variance).
result Trained Transformers achieve near Bayes-optimum performance, suggesting use of training distribution.
Enhances neural processes for better context handling.
problem Real-world context sets are complex, requiring richer prior distributions.
method Introduces a graphical model for a richer prior on latent variables, enabling end-to-end optimization.
result Improves function modeling and test-time robustness with mixture and Student-t assumptions.
We study the problem of identifying a probability distribution for some given randomly sampled data in the limit, in the context of algorithmic learning theory as proposed recently by Vinanyi and Chater. We show that there exists a computable partial learner for the computable probability measures, while by Bienvenu, M…
OCEAN infers online task identities from context variables.
problem Online task inference for compositional tasks with context adaptation.
method Variational inference framework OCEAN models global and local context variables in a joint latent space.
result OCEAN provides more effective task inference with sequential context adaptation.
The present paper unifies some aspects concerning the vertical Liouville distributions on the tangent (cotangent) bundle of a Finsler (Cartan) space in the context of generalized geometry. More exactly, we consider the big-tangent manifold associated to a Finsler space and of its -du…
We analyze the ideal gas like models of markets and review the different cases where a `savings' factor changes the nature and shape of the distribution of wealth. These models can produce similar distribution of wealth as observed across varied economies. We present a more realistic model where the saving factor can v…
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
EDRBO optimizes Bayesian optimization with continuous contexts using ensemble models and robust methods.
problem Bayesian optimization with unknown and continuous contextual distributions leads to suboptimal results.
method EDRBO uses ensemble surrogate models and Wasserstein ball ambiguity sets to handle uncertainty and maintain computational tractability.
result EDRBO achieves sublinear cumulative regret guarantees of order .
Global watermark for diffusion language models decouples detection from local contexts.
problem Watermarking in diffusion language models is challenging due to joint sampling of distributions over many unresolved positions.
method Proposes a global vector-valued sketch representation to control watermarking in masked diffusion language models.
result The method decouples detection from local contexts, resulting in an order-agnostic statistic and robustness.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.
Enhances image captioning with novel context combination methods.
problem Improving machine learning for image captioning with structured learning and meaningful interpretation.
method Combines Feature Distribution Composition (FDC), Multiple Role Representation Crossover (MRRC) attention layers, and language decoder.
result Significantly improved image captioning performance (35.3%) and established new standards.
Context Mover's Distance & Barycenters: Optimal Transport of Contexts for Building Representationscs.CL
We present a framework for building unsupervised representations of entities and their compositions, where each entity is viewed as a probability distribution rather than a vector embedding. In particular, this distribution is supported over the contexts which co-occur with the entity and are embedded in a suitable low…
Transformers can learn Markov processes with constant depth, surprising results.
problem Understanding how transformers learn context in Markov processes.
method Empirical study and theoretical analysis of attention-based transformers on Markov data.
result Transformers with constant depth can achieve low test loss on Markov sequences, matching empirical and theoretical findings.
A federated learning algorithm tackles unknown contexts in multi-arm bandits.
problem Learning optimal actions in federated multi-arm bandits with unobserved contexts.
method Elimination-based algorithm for linearly parametrized reward functions.
result Proved regret bound for linearly parametrized reward functions.
Attention-only transformers learn from context via two stages of inference.
problem Learning from corrupted token sequences in minimal transformers.
method Two-stage empirical Bayes interpretation: kernel-weighted posterior mean and particle dynamics.
result Effective denoising without explicit noise schedules, showing posterior-mean recovery under asymptotic conditions.
New techniques save bits in image compression with upsampling.
problem Lack of context dependence in current image compression methods with upsampling.
method Simple, inexpensive techniques exploiting context to predict Laplace distribution parameters.
result Average savings of 0.645 bits per difference, up to 1.489 bits.
CGD improves diffusion models' out-of-distribution generalization.
problem Reliable sampling from high-value regions beyond training data.
method Context-guided diffusion (CGD) using unlabeled data and smoothness constraints.
result Substantial performance gains across various diffusion processes.
Distributivity in algebraic structures appeared in many contexts such as in quasigroup theory, semigroup theory and algebraic knot theory. In this paper we give a survey of distributivity in quasigroup theory and in quandle theory.
Efficiently combines autoregressive and set-based models for joint distributions.
problem Joint distributions over multiple predictions from set-based models.
method Causal autoregressive buffer that caches context and captures dependencies.
result Up to 20x faster joint sampling and density evaluation, up to 7x lower memory usage.
Adaptive algorithms minimize regret in matching markets with contextual arm preferences.
problem Minimizing regret in matching markets with context-dependent player utilities.
method Developed adaptive algorithms for stochastic and adversarial contexts, providing upper and lower bounds.
result Achieved sublinear regret bounds for both stochastic and adversarial contexts.
Scalable method learns context-specific models for hundreds of variables.
problem Learning context-specific models for large numbers of variables.
method Order-based Markov chain Monte-Carlo search with context-specific sparsity assumption.
result Method scales to hundreds of variables and learns accurate models.
Trains word embeddings from music and text data to link music contexts.
problem Varying vocabulary size and musical relevance in word embeddings.
method Combines general text and music-specific data to train word embeddings.
result Trained embeddings better associate music contexts with compositions.
TTT improves transformer models for in-context learning.
problem Improving transformer models for efficient in-context learning.
method Gradient-based TTT method for linear transformers, with theoretical and empirical analysis.
result TTT significantly reduces the sample size required for in-context learning.
Paper solves stochastic contextual linear bandits using linear bandit algorithms.
problem Stochastic contextual linear bandits with unknown context distribution.
method Establishes a reduction framework to convert to linear bandit problems.
result Achieves nearly optimal regret bound of .
Paper solves nonparametric contextual bandit with unbounded contexts.
problem Sequential decision making with unbounded context distributions.
method Two nearest neighbor methods combined with UCB exploration.
result Achieves minimax optimal regret under weak margin condition and light-tailed distributions.
This study uses ICL to efficiently generate robust confidence intervals for noisy regression tasks.
problem Uncertainty quantification for in-context learning in noisy regression tasks.
method Proposes a method based on conformal prediction to construct prediction intervals with guaranteed coverage.
result Conformal prediction with in-context learning (CP with ICL) achieves robust and scalable uncertainty estimates.
Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivestat.ML
Paper provides a mathematical model for transformer ICL out-of-distribution generalization.
problem Understanding when transformer in-context learning can generalize beyond pre-training data.
method Minimal mathematical model of linear regression tasks with low-rank covariance matrices, analyzing distribution shifts as varying angles between subspaces.
result Transformers can generalize to all angle shifts if pre-training tasks are drawn from a union of subspaces, but not from a single Gaussian.
We consider the evolution of scale-free networks according to preferential attachment schemes and show the conditions for which the exponent characterizing the degree distribution is bounded by upper and lower values. Our framework is an agent model, presented in the context of economic networks of trades, which shows …
Transformers recall from long distributions with statistical guarantees.
problem Designing Transformers that can recall from arbitrarily long, distributional contexts.
method Recast associative memory as probability measures, decomposing the task into recall and prediction.
result A shallow measure-theoretic Transformer learns the recall-and-predict map under spectral assumptions.