New model uses attention for in-context learning of categorical data.
problem Learning from categorical data in context.
method Attention-based network with self-attention and cross-attention layers, using functional gradient descent.
result Model can perform multi-step inference for categorical observations.
ICEE learns new RL tasks in less time with a Transformer model.
problem Efficient in-context policy learning for reinforcement learning.
method In-context Exploration-Exploitation (ICEE) algorithm that optimizes efficiency without explicit Bayesian inference.
result ICEE solves Bayesian optimization problems as efficiently as Gaussian process biased methods but in significantly less time.
This work introduces a method to decompose uncertainty in in-context learning for large language models.
problem Understanding the sources of uncertainty in in-context learning for large language models.
method Variational uncertainty decomposition framework without sampling from latent parameter posterior.
result Quantitative and qualitative validation of decomposed epistemic and aleatoric uncertainties.
Attention-only transformers learn from context via two stages of inference.
problem Learning from corrupted token sequences in minimal transformers.
method Two-stage empirical Bayes interpretation: kernel-weighted posterior mean and particle dynamics.
result Effective denoising without explicit noise schedules, showing posterior-mean recovery under asymptotic conditions.
Transformer learns to infer partial MDPs for efficient in-context adaptation and exploration.
problem Efficiently adapt and explore in-context without gradient-based updates.
method Uses a transformer to learn inference from training tasks, considering hypothesis space of partial models.
result Adaptation speed and exploration-exploitation balance approach those of an exact posterior sampling oracle.
Bayesian analysis reveals epistemic uncertainty as a key diagnostic for delayed generalization in in-context learning.
problem Delayed generalization in in-context learning from few examples.
method Bayesian perspective, modular arithmetic tasks, approximate Bayesian techniques, spectral mechanism analysis.
result Epistemic uncertainty collapses sharply when the model groks, indicating a practical diagnostic of generalization.
Transformers prefer simpler explanations in hierarchical tasks.
problem Navigating tasks with varying complexity levels.
method Well-controlled testbeds based on Markov chains and linear regression.
result Transformers favor the least complex sufficient explanation when presented with simpler data.
Fine-tuning harms in-context learning, but restricting updates to the value matrix improves zero-shot performance.
problem Fine-tuning harms in-context learning, reducing zero-shot performance on unseen tasks.
method Theoretical analysis of linear attention models, identifying conditions for degraded few-shot performance.
result Restricting updates to the value matrix improves zero-shot performance while preserving in-context learning.
TabPFN doesn't outperform GLM and XGBoost for motor insurance pricing.
problem Improving insurance pricing models using Tabular Foundation Models (TFMs).
method Pre-training on synthetic datasets and in-context learning for inference.
result TabPFN does not consistently outperform established baselines, has longer inference times, and is sensitive to training set size.
Transformers converge linearly to optimal models for Gaussian mixtures classification.
problem Theoretical understanding of transformers' in-context classification.
method Gradient descent training of a single-layer transformer for Gaussian mixtures classification.
result Transformers converge linearly to globally optimal models for Gaussian mixtures classification.
Study on neural scaling laws for solving linear systems in-context.
problem Theoretical guarantees for solving linear systems using a linear transformer architecture.
method Neural scaling laws and task diversity for in-domain and out-of-domain generalization.
result Novel notion of task diversity for necessary and sufficient condition of generalization under task shifts.
Transformers learn algorithms for in-context learning with bounds and stability analysis.
problem Understanding and formalizing in-context learning as an algorithm learning problem.
method Formalizing ICL as a multitask learning problem, deriving generalization bounds, and analyzing stability.
result Transformers can implement near-optimal algorithms for classical regression tasks with i.i.d. and dynamic data.
This paper compares amortized methods for Bayesian posterior estimation.
problem Bayesian inference difficulties with iterative routines.
method Amortized in-context Bayesian posterior estimation using transformers and normalizing flows.
result Reverse KL estimator superior for predictive problems.
TTT improves transformer models for in-context learning.
problem Improving transformer models for efficient in-context learning.
method Gradient-based TTT method for linear transformers, with theoretical and empirical analysis.
result TTT significantly reduces the sample size required for in-context learning.
ICON-OCnet solves optimal execution problems with neural networks and few examples.
problem Optimal order execution in markets with unknown price impact.
method Transformer-based neural network architecture (ICON-OCnet) that learns price impact from few examples and applies it to optimal execution strategies.
result ICON-OCnet accurately infers price impact models and retrieves optimal execution strategies for various propagator kernels.
Transformers can implement reinforcement learning algorithms from data without updates.
problem Training reinforcement learning algorithms from data without parameter updates.
method Design a teacher-mimicking training procedure for transformers to implement policy-improvement methods.
result Gradient flow converges to an optimal parameter manifold corresponding to the desired RL update.
Unified Bayesian model explains in-context learning and activation steering in LLMs.
problem Understanding and controlling the behavior of large language models (LLMs) through prompts and activations.
method Developed a Bayesian model to explain and predict the effects of in-context learning and activation steering.
result Unified model predicts distinct phases and sudden shifts in LLM behavior, explaining prior empirical phenomena.
New analysis shows LLMs don't follow Bayesian inference in ICL.
problem Does in-context learning in LLMs follow Bayesian inference?
method Analyzes ICL through the martingale property, a requirement for Bayesian inference.
result Violations of the martingale property show LLMs don't follow Bayesian inference.
The paper proves how Transformers learn from context and generalize well.
problem Understanding how Transformers generalize from diverse tasks.
method Developed a statistical theory for in-context learning, separating risk into Bayes Gap and Posterior Variance.
result The Posterior Variance is task-independent, and the Bayes Gap decreases with more in-context examples.
In-context learning solves PU classification without iterative optimization.
problem Binary classification with only labeled positives and unlabeled samples.
method Pretrained transformer (PUICL) that learns from synthetic PU datasets.
result Outperforms four standard PU learning baselines on 20 benchmarks.
Transformers struggle with time series forecasting, especially in context.
problem Transformers' limitations in time series forecasting, particularly in in-context settings.
method Theoretical analysis of Transformers' limitations through In-Context Learning theory.
result Linear Self-Attention models cannot outperform classical linear models in in-context time series forecasting.
This study uses ICL to efficiently generate robust confidence intervals for noisy regression tasks.
problem Uncertainty quantification for in-context learning in noisy regression tasks.
method Proposes a method based on conformal prediction to construct prediction intervals with guaranteed coverage.
result Conformal prediction with in-context learning (CP with ICL) achieves robust and scalable uncertainty estimates.
SurvivalPFN simplifies survival analysis through amortized Bayesian inference.
problem Selecting appropriate survival analysis methods requires expertise and can be time-consuming.
method SurvivalPFN uses a prior-data fitted network for in-context Bayesian inference.
result SurvivalPFN achieves strong predictive performance across diverse datasets.
Transformers learn to solve various tasks without explicit design.
problem Training models with minimal inductive bias.
method Meta-learning approach to train general-purpose in-context learning algorithms.
result Transformers can be meta-trained to solve a wide range of tasks.
Paper explores privacy in machine learning models using ICL.
problem Privacy in in-context learning models.
method Differentially private pretraining algorithm for linear attention heads.
result Characterizes privacy-accuracy trade-off for ICL in linear regression.
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
Enhances neural processes to learn from multiple related datasets.
problem Improving predictions from datasets with shared similarities.
method Developed the in-context in-context learning pseudo-token TNP (ICICL-TNP) to condition on both sets of datapoints and sets of datasets.
result Demonstrated the importance and effectiveness of in-context in-context learning.
This paper investigates how transformers can learn to generalize to unseen examples in context.
problem Understanding how transformers can generalize to unseen examples in a prompt.
method Gradient descent analysis of one-layer multi-head transformers for in-context learning.
result The training loss for a one-layer multi-head transformer converges linearly to a global minimum, effectively learning ridge regression over basis functions.
Fine-tunes language models with captions for differential equation solving.
problem Overreliance on function data in operator learning.
method Integrates human knowledge through captions and fine-tunes language models.
result Significantly enhanced performance and reduced function data requirements.
The study compares Bayesian and frequentist approaches in deep learning.
problem Comparing Bayesian and frequentist inference in deep learning.
method Conducts a comparative analysis of point and posterior estimators across various settings.
result Amortized point estimators generally outperform posterior inference, though posterior inference remains competitive in some low-dimensional problems.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
Transformer learns context and regularization for ICL in inverse problems.
problem Learning context and effective regularization for transformer-based in-context learning (ICL) in inverse problems.
method Introduced a linear transformer to learn inverse mapping from contextual examples to weight vectors, addressing rank-deficient problems.
result Transformer implicitly learns a prior distribution and effective regularization strategy, outperforming traditional methods.
New study reveals task alignment is key to ICL performance.
problem Understanding how task alignment affects generalization in in-context learning.
method Derived an exact expression for ICL generalization error in high dimensions under task covariance mismatch.
result Identified train-test task alignment as a key determinant of generalization in ICL.
Pipeline combines ETF preprocessing with tabular model for cross-modal inference.
problem Transferability of tabular models across different modalities.
method Fixed comparison object, ETF preprocessing, in-context inference.
result Pipeline is broadly competitive, runs faster, and produces well-calibrated probabilities.
Transformers learn to perform logistic regression in-context.
problem Understanding how transformers learn to perform specific tasks in-context.
method Constructed multi-layer transformers that perform in-context logistic regression through normalized gradient descent.
result Transformers can be trained to perform in-context logistic regression effectively.
Paper explores how uncertainty quantification improves Transformer's in-context learning ability.
problem Understanding and quantifying in-context learning ability of Transformers.
method Revisit linear regression tasks with bi-objective prediction (conditional expectation and variance).
result Trained Transformers achieve near Bayes-optimum performance, suggesting use of training distribution.
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
LLMs learn probability density functions in-context, showing distinct learning trajectories.
problem Density estimation of time series data in LLMs.
method Intensive Principal Component Analysis (InPCA) to visualize and analyze LLMs' learning dynamics.
result LLMs follow similar learning trajectories in a low-dimensional InPCA space, distinct from traditional methods.
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
Transformers learn to play games in-context, proving Nash equilibrium.
problem Understanding in-context game-playing capabilities of pre-trained transformers.
method Theoretical guarantees and constructional results for transformer architecture in multi-agent games.
result Pre-trained transformers can learn Nash equilibrium in-context for two-player zero-sum games.
Transformers forecast time series in-context, improving efficiency and performance.
problem Overfitting and limited performance in time series forecasting.
method Reformulate time series forecasting as input tokens, aligning with in-context learning mechanisms.
result Consistently better performance across various settings (full-data, few-shot, zero-shot).
Transformers can adaptively select and perform various machine learning tasks in context.
problem Transformers' ability to perform multiple machine learning tasks without explicit prompting.
method Statistical theory and efficient implementation of in-context gradient descent, enabling transformers to implement various algorithms and tasks.
result Transformers can implement a broad class of standard machine learning algorithms in context, including algorithm selection.
Continuum transformers learn operators in context via gradient descent.
problem Generalizing transformers to handle infinite-dimensional inputs for in-context learning.
method Gradient descent in an operator RKHS, leveraging generalized representer theorems and gradient flows.
result Operator learned in context is Bayes Optimal Predictor in infinite depth limit.
LLMs perform well in financial sentiment analysis without fine-tuning.
problem Challenges in financial terminology, emotions, and ambiguous expressions.
method In-context learning methods for financial document-sentiment pairs.
result LLMs can generalize in-context demonstrations to new financial documents.
This paper explains how transformers learn from unstructured data in ICL.
problem Understanding how transformers learn from unstructured data in in-context learning.
method A simple transformer model with one or two attention layers and positional encoding is used to study the role of each component in ICL.
result A transformer with two attention layers and a look-ahead attention mask can learn from unstructured data.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.