New study reveals task alignment is key to ICL performance.
problem Understanding how task alignment affects generalization in in-context learning.
method Derived an exact expression for ICL generalization error in high dimensions under task covariance mismatch.
result Identified train-test task alignment as a key determinant of generalization in ICL.
This study uses ICL to efficiently generate robust confidence intervals for noisy regression tasks.
problem Uncertainty quantification for in-context learning in noisy regression tasks.
method Proposes a method based on conformal prediction to construct prediction intervals with guaranteed coverage.
result Conformal prediction with in-context learning (CP with ICL) achieves robust and scalable uncertainty estimates.
This study examines how large language models learn in-context and provides insights into their performance and architecture.
problem Understanding how large language models learn in-context and their performance metrics.
method Bayesian model averaging, parameterization, and statistical analysis of transformer architecture.
result Transformer architecture enables in-context learning through attention mechanisms and fine-grained statistical analysis.
We show how to convert ICL in linearized transformers into model weights.
problem Making in-context learning interpretable and permanent in large language models.
method Demonstrates equivalence between ICL and bias terms in linearized transformers, and develops ICLCA for exact conversion.
result Exact conversion of in-context learning into model weights is possible for linearized transformers.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
Transformers can adaptively select and perform various machine learning tasks in context.
problem Transformers' ability to perform multiple machine learning tasks without explicit prompting.
method Statistical theory and efficient implementation of in-context gradient descent, enabling transformers to implement various algorithms and tasks.
result Transformers can implement a broad class of standard machine learning algorithms in context, including algorithm selection.
This study explores how examples influence ICL in LLMs.
problem Understanding how examples impact in-context learning in large language models.
method Theoretical study with a probabilistic model extending from Gaussian mixture model.
result The impact of pre-training knowledge and examples on ICL prediction accuracy.
Training task diversity improves ICL with linear attention.
problem Understanding the impact of training task diversity on in-context learning.
method Modeling training task vectors as a mixture of low-rank Gaussians.
result Our model explains why training with task diversity shortens the ICL plateau and achieves out-of-distribution generalization.
LLMs learn new tasks from unstructured data, but it depends on word co-occurrence and positional information.
problem Understanding how LLMs can learn new tasks from unstructured data without explicit training.
method Examined the capabilities of LLMs trained on unstructured data, focusing on sequence model requirements and training data structure.
result Many ICL capabilities can emerge from word co-occurrence in unstructured data, but positional information is crucial for certain tasks.
Linear Transformer Block combines MLP and linear attention for near-optimal ICL in linear regression.
problem Achieving near-optimal in-context learning (ICL) risk for linear regression with a Gaussian prior.
method Combines linear attention and MLP components in a Linear Transformer Block (LTB). Establishes correspondence with one-step gradient descent estimators (GDext−β). result LTB achieves nearly Bayes optimal ICL risk for linear regression with a Gaussian prior.
This paper explains how transformers learn from unstructured data in ICL.
problem Understanding how transformers learn from unstructured data in in-context learning.
method A simple transformer model with one or two attention layers and positional encoding is used to study the role of each component in ICL.
result A transformer with two attention layers and a look-ahead attention mask can learn from unstructured data.
Paper provides a mathematical model for transformer ICL out-of-distribution generalization.
problem Understanding when transformer in-context learning can generalize beyond pre-training data.
method Minimal mathematical model of linear regression tasks with low-rank covariance matrices, analyzing distribution shifts as varying angles between subspaces.
result Transformers can generalize to all angle shifts if pre-training tasks are drawn from a union of subspaces, but not from a single Gaussian.
Transformers fine-tuned on synthetic data boost tabular data classification performance.
problem Improving tabular data classification accuracy.
method Fine-tuning ICL-transformers on synthetic datasets with complex decision boundaries.
result Fine-tuned ICL-transformers outperform regular neural networks on real-world datasets.
Paper estimates AI hallucinations in conditional generation tasks.
problem Estimating the frequency of AI-generated incorrect responses.
method Developed a method to estimate hallucination probability from generated responses and log probabilities.
result Method accurately estimates hallucination rate in natural language and synthetic tasks.
Paper explores privacy in machine learning models using ICL.
problem Privacy in in-context learning models.
method Differentially private pretraining algorithm for linear attention heads.
result Characterizes privacy-accuracy trade-off for ICL in linear regression.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
New insights into how depth and width affect in-context learning in deep models.
problem Understanding how various resources impact in-context learning in deep models.
method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.
result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.
New research reveals how the pretraining distribution affects in-context learning in large language models.
problem Understanding how the pretraining distribution influences in-context learning in large language models.
method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.
result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.
AdaDPSyn generates synthetic examples to protect private data in ICL.
problem Protecting private data in in-context learning with large language models.
method Data-adaptive differentially private algorithm that dynamically adjusts noise level based on data properties.
result AdaDPSyn outperforms existing methods in preserving high ICL accuracy while maintaining differential privacy.
Random Transformers behave like polynomial models in ICL with asymptotic growth.
problem Understanding in-context learning capabilities of pretrained Transformers.
method Asymptotic analysis of a random Transformer with a fixed first layer and a trained second layer, considering growth in context length, input dimension, hidden dimension, and training parameters.
result The random Transformer's ICL error is equivalent to a finite-degree Hermite polynomial model.
Generative AI can solve in-context learning problems using a martingale perspective.
problem Estimating when a conditional generative model can solve an in-context learning problem.
method Bayesian interpretation, ancestral sampling, generative predictive p-value.
result Developed a method to assess the suitability of CGMs for ICL problems using generative predictive p-values.
Transformers with MLP heads outperform linear baselines in in-context learning.
problem Understanding in-context learning in Transformers with nonlinear MLP heads.
method Analyzing a model with two-layer MLPs trained via gradient steps and fully optimized, under high-dimensional asymptotics.
result Nonlinear MLPs enhance ICL performance, particularly on nonlinear tasks.
Transformers struggle to learn Markovian dynamics, showing NP-hard optimization challenges.
problem Understanding transformers' limitations in learning Markovian dynamical functions.
method Investigated through a structured ICL setup, analyzing loss landscapes and parameter optimization.
result Recovering optimal transformer parameters for Markovian functions is NP-hard.
Paper analyzes risk bounds for in-context learning in multiclass classification.
problem Risk bounds for in-context learning in multiclass classification.
method Formalizes tasks as sequences of labeled examples and queries, estimates conditional class probabilities, establishes oracle inequality for KL divergence.
result ICL achieves minimax optimal rate for conditional probability estimation.
New analysis shows LLMs don't follow Bayesian inference in ICL.
problem Does in-context learning in LLMs follow Bayesian inference?
method Analyzes ICL through the martingale property, a requirement for Bayesian inference.
result Violations of the martingale property show LLMs don't follow Bayesian inference.
The study explains how transformer components enable in-context learning.
problem Understanding how transformer components contribute to in-context learning.
method Analyzed a two-attention-layer transformer model trained on Markov chain data.
result Gradient flow converges to a limiting model with a copier, selector, and classifier mechanism.
Transformers learn to generalize out-of-distribution with diverse pretraining tasks.
problem Conditions for pretrained transformers to generalize out-of-distribution.
method Empirical study of task diversity and pretraining distribution.
result As task diversity increases, transformers transition from specialized to generalized solutions.
Transformers converge linearly to optimal models for Gaussian mixtures classification.
problem Theoretical understanding of transformers' in-context classification.
method Gradient descent training of a single-layer transformer for Gaussian mixtures classification.
result Transformers converge linearly to globally optimal models for Gaussian mixtures classification.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
Transformers learn low-dimensional target functions efficiently in-context.
problem Efficiently learning nonlinear target functions in-context using transformers.
method Nonlinear MLP layer in transformers optimized by gradient descent, focusing on single-index target functions.
result Transformers can learn target functions with low-dimensional structures efficiently in-context.
Transformers learn to perform logistic regression in-context.
problem Understanding how transformers learn to perform specific tasks in-context.
method Constructed multi-layer transformers that perform in-context logistic regression through normalized gradient descent.
result Transformers can be trained to perform in-context logistic regression effectively.
This work shows how transformers use multi-concept word semantics for efficient in-context learning.
problem Understanding the connection between transformer-based LLMs' multi-concept semantic representation and their innovative in-context learning abilities.
method A concept-based low-noise sparse coding prompt model, leveraging advanced techniques to analyze the exponential convergence of 0-1 loss over non-convex training dynamics.
result Transformers leverage multi-concept word semantics to enable powerful and excellent out-of-distribution in-context learning.
Transformers learn new tasks from few examples via optimal approximation.
problem Learning new tasks from limited examples using large language models.
method Developed approximation and generalization error bounds for transformers trained on nonparametric regression tasks.
result Transformers achieve minimax optimal estimation risk in context.
SC unifies ICL calibration methods and improves LLM performance.
problem Systematic biases in LLM predictions leading to unstable performance.
method Supervised Calibration (SC) learns optimal affine transformations in logit space.
result SC delivers state-of-the-art performance across multiple datasets.
The paper addresses adversarial robustness in in-context learning models.
problem Adversarial distribution shifts threaten the reliability of in-context learning models.
method A distributionally robust meta-learning framework is introduced to provide worst-case performance guarantees under Wasserstein-based distribution shifts.
result Model robustness scales with the square root of its capacity and is penalized by the square of the perturbation magnitude.
The paper proves how Transformers learn from context and generalize well.
problem Understanding how Transformers generalize from diverse tasks.
method Developed a statistical theory for in-context learning, separating risk into Bayes Gap and Posterior Variance.
result The Posterior Variance is task-independent, and the Bayes Gap decreases with more in-context examples.
TabDPT combines ICL and self-supervised learning to train tabular models on real data.
problem Slow generalization of tabular foundation models to unseen datasets.
method In-Context Learning (ICL) combined with self-supervised learning, using real data for pre-training.
result TabDPT achieves strong performance on regression and classification benchmarks.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
DEFINED uses decision feedback ICL to detect symbols with minimal pilot data.
problem Limited pilot data in wireless receivers.
method In-context learning with decision feedback mechanism.
result Small Transformer trained with DEFINED achieves significant performance improvements.
As systems are getting more autonomous with the development of artificial intelligence, it is important to discover the causal knowledge from observational sensory inputs. By encoding a series of cause-effect relations between events, causal networks can facilitate the prediction of effects from a given action and anal…
The study provides precise asymptotic theory for in-context learning by Transformers.
problem Understanding the sample complexity, pretraining task diversity, and context length for successful in-context learning.
method An exactly solvable model of linear regression task by linear attention, deriving sharp asymptotics.
result Double-descent learning curve with increasing pretraining examples, phase transition between low and high task diversity regimes.
Improves many-shot learning by optimizing and generating influential examples.
problem Limited performance in many-shot in-context learning (ICL).
method Iterative optimization and generation of influential examples.
result Significant improvements across various tasks using BRIDGE.
Transformer learns context and regularization for ICL in inverse problems.
problem Learning context and effective regularization for transformer-based in-context learning (ICL) in inverse problems.
method Introduced a linear transformer to learn inverse mapping from contextual examples to weight vectors, addressing rank-deficient problems.
result Transformer implicitly learns a prior distribution and effective regularization strategy, outperforming traditional methods.
This work analyzes how transformers learn common linear regression tasks.
problem Understanding how in-context learning operates in real-world applications with common task structures.
method Analyzing a linear attention model trained on low-rank regression tasks.
result Statistical fluctuations in finite pre-training data induce an implicit regularization, leading to a sharp phase transition in generalization error.
New framework analyzes LLM personalization trade-offs under congestion.
problem Tension between personalization and resource sharing in LLMs.
method Developed a statistical-economic framework to model user incentives.
result Congestion can flip rankings of SFT and ICL, and offers both methods never hurt profits.
Paper explores how uncertainty quantification improves Transformer's in-context learning ability.
problem Understanding and quantifying in-context learning ability of Transformers.
method Revisit linear regression tasks with bi-objective prediction (conditional expectation and variance).
result Trained Transformers achieve near Bayes-optimum performance, suggesting use of training distribution.