Transformers learn a mesa-optimizer to implement in-context learning.
problem Understanding the convergence of autoregressive training to a mesa-optimizer.
method Investigated a one-layer linear causal self-attention model autoregressively trained by gradient flow.
result Proved that autoregressive training converges to a gradient descent step for an OLS problem, validating the mesa-optimizer hypothesis.