Adafactor optimizes neural networks with less memory and similar performance.
problem Memory constraints in adaptive optimization methods.
method Adafactor uses row and column sums of moving averages to estimate per-parameter second moments, reducing memory usage.
result Adafactor achieves similar performance to Adam with minimal auxiliary storage.