This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
Improved scaling laws in linear regression using data reuse.
problem Sustainability of neural scaling laws when running out of new data.
method Data reuse in multi-pass stochastic gradient descent (multi-pass SGD) for M M M -dimensional linear models trained on N N N data with sketched features. result Multi-pass SGD achieves a test error of Θ ( M 1 − b + L ( 1 − b ) / a ) Θ(M^{1-b} + L^{(1-b)/a}) Θ ( M 1 − b + L ( 1 − b ) / a ) with L > N L>N L > N , improving scaling laws in data-constrained regimes. Stochastic Gradient Descent can overfit after just a few passes, contrary to initial expectations.
problem Understanding the out-of-sample performance of multi-pass SGD in stochastic convex optimization.
method Analysis of multi-pass SGD in the stochastic convex optimization model.
result Multi-pass SGD can lead to significant overfitting after just a few passes, contrary to initial expectations.
Parameterised actions in reinforcement learning are composed of discrete actions with continuous action-parameters. This provides a framework for solving complex domains that require combining high-level actions with flexible control. The recent P-DQN algorithm extends deep Q-networks to learn over such action spaces. …
Analyzes high-dimensional SGD dynamics using DMFT.
problem Understanding the high-dimensional behavior of multi-pass SGD with small batch sizes.
method Derives DMFT equations for high-dimensional SGD dynamics.
result Proves DMFT equations characterize the asymptotic distribution of SGF parameters.
New algorithm reduces memory usage for multi-pass bandit problems.
problem Memory-efficient multi-pass bandit algorithms for large action spaces.
method Develops a B B B -pass algorithm with O ( 1 ) O(1) O ( 1 ) memory that achieves optimal regret. result Sharp memory-regret trade-off: O ( 1 ) O(1) O ( 1 ) memory suffices for Θ ( T 1 / 2 ) Θ(T^{1/2}) Θ ( T 1/2 ) regret in B B B passes. Two-layer networks learn faster with batch reuse, overcoming information and leap exponents.
problem Limitations of gradient flow and single-pass GD in learning multi-index target functions.
method Multi-pass gradient descent that reuses batches, analyzed using Dynamical Mean-Field Theory.
result Two-time-step overlap with target subspace for non-staircase functions, overcoming information and leap exponents.
FinReflectKG builds a comprehensive financial knowledge graph from SEC filings, improving extraction quality.
problem Lack of large-scale, open-source financial knowledge graph datasets.
method Intelligent document parsing, table-aware chunking, schema-guided iterative extraction, reflection-driven feedback loop.
result Reflection-agent-based mode achieves best balance of efficiency, accuracy, and reliability.
The paper investigates heavy-tailed behavior in offline SGD, showing it approximates power-law tails.
problem Understanding heavy-tailed behavior in offline (multi-pass) SGD with finite data.
method Proves nonasymptotic Wasserstein convergence bounds for offline SGD to online SGD.
result Offline SGD exhibits approximate power-law tails as the number of data points increases.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
SGD outperforms GD in high dimensions via implicit conditioning, revealed by asymptotic analysis.
problem Understanding why SGD outperforms GD in high-dimensional convex problems.
method Asymptotic analysis of multi-pass SGD on high-dimensional convex quadratics, establishing an equivalence to HSGD.
result SGD's efficiency is explained by implicit conditioning, not regularization.
We analyze the learning properties of the stochastic gradient method when multiple passes over the data and mini-batches are allowed. We study how regularization properties are controlled by the step-size, the number of passes and the mini-batch size. In particular, we consider the square loss and show that for a unive…
We study generalization properties of distributed algorithms in the setting of nonparametric regression over a reproducing kernel Hilbert space (RKHS). We first investigate distributed stochastic gradient methods (SGM), with mini-batches and multi-passes over the data. We show that optimal generalization error bounds c…
Improved DP optimization for nonconvex, nonsmooth objectives with reduced sample complexity.
problem Differentially private optimization of nonconvex, nonsmooth objectives.
method Proposes single-pass and multi-pass DP algorithms with improved sample complexity.
result Sample complexity bounds improved by factors of Ω ( d ) Ω(\sqrt{d}) Ω ( d ) and Ω ( d 3 / 4 ) Ω(d^{3/4}) Ω ( d 3/4 ) . Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
A framework to explain decoder-only sequence classification models using intermediate predictions.
problem Explaining predictions of decoder-only sequence classification models.
method Progressive Inference framework with Single Pass-Progressive Inference and Multi Pass-Progressive Inference methods.
result Significantly better attributions compared to prior work on text classification tasks.
Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.
problem Learning a single-index model with quadratic activation using gradient descent.
method Full-batch gradient descent compared to one-pass stochastic gradient descent (SGD) on a correlation loss.
result Full-batch GD requires only n ≃ d n \simeq d n ≃ d samples for strong recovery, while one-pass SGD requires n ≳ d log d n \gtrsim d\log d n ≳ d log d samples. New stability bounds for SGD on nonsmooth convex losses.
problem Understanding stability of SGD on nonsmooth convex losses.
method Sharp upper and lower bounds for SGD and full-batch GD on nonsmooth convex losses.
result SGD can be less stable but still useful for generalization bounds.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
Algorithmic stability is a classical approach to understanding and analysis of the generalization error of learning algorithms. A notable weakness of most stability-based generalization bounds is that they hold only in expectation. Generalization with high probability has been established in a landmark paper of Bousque…
ORFit trains models on streaming data with one pass, minimizing memory and computational costs.
problem Training large models on a stream of data without retraining on previous data.
method Orthogonal Recursive Fitting (ORFit) using orthogonal gradient descent and recursive least-squares.
result ORFit updates parameters orthogonally to past gradients, leading to efficient memory and computational usage.