This paper analyzes multi-pass SGD for least squares, improving generalization bounds.
problem Improving generalization bounds for multi-pass SGD in the least squares problem.
method Develops an instance-dependent excess risk bound for least squares in the interpolation regime.
result SGD performs worse than GD instance-wise but saves computational time.
Optimal learning rates achieved with multi-pass stochastic gradient methods.
problem Improving learning rates in stochastic gradient methods with multiple passes and mini-batches.
method Analysis of learning properties with multiple passes and mini-batches, considering step-size, number of passes, and mini-batch size.
result Optimal finite sample bounds can be achieved by early-stopping and larger step-sizes are allowed with mini-batches.
Improved scaling laws in linear regression using data reuse.
problem Sustainability of neural scaling laws when running out of new data.
method Data reuse in multi-pass stochastic gradient descent (multi-pass SGD) for M M M -dimensional linear models trained on N N N data with sketched features. result Multi-pass SGD achieves a test error of Θ ( M 1 − b + L ( 1 − b ) / a ) Θ(M^{1-b} + L^{(1-b)/a}) Θ ( M 1 − b + L ( 1 − b ) / a ) with L > N L>N L > N , improving scaling laws in data-constrained regimes. Stochastic Gradient Descent can overfit after just a few passes, contrary to initial expectations.
problem Understanding the out-of-sample performance of multi-pass SGD in stochastic convex optimization.
method Analysis of multi-pass SGD in the stochastic convex optimization model.
result Multi-pass SGD can lead to significant overfitting after just a few passes, contrary to initial expectations.
Analyzes high-dimensional SGD dynamics using DMFT.
problem Understanding the high-dimensional behavior of multi-pass SGD with small batch sizes.
method Derives DMFT equations for high-dimensional SGD dynamics.
result Proves DMFT equations characterize the asymptotic distribution of SGF parameters.
MP-DQN improves deep Q-learning for complex action spaces.
problem Learning with discrete actions and continuous parameters in reinforcement learning.
method Multi-pass deep Q-networks (MP-DQN) to handle parameterised actions.
result Significantly outperforms P-DQN and other methods in data efficiency and policy performance.
New algorithm reduces memory usage for multi-pass bandit problems.
problem Memory-efficient multi-pass bandit algorithms for large action spaces.
method Develops a B B B -pass algorithm with O ( 1 ) O(1) O ( 1 ) memory that achieves optimal regret. result Sharp memory-regret trade-off: O ( 1 ) O(1) O ( 1 ) memory suffices for Θ ( T 1 / 2 ) Θ(T^{1/2}) Θ ( T 1/2 ) regret in B B B passes. Two-layer networks learn faster with batch reuse, overcoming information and leap exponents.
problem Limitations of gradient flow and single-pass GD in learning multi-index target functions.
method Multi-pass gradient descent that reuses batches, analyzed using Dynamical Mean-Field Theory.
result Two-time-step overlap with target subspace for non-staircase functions, overcoming information and leap exponents.
FinReflectKG builds a comprehensive financial knowledge graph from SEC filings, improving extraction quality.
problem Lack of large-scale, open-source financial knowledge graph datasets.
method Intelligent document parsing, table-aware chunking, schema-guided iterative extraction, reflection-driven feedback loop.
result Reflection-agent-based mode achieves best balance of efficiency, accuracy, and reliability.
The paper investigates heavy-tailed behavior in offline SGD, showing it approximates power-law tails.
problem Understanding heavy-tailed behavior in offline (multi-pass) SGD with finite data.
method Proves nonasymptotic Wasserstein convergence bounds for offline SGD to online SGD.
result Offline SGD exhibits approximate power-law tails as the number of data points increases.
Study SGD dynamics in high-dimensional models, revealing consistent behavior across different batch sizes and learning rates.
problem Understanding SGD dynamics in high-dimensional multi-index models.
method Asymptotic analysis of SGD, developing mean-field equations and Gaussian diffusion approximations.
result Consistent SGD dynamics across different batch sizes and learning rates, distinct from gradient flow and online SGD.
SGD outperforms GD in high dimensions via implicit conditioning, revealed by asymptotic analysis.
problem Understanding why SGD outperforms GD in high-dimensional convex problems.
method Asymptotic analysis of multi-pass SGD on high-dimensional convex quadratics, establishing an equivalence to HSGD.
result SGD's efficiency is explained by implicit conditioning, not regularization.
Improved DP optimization for nonconvex, nonsmooth objectives with reduced sample complexity.
problem Differentially private optimization of nonconvex, nonsmooth objectives.
method Proposes single-pass and multi-pass DP algorithms with improved sample complexity.
result Sample complexity bounds improved by factors of Ω ( d ) Ω(\sqrt{d}) Ω ( d ) and Ω ( d 3 / 4 ) Ω(d^{3/4}) Ω ( d 3/4 ) . Optimal convergence rates for distributed learning methods are derived.
problem Generalization properties of distributed algorithms in nonparametric regression.
method Distributed stochastic gradient methods (SGM) and spectral-regularization algorithms (SRA).
result Optimal generalization error bounds for distributed SGM are retained with appropriate partition levels.
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
The paper characterizes information complexity for testing binary distributions in the broadcast model.
problem Testing binary distributions with constant advantage in the broadcast model.
method Characterization of information complexity using mixed Hellinger--Jensen--Shannon inequality and optimisation over channels.
result Identifies three parameter regimes with optimal protocols based on different channel models.
A framework to explain decoder-only sequence classification models using intermediate predictions.
problem Explaining predictions of decoder-only sequence classification models.
method Progressive Inference framework with Single Pass-Progressive Inference and Multi Pass-Progressive Inference methods.
result Significantly better attributions compared to prior work on text classification tasks.
Full-batch GD outperforms one-pass SGD in learning a single-index model with quadratic activation.
problem Learning a single-index model with quadratic activation using gradient descent.
method Full-batch gradient descent compared to one-pass stochastic gradient descent (SGD) on a correlation loss.
result Full-batch GD requires only n ≃ d n \simeq d n ≃ d samples for strong recovery, while one-pass SGD requires n ≳ d log d n \gtrsim d\log d n ≳ d log d samples. New stability bounds for SGD on nonsmooth convex losses.
problem Understanding stability of SGD on nonsmooth convex losses.
method Sharp upper and lower bounds for SGD and full-batch GD on nonsmooth convex losses.
result SGD can be less stable but still useful for generalization bounds.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
New bound reduces generalization error for stable algorithms, nearly optimal.
problem Improving generalization bounds for uniformly stable algorithms.
method Developed a new high-probability generalization bound with nearly optimal rate.
result Achieved an estimation error bound of O ( γ log ( n ) log ( n / δ ) + log ( 1 / δ ) / n ) O(γ\log(n)\log(n/δ) + \sqrt{\log(1/δ)/n}) O ( γ log ( n ) log ( n / δ ) + log ( 1/ δ ) / n ) for γ γ γ -uniformly stable algorithms. ORFit trains models on streaming data with one pass, minimizing memory and computational costs.
problem Training large models on a stream of data without retraining on previous data.
method Orthogonal Recursive Fitting (ORFit) using orthogonal gradient descent and recursive least-squares.
result ORFit updates parameters orthogonally to past gradients, leading to efficient memory and computational usage.