DReg boosts large-batch SGD's generalization and convergence.
problem Large-batch SGD struggles with generalization in deep learning.
method DReg replicates a layer to encourage parameter diversity.
result DReg improves generalization and convergence with large-batch SGD.
The paper shows how to efficiently generate large Gaussian process samples with reliability guarantees.
problem Generating large-scale Gaussian process samples efficiently and with reliability.
method Demonstrates scaling data generation to large \(n\) while providing high probability guarantees.
result Efficiently generates large Gaussian process samples with reliability guarantees.
Paper presents an efficient algorithm for learning minimax risk classifiers with large-scale data.
problem Efficient learning of minimax risk classifiers for large-scale data with multiple classes.
method Combination of constraint and column generation for efficient learning.
result 10x speedup for general large-scale data and 100x speedup with many classes.
Paper develops large time series models using pre-trained transformers.
problem Performance bottlenecks in small models on data-scarce scenarios.
method Large-scale pre-training, unified time series format, GPT-style architecture.
result Generative pre-trained Time Series Transformer (Timer) for diverse tasks.
Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.
problem Understanding inductive bias and generalization in large neural networks.
method Dynamical mean field theory applied to large two-layer networks.
result Training dynamics of large networks exhibit a separation of timescales, decoupling feature learning and overfitting.
SNGM improves large-batch training accuracy.
problem Improving generalization in large-batch training.
method Stochastic Normalized Gradient Descent with Momentum.
result SNGM achieves better test accuracy than MSGD and other large-batch methods.
Large learning rates improve generalization, but optimal ranges are narrower than previously thought.
problem Optimizing learning rates for neural network training.
method Detailed exploration of learning rate ranges in a simplified setup, validating findings in a practical setting.
result Optimal learning rate ranges are significantly narrower than previously assumed.
Introduces resemblance structure for large scale geometry.
problem Defining similarity in large scale geometry.
method Axiomatizing the concept of resemblance for subsets of a set.
result Large scale resemblance structures can induce nearness and generalize large scale properties.
For several instances of metric largeness like enlargeability or having hyperspherical universal covers, we construct non-large vector subspaces in the rational homology of finitely generated groups. The functorial properties of this construction imply that the corresponding largeness properties of closed manifolds dep…
Method generates plausible financial stress scenarios using large deviations.
problem Misleading risk management by overlooking or overemphasizing implausible scenarios.
method Exploits large-deviations principle to concentrate risk factors near most likely stress configurations.
result Can generate informative stress scenarios even with limited historical data.
New kernel model scales to large datasets.
problem Challenges in scaling kernel machines to large datasets.
method Introduced EigenPro 3.0, an algorithm based on projected dual preconditioned SGD.
result Ability to scale model and data sizes independently.
Paper proves large deviation principle for stochastic approximations.
problem Asymptotic estimates of learning algorithm deviations.
method Weak convergence approach to large deviations.
result Identifies appropriate scaling sequence and new representation for rate function.
The paper investigates why GNNs struggle to generalize from small to large graphs.
problem Challenges in graph neural networks' ability to generalize across different graph sizes.
method Identified and studied the effect of local structure on size generalization; proposed a novel SSL task.
result GNNs can converge to non-generalizing solutions when there is a discrepancy in local structure.
Noise in SGD helps deep nets generalize better, even with smaller batch sizes.
problem The generalization benefit of using noise in SGD over large batch sizes.
method Carefully designed experiments and rigorous hyperparameter sweeps on various models.
result Small or moderately large batch sizes outperform very large batches on test sets.
In this paper, we extend the Witten-Helffer-Sjöstrand theory from Morse functions to generalized Morse functions. In this case, the spectrum of the Witten deformed Laplacian Δ(t), for large t, can be seperated into the small eigenvalues (which tend to 0 as t→∞), large and very large eigenvalues (both…
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
MiMuon optimizer improves generalization for large models by reducing generalization error.
problem Improving generalization of Muon optimizer for large models.
method Enhanced Muon optimizer using orthogonalization of gradient, proving lower generalization error.
result MiMuon optimizer has a lower generalization error of O(N1) compared to Muon's O(NκT1). Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is anne…
New bounds show large language models can generalize beyond training data.
problem Generalization of large language models beyond training data.
method Compression bound derivation using prediction smoothing and SubLoRA.
result Large language models can discover generalizable regularities.
New bounds for large language models using token properties.
problem Vacuous generalization bounds for large language models.
method Martingale properties and Monarch matrices.
result Non-vacuous generalization bounds for LLMs up to 70B parameters.
Characterizes and analyzes the large scale geometry of big mapping class groups of surfaces.
problem Analyzing the large scale geometry of big mapping class groups of surfaces with a unique maximal end.
method Building on previous work, the paper characterizes and analyzes the large scale geometry of big mapping class groups of surfaces with a unique maximal end.
result Proves that any locally CB big mapping class group is CB generated and gives an explicit criterion for determining which big mapping class groups are CB generated.
IVON optimizes large neural networks, matching or outperforming Adam.
problem The inefficacy of variational learning in large neural networks.
method Improved Variational Online Newton (IVON) optimizer.
result IVON consistently matches or outperforms Adam for large networks.
We use the theory of large deviations to study the pricing of investment-grade tranches of synthetic CDO's. In this paper, we consider a heterogeneous pool of names. Our main tool is a large-deviations analysis which allows us to precisely study the behavior of a large amount of idiosyncratic randomness. Our calculatio…
Study large deviations in random walks on Lie groups.
problem Large deviations in sub-Riemannian random walks.
method Prove large deviation principle for random walks on stratified Lie groups.
result Proved a large deviation principle with a rate function adapted to sub-Riemannian geometry.
The key issue of few-shot learning is learning to generalize. This paper proposes a large margin principle to improve the generalization capacity of metric based methods for few-shot learning. To realize it, we develop a unified framework to learn a more discriminative metric space by augmenting the classification loss…
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
problem Worse generalization performance with large mini-batch sizes in deep learning.
method SP-NGD, a natural gradient descent approach for large-scale deep learning.
result SP-NGD achieves similar generalization performance to first-order methods with accelerated convergence and negligible overhead.
Generative models enhance BO for large batch optimization.
problem Efficiently sampling solutions in high-dimensional, combinatorial design spaces.
method Train generative models to sample solutions proportional to expected utility.
result Generative models can approximate optimal target distributions under certain conditions.
The probability minimizing problem of large losses of portfolio in discrete and continuous time models is studied. This gives a generalization of quantile hedging presented in [3].
In Riemannian manifolds, minimal hypersurfaces with large area exist or have complex structures.
problem Existence of minimal hypersurfaces with arbitrarily large area in closed Riemannian manifolds.
method Almgren-Pitts min-max theory, Marques-Neves ideas, Song's proof of Yau's conjecture, Zhou's resolution of generic multiplicity-one conjecture.
result Existence of minimal hypersurfaces with arbitrarily large area or pathological Cantor set structures in certain manifolds.
We present a theoretical algorithm which, given any finite presentation of a group as input, will terminate with answer yes if and only if the group is large. We then implement a practical version of this algorithm using Magma and apply it to a range of presentations. Our main focus is on 2-generator 1-relator presenta…
Large learning rates improve neural network generalization, study shows.
problem Understanding why large learning rates lead to better neural network generalization.
method Visual analysis of training and testing loss landscapes, introduction of a nonlinear model.
result Extended phase with large learning rates leads to near-optimal generalization.
The study finds infinitely many hyperbolic 3-manifolds with large rank and generalized torsion elements.
problem Finding hyperbolic 3-manifold groups with large rank and generalized torsion elements.
method Constructing specific hyperbolic 3-manifolds with given properties.
result Infinitely many hyperbolic 3-manifolds with generalized torsion elements of arbitrarily large order.
Study large deviations and speed of random walks in hyperbolic spaces.
problem Understanding the speed of random walks in hyperbolic spaces.
method Large deviations analysis for random walks with a non-elementary semi-group.
result Established large deviations results for random walk distances.
Efficient exploration improves large language model performance with fewer queries.
problem Improving large language model performance with fewer human feedback queries.
method Sequentially generates queries, fits a reward model to feedback, uses double Thompson sampling with epistemic neural network uncertainty.
result Efficient exploration enables high performance with far fewer queries.
Study large deviations rates for SGD with strongly convex functions.
problem High probability metrics with SGD.
method Large deviations theory, generic gradient noise, strongly convex functions.
result Upper large deviations bound for SGD with strongly convex functions.
In this article we construct a piecewise Euclidean, non-positively curved 2-complex for the 3-generator Artin groups of large type. As a consequence we show that these groups are biautomatic. A slight modification of the proof shows that many other Artin groups are also biautomatic. The general question (whether all Ar…
Derives asymptotic generalization error for large-margin classifiers.
problem Understanding the generalization error of large-margin classifiers.
method Statistical physics replica method for deriving asymptotic expression.
result Establishes phase transition boundary for class separability.
The study explores generalized divergences and exponential families with a focus on sufficient conditions and laws of large numbers.
problem Generalization of Kullback-Leibler divergence and exponential families.
method Investigation of (h,τ)-divergence and (h,τ)-exponential families, definition of (h,τ)-dependence, proof of law of large numbers. result Sufficient condition for (h,τ)-divergence to induce Hessian structure on (h,τ)-exponential family, proof of law of large numbers. Method reveals hidden token embeddings of large language models.
problem Understanding hidden token embeddings in large language models.
method Structured prompts to expose token input embeddings up to homeomorphism.
result Mathematical proof for generic LLMs shows effectiveness of method.
Large deviations theory applied to policy gradient methods.
problem Understanding convergence of policy gradient methods in reinforcement learning.
method Large deviation rate function and contraction principle from large deviations theory.
result Convergence properties of policy gradient methods can be extended to various policy parametrizations.
Study examines large deviations in random walks on hyperbolic spaces.
problem Large deviations in random walks on Gromov-hyperbolic spaces.
method Established large deviations results for distance and translation length of random walks.
result Deduced a special case of a conjecture regarding spectral radii of random matrix products.
Large learning rates lead to optimal generalization if chosen carefully.
problem Understanding the optimal range of large learning rates for neural network training.
method Empirical study focusing on two questions: optimal initial LR range and differences between models trained with different LRs.
result Optimal initial learning rates slightly above the convergence threshold lead to optimal results after fine-tuning with a small LR or weight averaging.
Model inference, such as model comparison, model checking, and model selection, is an important part of model development. Leave-one-out cross-validation (LOO) is a general approach for assessing the generalizability of a model, but unfortunately, LOO does not scale well to large datasets. We propose a combination of u…
PAVI speeds up Bayesian inference for large datasets.
problem Challenges in Bayesian inference for large population studies.
method Plate-amortized Variational Inference (PAVI) that shares parameterization across i.i.d. variables.
result Significant speedup in training variational distributions, orders of magnitude faster.
Deep learning models can overfit noisy data without losing generalization.
problem Understanding the generalization of deep learning models in noisy data.
method Empirical investigation of epoch-wise double descent in fully connected neural networks trained on CIFAR-10 with 30% label noise.
result The model achieves strong re-generalization on test data after overfitting noisy training data, corresponding to a 'benign overfitting' state.
A new learning method uses data to learn from large model sets.
problem Learning with large sets of candidate models where uniform convergence is hard.
method Data-dependent learning that incorporates empirical data less reliant on prior assumptions.
result Demonstrates improved generalization in various learning assumptions.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
It is well known that, for most datasets, the use of large-size minibatches for Stochastic Gradient Descent (SGD) typically leads to slow convergence and poor generalization. On the other hand, large minibatches are of great practical interest as they allow for a better exploitation of modern GPUs. Previous literature …