New HPO frameworks needed for continual learning.
problem No standard HPO for continual learning.
method Comparative study of HPO frameworks.
result No HPO framework consistently outperforms others.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
22 results for “end-of-training”
IMP finds sparse subnetworks that match full networks, revealing geometric insights.
problem Finding sparse subnetworks that match full, overparameterized networks.
method Iterative magnitude pruning (IMP) algorithm, analyzing error landscape geometry.
result IMP masks found at end of training convey useful information for rewound networks.
Computing optimal transport maps between high-dimensional and continuous distributions is a challenging problem in optimal transport (OT). Generative adversarial networks (GANs) are powerful generative models which have been successfully applied to learn maps across high-dimensional domains. However, little is known ab…
Wider neural networks have predominantly positive curvature, aiding optimization.
problem Understanding the convex behavior of deep neural networks with varying layer widths.
method Hessian decomposition and gradient analysis of over-parameterized networks.
result For wide networks, the Hessian is dominated by the positive component G, leading to positive curvature.
Closed-form polynomial approximations replace MLPs in transformers, enabling new interpretability methods.
problem Replacing MLPs with polynomial approximations for transformer models.
method Theoretical derivation of closed-form least-squares approximations of MLPs and GLUs using polynomial functions.
result Polynomial approximations explain over 95% of MLP and GLU outputs' variance, enabling interpretability.
There are classification tasks that take as inputs groups of images rather than single images. In order to address such situations, we introduce a nested multi-instance deep network. The approach is generic in that it is applicable to general data instances, not just images. The network has several convolutional neural…
Study on adversarial robustness in neural networks across initialization and training phases.
problem Understanding adversarial robustness in neural networks during different learning stages.
method Analyzes adversarial robustness in various scenarios of over-parameterized networks with quadratic targets and infinite samples.
result Robustness can worsen when test error improves, and vice versa, revealing new tradeoffs.
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
New method for neural network uncertainty quantification using empirical Neural Tangent Kernel.
problem Accurately quantify uncertainty in neural network predictions.
method Post-hoc, sampling-based approach using gradient-descent on linearized networks.
result Method effectively approximates Gaussian process posterior and outperforms existing methods in efficiency and accuracy.
A new pruning method reduces DNN size and interconnectivity using brain network principles.
problem Over-parameterization in deep neural networks causes memory and hardware cost issues.
method Structural pruning scheme based on Small-World model, trimming network before training.
result Reduced model size by 2.3% on LeNet-5 for MNIST and 9.02% on VGG-16 for CIFAR-10.
Gradient descent learns useful features even in the NTK regime.
problem The ability of neural networks to learn useful features.
method Local convergence analysis of gradient descent with regularization.
result Gradient descent can capture ground-truth directions for feature learning even after the loss threshold is reached.
Private PGB boosts synthetic data quality using GANs and privacy techniques.
problem Differentially private GANs struggle with convergence and poor output quality.
method Combines reweighted samples from GAN training using Private Multiplicative Weights method.
result Improves synthetic data quality across various datasets and tasks.
AIR-Net adapts low-rank regularization dynamically for better image completion.
problem Fixed low-rank regularization limits adaptability to different images.
method AIR-Net uses adaptive and implicit regularization parameterized by a dynamic Laplacian matrix.
result AIR-Net enhances implicit regularization and outperforms fixed methods in non-uniform missing data scenarios.
ShuffleNet is a state-of-the-art light weight convolutional neural network architecture. Its basic operations include group, channel-wise convolution and channel shuffling. However, channel shuffling is manually designed empirically. Mathematically, shuffling is a multiplication by a permutation matrix. In this paper, …
Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networkscs.LG
New scaling laws explain deep learning performance growth.
problem Understanding neural network performance growth.
method Analyzed entire training dynamics of various architectures.
result Identified two dynamical scaling laws.
SAM selects flatter minima late in training, improving generalization.
problem Improving neural network generalization under various settings.
method Sharpness-Aware Minimization (SAM) applied late in training.
result SAM efficiently selects flatter minima late in training, improving generalization.
Optimal learning rate schedules derived for various tasks.
problem Inadequate learning rate schedules in practice compared to theory.
method Refined analysis of learning rate schedules for optimization algorithms.
result Derives new problem-adaptive learning rate schedules.
batchboost: regularization for stabilizing training with resistance to underfitting & overfittingcs.LG
Batchboost stabilizes training by mixing and pairing samples, improving accuracy.
problem Stabilizing training in machine learning, especially avoiding overfitting and underfitting.
method Batchboost pipeline with three stages: pairing, mixing, and feeding. Mixing uses mixup technique.
result Batchboost achieves 0.5-3% better accuracy than mixup on CIFAR-10 & Fashion-MNIST.
Anomalous diffusion in SGD reveals interactions between hyperparameters and Hessian.
problem Understanding the limiting dynamics of SGD in deep neural networks.
method Continuous-time model of SGD as an underdamped Langevin equation, derived for linear regression.
result Anomalous diffusion is explained by modified loss and probability currents in phase space.
New study finds optimal hyperparameter tuning crucial for fair optimizer comparisons.
problem Inadequate hyperparameter tuning and misleading evaluation setups hinder fair comparisons of optimizers.
method Systematic study of ten optimizers across four model scales and data-to-model ratios.
result Optimal hyperparameters for one optimizer may be suboptimal for another, and many claimed speedups are lower than expected.
Neural networks compress uninformative input directions, improving test error.
problem Data lie in a high-dimensional space but labels vary along a lower-dimensional manifold.
method One-hidden layer network trained with gradient descent, analyzing weight evolution and compression.
result Compression factor λ ∼ √p improves test error, with β Feature > β Lazy.
New bounds on trajectory safety in training models with Langevin Dynamics.
problem Bounding the probability of a model's trajectory staying away from a designated failure region.
method Analyzes Langevin dynamics on smooth, strongly convex loss landscapes, introducing shape-free and local relaxation bounds.
result The in-set probability relaxes to the static value after a burn-in time of order d, using only the global spectral gap of the loss.