Averaging recent model checkpoints speeds up training time.
problem Training large vision or language models is time-consuming.
method Average the weights of the k latest checkpoints.
result Speeds up training by dozens of epochs, saving up to 68 GPU hours.
Proposes FMS for more efficient neural network hyperparameter optimization.
problem Efficient hyperparameter optimization for deep learning models.
method Uses logged checkpoints of trained weights to guide hyperparameter selections.
result Proposes Forecasting Model Search (FMS) method.
TopoGeoScore selects robust checkpoints using only source-domain representations.
problem Selecting robust checkpoints without target-domain labels or samples.
method Constructs class-conditional mutual k-nearest-neighbour graphs and extracts three interpretable signals.
result Source representations contain measurable global-local-topological evidence of robustness.
Paper proposes LC-Checkpoint for efficient deep learning model checkpoints.
problem Efficient construction of checkpoints for deep learning models.
method Lossy compression scheme using quantization and priority promotion with Huffman coding.
result LC-Checkpoint achieves up to 28x compression and 5.77x speedup over SCAR.
Modeling human language learning with multi-checkpoint machine translation.
problem Improving machine translation quality for language education.
method Ensemble of multi-checkpoints from a single model, sampling n-best sequences.
result Achieved 37.57 macro F1 score, outperforming baseline.
The paper tackles model selection for unseen tasks by capturing relationships among checkpoints.
problem Deciding which model combinations are likely to be effective for a new task is difficult.
method The paper models the task space as a Gaussian process and identifies representative checkpoints using mutual information and a greedy algorithm.
result Representative checkpoints generalize to new tasks with superior performance.
DSI improves tail-risk estimation in generative models by averaging checkpoints.
problem Generative models' instability in rare adverse scenarios.
method Diachronic Sample Integration (DSI) ensembles generated samples across checkpoints.
result DSI reduces tail-estimation error compared to single-checkpoint baselines.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
SEP uses checkpoints to protect data from training good models.
problem Protecting data from competitors training high-performance models.
method Forming perturbed examples using model checkpoints' gradients to ensure they are always unrecognized.
result SEP significantly reduces model accuracy when trained on perturbed data, demonstrating its effectiveness.
The paper investigates how neural network weights evolve to monitor training progress.
problem Monitoring the training progress of neural networks in a cost-effective manner.
method Investigates the evolution of neural network weights in weight space.
result DNN models evolve on unique, smooth trajectories in weight space that can be used to track training progress.
Signed-permutation coordinate transport improves model alignment across checkpoints.
problem Improper alignment of coordinate-indexed objects across model checkpoints.
method Introduces sign-marginalized Hungarian matching and coordinate-preserving transport.
result Recovering signed-permutation gauge improves coordinate alignment and model performance.
ACA method improves gradient estimation for neural ODEs, reducing error and training time.
problem Inaccurate gradient estimation methods hinder the performance of neural ODEs on benchmark tasks.
method Adaptive Checkpoint Adjoint (ACA) method that applies trajectory checkpointing, deletes redundant components, and supports adaptive solvers.
result ACA reduces error rate by half and training time by half compared to adjoint and naive methods on image classification tasks.
New ODE solvers improve training efficiency and accuracy.
problem Training Neural ODEs requires efficient and accurate gradient calculation.
method Presented algebraically reversible ODE solvers that are time and memory efficient, calculate exact gradients, and are numerically stable.
result Reversible solvers strictly improve upon previous architectures in efficiency and accuracy.
Deleting refusal directions from models leads to systematically more optimistic decisions.
problem The impact of removing refusal directions from models on decision-making outcomes.
method Ablation study using a frozen pipeline of 21,600 weekly equity decisions.
result Ablation of refusal directions makes models more optimistic and justifies themselves more, but also reduces confidence.
Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the G…
Recurrent Neural Networks (RNNs) are extensively used for time-series modeling and prediction. We propose an approach for automatic construction of a binary classifier based on Long Short-Term Memory RNNs (LSTM-RNNs) for detection of a vehicle passage through a checkpoint. As an input to the classifier we use multidime…
Machine learning (ML) training algorithms often possess an inherent self-correcting behavior due to their iterative-convergent nature. Recent systems exploit this property to achieve adaptability and efficiency in unreliable computing environments by relaxing the consistency of execution and allowing calculation errors…
Since DeepMind's AlphaZero, Zero learning quickly became the state-of-the-art method for many board games. It can be improved using a fully convolutional structure (no fully connected layer). Using such an architecture plus global pooling, we can create bots independent of the board size. The training can be made more …
Weibull weight-scale parameter λ evolves during AdamW training, with alignment, injection, and decay forces driving its growth and relaxation.
problem Understanding the evolution of the Weibull weight-scale parameter λ during AdamW training. method Deriving a leading-order three-force decomposition of the squared weight norm from AdamW updates.
result The alignment force dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds.
Extracting a curriculum from a teacher network improves distillation efficiency.
problem Efficiently training a small network using a large teacher network's output.
method Random projection of teacher network's hidden representations to progressively train the student network.
result Extracted curriculum significantly outperforms one-shot distillation and achieves similar performance to progressive distillation.
Optimizes sparse fine-tuning for privacy in neural networks.
problem Performance gap between DP-SGD and non-private fine-tuning.
method Optimization-based approach using private gradient information for selecting trainable weights.
result Our selection method leads to better prediction accuracy compared to existing approaches.
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
This paper describes FBK's submission to the end-to-end English-German speech translation task at IWSLT 2018. Our system relies on a state-of-the-art model based on LSTMs and CNNs, where the CNNs are used to reduce the temporal dimension of the audio input, which is in general much higher than machine translation input…
TracIn measures how training examples affect model predictions.
problem Measuring the influence of training examples on model predictions.
method Tracing changes in loss during training when a specific example is used.
result TracIn provides a scalable method to estimate example influence.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
This paper explores Bayesian Neural Network posteriors, uncovering symmetries and their impact.
problem Understanding the complex posterior distribution of deep Bayesian Neural Networks.
method Investigates optimal approaches for approximating posteriors, analyzes modes, and explores visualizations.
result Uncovered weight-space symmetries and their impact on the posterior, particularly scaling symmetries.
Weibull framework diagnoses transformer weight distributions, revealing distinct patterns across modules.
problem Diagnosing weight distributions in transformer models for better understanding of training dynamics.
method Applied Weibull distribution to diagnose weight magnitude distributions in transformer models, fitting each weight matrix independently.
result Distinct patterns of weight distributions across different transformer modules were identified.
StableLM 2 1.6B is a new language model series with detailed evaluations and performance metrics.
problem Improving language models with smaller sizes and better performance.
method Detailed data and training procedure for StableLM 2 1.6B, including zero- and few-shot benchmarks and multilingual evaluations.
result StableLM 2 1.6B is the state-of-the-art open model under 2B parameters.
Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.
problem Difficulty in collecting diverse and accurate deep learning models through single training.
method Auto-Ensemble collects model checkpoints and uses adaptive learning rate scheduling to ensemble them.
result Ensembled models converge to various local optima, improving performance on few-shot learning.
OpTorch optimizes deep learning for resource-limited environments.
problem Resource constraints in deep learning training.
method Optimized deep learning pipelines in training time and memory.
result Achieved similar accuracy to existing libraries with reduced memory usage.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
ChatGPT snapshots predict future stock returns.
problem Predicting future stock returns using pre-cutoff text.
method Extracted LLM outlook scores from OpenAI snapshots.
result Outlook scores positively correlate with future stock returns.
New framework uses geometry of embeddings to predict robustness.
problem Monitoring robustness in models without OOD labels.
method Constructs graphs from embeddings, measures spectral complexity and curvature.
result Representation geometry predicts robustness reliably.
New optimizer SF-NorMuon matches tuned AdamW across various horizons.
problem Fixed learning-rate schedules in neural network training lead to strong path dependence and costly re-tuning.
method Schedule-Free Spectral Optimization (SF-NorMuon)
result SF-NorMuon outperforms tuned AdamW on 125M and 772M parameter models across different horizons.
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
Resetting from checkpoints improves DNN training with noisy labels.
problem Latent gradient bias induced by noisy labels causes overfitting.
method Stochastic resetting applied to SGD to mitigate latent gradient bias.
result Resetting significantly improves DNN generalization performance.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
ScoreFusion fuses multiple diffusion models to enhance generative modeling of a target population.
problem Enhancing generative modeling of a target population with limited data.
method ScoreFusion uses KL barycenters of auxiliary populations and recasts the learning problem as score matching in denoising diffusion.
result ScoreFusion achieves a dimension-free sample complexity bound in total variation distance.
Spiking neuronal networks are usually simulated with three main simulation schemes: the classical time-driven and event-driven schemes, and the more recent hybrid scheme. All three schemes evolve the state of a neuron through a series of checkpoints: equally spaced in the first scheme and determined neuron-wise by spik…
DoRA improves adaptation efficiency for large models by factoring norms and fusing kernels.
problem High-rank DoRA is computationally expensive and infeasible on common GPUs.
method Factored norms and fused Triton kernels to reduce memory and speed up computation.
result Fused implementation is up to 2.0x faster for inference and 1.9x faster for gradient computation.
This paper introduces a deep learning ensemble forecasting model using Dirichlet process.
problem Forecasting with deep learning ensemble models.
method Infinite mixture model based on Dirichlet process, with decaying learning rate strategy.
result The ensemble model outperforms single benchmark models in prediction accuracy and stability.
We formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies. We introduce Checkmate, a system that solves for optimal rematerialization schedules in reasonable times (under an hour) using off-t…
LayerNorm transformers have dead directions that can be read from their parameters alone.
problem Locating dead directions in LayerNorm transformers
method Using the inverse-scale direction of LayerNorm affine parameters
result Predicted dead direction matches measured bottom singular direction
Estimates model performance based on compute budget and evaluates stability over time.
problem Understanding how model performance evolves with compute budget and stability over time.
method Large-scale observational evaluations, prescriptive scaling laws, smoothed quantile regression, I-optimal sampling algorithm.
result Estimates attainable accuracies and stability of model performance over time.
FLaPS improves scalability and privacy in federated learning.
problem No scalability and security/privacy trade-offs in federated learning.
method Clusters devices, uses differential privacy, iterative shuffling.
result Improved scalability and privacy with comparable performance.
Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble meth…
Paper introduces SALE for better state-action learning in RL.
problem Challenges in representation learning for low-level states in RL.
method Introduces SALE, a novel approach for learning embeddings of state-action interactions.
result TD7 algorithm significantly outperforms existing continuous control algorithms.
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017;…