Averaging recent model checkpoints speeds up training time.
problem Training large vision or language models is time-consuming.
method Average the weights of the k latest checkpoints.
result Speeds up training by dozens of epochs, saving up to 68 GPU hours.
DSI improves tail-risk estimation in generative models by averaging checkpoints.
problem Generative models' instability in rare adverse scenarios.
method Diachronic Sample Integration (DSI) ensembles generated samples across checkpoints.
result DSI reduces tail-estimation error compared to single-checkpoint baselines.
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
Paper proposes LC-Checkpoint for efficient deep learning model checkpoints.
problem Efficient construction of checkpoints for deep learning models.
method Lossy compression scheme using quantization and priority promotion with Huffman coding.
result LC-Checkpoint achieves up to 28x compression and 5.77x speedup over SCAR.
Modeling human language learning with multi-checkpoint machine translation.
problem Improving machine translation quality for language education.
method Ensemble of multi-checkpoints from a single model, sampling n-best sequences.
result Achieved 37.57 macro F1 score, outperforming baseline.
The paper tackles model selection for unseen tasks by capturing relationships among checkpoints.
problem Deciding which model combinations are likely to be effective for a new task is difficult.
method The paper models the task space as a Gaussian process and identifies representative checkpoints using mutual information and a greedy algorithm.
result Representative checkpoints generalize to new tasks with superior performance.
TopoGeoScore selects robust checkpoints using only source-domain representations.
problem Selecting robust checkpoints without target-domain labels or samples.
method Constructs class-conditional mutual k-nearest-neighbour graphs and extracts three interpretable signals.
result Source representations contain measurable global-local-topological evidence of robustness.
A new algorithm reduces memory usage for deep learning models.
problem Training deep learning models requires significant memory.
method Dynamic Tensor Rematerialization (DTR) is a greedy online algorithm that dynamically plans recomputations.
result DTR achieves comparable performance to optimal static checkpointing with only a small memory budget.
SEP uses checkpoints to protect data from training good models.
problem Protecting data from competitors training high-performance models.
method Forming perturbed examples using model checkpoints' gradients to ensure they are always unrecognized.
result SEP significantly reduces model accuracy when trained on perturbed data, demonstrating its effectiveness.
Signed-permutation coordinate transport improves model alignment across checkpoints.
problem Improper alignment of coordinate-indexed objects across model checkpoints.
method Introduces sign-marginalized Hungarian matching and coordinate-preserving transport.
result Recovering signed-permutation gauge improves coordinate alignment and model performance.
ACA method improves gradient estimation for neural ODEs, reducing error and training time.
problem Inaccurate gradient estimation methods hinder the performance of neural ODEs on benchmark tasks.
method Adaptive Checkpoint Adjoint (ACA) method that applies trajectory checkpointing, deletes redundant components, and supports adaptive solvers.
result ACA reduces error rate by half and training time by half compared to adjoint and naive methods on image classification tasks.
New ODE solvers improve training efficiency and accuracy.
problem Training Neural ODEs requires efficient and accurate gradient calculation.
method Presented algebraically reversible ODE solvers that are time and memory efficient, calculate exact gradients, and are numerically stable.
result Reversible solvers strictly improve upon previous architectures in efficiency and accuracy.
Proposes FMS for more efficient neural network hyperparameter optimization.
problem Efficient hyperparameter optimization for deep learning models.
method Uses logged checkpoints of trained weights to guide hyperparameter selections.
result Proposes Forecasting Model Search (FMS) method.
Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the G…
Recurrent Neural Networks (RNNs) are extensively used for time-series modeling and prediction. We propose an approach for automatic construction of a binary classifier based on Long Short-Term Memory RNNs (LSTM-RNNs) for detection of a vehicle passage through a checkpoint. As an input to the classifier we use multidime…
Machine learning (ML) training algorithms often possess an inherent self-correcting behavior due to their iterative-convergent nature. Recent systems exploit this property to achieve adaptability and efficiency in unreliable computing environments by relaxing the consistency of execution and allowing calculation errors…
Study shows training duration impacts model merging quality, suggesting joint selection of duration and method.
problem Impact of expert training duration on model merging quality for large language models (LLMs).
method Systematically fine-tuned experts on five domains across three model sizes, evaluating five merging methods at each duration.
result Training duration affects merging quality, with simple averaging degrading sharply and sparsification-based methods performing well past the validation optimum.
Since DeepMind's AlphaZero, Zero learning quickly became the state-of-the-art method for many board games. It can be improved using a fully convolutional structure (no fully connected layer). Using such an architecture plus global pooling, we can create bots independent of the board size. The training can be made more …
Study shows training duration affects model merging quality, suggesting joint selection of duration and method.
problem Impact of expert training duration on model merging quality for large language models (LLMs).
method Systematically fine-tuned experts on five domains across three model sizes, evaluated five merging methods at each duration.
result Training duration and merging method should be chosen jointly, not independently.
Artificial neural networks have been successfully applied to a variety of machine learning tasks, including image recognition, semantic segmentation, and machine translation. However, few studies fully investigated ensembles of artificial neural networks. In this work, we investigated multiple widely used ensemble meth…
This paper describes FBK's submission to the end-to-end English-German speech translation task at IWSLT 2018. Our system relies on a state-of-the-art model based on LSTMs and CNNs, where the CNNs are used to reduce the temporal dimension of the audio input, which is in general much higher than machine translation input…
Paper introduces SALE for better state-action learning in RL.
problem Challenges in representation learning for low-level states in RL.
method Introduces SALE, a novel approach for learning embeddings of state-action interactions.
result TD7 algorithm significantly outperforms existing continuous control algorithms.
Extracting a curriculum from a teacher network improves distillation efficiency.
problem Efficiently training a small network using a large teacher network's output.
method Random projection of teacher network's hidden representations to progressively train the student network.
result Extracted curriculum significantly outperforms one-shot distillation and achieves similar performance to progressive distillation.
We introduce a method called TracIn that computes the influence of a training example on a prediction made by the model. The idea is to trace how the loss on the test point changes during the training process whenever the training example of interest was utilized. We provide a scalable implementation of TracIn via: (a)…
Post-hoc transforms can reverse model performance trends, especially in noisy settings.
problem Post-hoc transforms can reverse model performance trends, especially in noisy settings.
method Empirical study and analysis of post-hoc transforms like temperature scaling, ensembling, and SWA.
result Post-hoc reversal can prevent double descent and mitigate mismatches between test loss and test error.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
Auto-Ensemble automates deep learning model ensembling with adaptive learning rate scheduling.
problem Difficulty in collecting diverse and accurate deep learning models through single training.
method Auto-Ensemble collects model checkpoints and uses adaptive learning rate scheduling to ensemble them.
result Ensembled models converge to various local optima, improving performance on few-shot learning.
WSD schedule improves model training efficiency by adapting learning rates dynamically.
problem Fixed compute budgets limit training efficiency of language models.
method Introduces a WSD schedule that uses a constant learning rate followed by a rapid decay phase.
result WSD schedule generates a non-traditional loss curve with stable and decay phases.
ChatGPT snapshots predict future stock returns.
problem Predicting future stock returns using pre-cutoff text.
method Extracted LLM outlook scores from OpenAI snapshots.
result Outlook scores positively correlate with future stock returns.
New framework uses geometry of embeddings to predict robustness.
problem Monitoring robustness in models without OOD labels.
method Constructs graphs from embeddings, measures spectral complexity and curvature.
result Representation geometry predicts robustness reliably.
Resetting from checkpoints improves DNN training with noisy labels.
problem Latent gradient bias induced by noisy labels causes overfitting.
method Stochastic resetting applied to SGD to mitigate latent gradient bias.
result Resetting significantly improves DNN generalization performance.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
Spiking neuronal networks are usually simulated with three main simulation schemes: the classical time-driven and event-driven schemes, and the more recent hybrid scheme. All three schemes evolve the state of a neuron through a series of checkpoints: equally spaced in the first scheme and determined neuron-wise by spik…
LUNA improves linear attention for long sequences without sacrificing accuracy.
problem Quadratic computational cost of softmax attention in long-sequence domains.
method LUNA learns a learnable kernel feature map to reduce attention cost to linear while maintaining accuracy.
result LUNA achieves state-of-the-art performance on the LRA and excels at post-hoc conversion.
FLaPS improves scalability and privacy in federated learning.
problem No scalability and security/privacy trade-offs in federated learning.
method Clusters devices, uses differential privacy, iterative shuffling.
result Improved scalability and privacy with comparable performance.
We formalize the problem of trading-off DNN training time and memory requirements as the tensor rematerialization optimization problem, a generalization of prior checkpointing strategies. We introduce Checkmate, a system that solves for optimal rematerialization schedules in reasonable times (under an hour) using off-t…
LayerNorm transformers have dead directions that can be read from their parameters alone.
problem Locating dead directions in LayerNorm transformers
method Using the inverse-scale direction of LayerNorm affine parameters
result Predicted dead direction matches measured bottom singular direction
Deleting refusal directions from models leads to systematically more optimistic decisions.
problem The impact of removing refusal directions from models on decision-making outcomes.
method Ablation study using a frozen pipeline of 21,600 weekly equity decisions.
result Ablation of refusal directions makes models more optimistic and justifies themselves more, but also reduces confidence.
RG-TTA adapts neural forecasters to streaming time series shifts by modulating adaptation intensity.
problem Adapting neural forecasters to distribution shifts in streaming time series data.
method RG-TTA uses a meta-controller that continuously modulates adaptation intensity based on distributional similarity.
result RG-TTA achieves the lowest MSE in 156 of 224 seed-averaged experiments, reducing MSE by 5.7% vs TTA.
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017;…
New framework compresses and recovers scientific data efficiently.
problem Efficiently managing and recovering from large scientific datasets.
method Grounded in learning exponential families, preserves uncertainty and supports trade-offs.
result Preserves physical features and quantities of interest in compressed representations.
Training-free looped transformers improve model performance without additional training.
problem Improving model performance without additional training or fine-tuning.
method A lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning.
result Our method improves model performance across various model families.
The paper investigates how neural network weights evolve to monitor training progress.
problem Monitoring the training progress of neural networks in a cost-effective manner.
method Investigates the evolution of neural network weights in weight space.
result DNN models evolve on unique, smooth trajectories in weight space that can be used to track training progress.
StableLM 2 1.6B is a new language model series with detailed evaluations and performance metrics.
problem Improving language models with smaller sizes and better performance.
method Detailed data and training procedure for StableLM 2 1.6B, including zero- and few-shot benchmarks and multilingual evaluations.
result StableLM 2 1.6B is the state-of-the-art open model under 2B parameters.
OpTorch optimizes deep learning for resource-limited environments.
problem Resource constraints in deep learning training.
method Optimized deep learning pipelines in training time and memory.
result Achieved similar accuracy to existing libraries with reduced memory usage.
Study identifies biomarkers for lung cancer in female non-smokers.
problem Identifying prognostic biomarkers for stage III NSCLC in non-smoking females.
method Gene expression profiling and XGBoost machine learning algorithm.
result Top biomarkers validated in literature, with AUC score of 0.835.
In many fields such as digital marketing, healthcare, finance, and robotics, it is common to have a well-tested and reliable baseline policy running in production (e.g., a recommender system). Nonetheless, the baseline policy is often suboptimal. In this case, it is desirable to deploy online learning algorithms (e.g.,…
Convolutional neural network (CNN) architectures have originated and revolutionized machine learning for images. In order to take advantage of CNNs in predictive modeling with audio data, standard FFT-based signal processing methods are often applied to convert the raw audio waveforms into an image-like representations…