Improves neural network performance by enriching training dataset.
problem Achieving worst-case performance guarantees in neural networks.
method Adapting training dataset during training to reduce worst-case violations.
result Improved worst-case performance guarantees in neural networks.
Un-trained neural networks outperform trained methods in MRI reconstruction.
problem Accelerated MRI reconstruction with minimal training data.
method Variation of Deep Decoder without training data.
result Un-trained approach significantly outperforms other methods in reconstruction accuracy.
FIRE PBT improves neural network training by focusing on long-term performance.
problem Greedy decision mechanisms in PBT lead to poor long-term performance.
method FIRE PBT uses a fitness metric to encourage long-term performance over short-term improvements.
result FIRE PBT outperforms PBT on ImageNet and matches hand-tuned learning rates.
Deep learning improves image reconstruction, but scaling up training sets doesn't significantly boost performance.
problem Understanding the impact of training set size on deep learning image reconstruction.
method Empirical study and analytical characterization of performance scaling laws.
result Scaling up training set size does not significantly improve reconstruction quality for deep learning.
Regurgitative training with synthetic data harms LLM performance.
problem The impact of training LLMs with synthetic data generated by other LLMs.
method Fine-tuning GPT-3.5 with synthetic and real data in a machine translation task.
result Regurgitative training significantly reduces LLM performance.
Training on mixed distributions improves test performance even when components are unrelated.
problem Improving test performance with mismatched training and test distributions.
method Analyzing mixture distributions with different training and test proportions.
result Distribution shift can be beneficial, improving test performance even when components are unrelated.
Gradient amplification boosts deep learning model performance without increasing training time.
problem Vanishing gradients in deep neural networks.
method Gradient amplification approach to prevent vanishing gradients and training strategy to enable/disable across epochs.
result Improves performance of deep learning models with reduced training time.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
In supervised machine learning for author name disambiguation, negative training data are often dominantly larger than positive training data. This paper examines how the ratios of negative to positive training data can affect the performance of machine learning algorithms to disambiguate author names in bibliographic …
Paper develops an efficient approach to reduce HPO time.
problem Challenges in determining optimal hyperparameters due to large number and training time.
method Nested Latin hypercube design for initialization, truncated additive Gaussian process model for calibration, sequential model-based algorithm for optimization.
result Demonstrates competitive performance on various machine learning models.
Paper shows training can improve GCN performance without changing architecture.
problem Training difficulty of GCNs limits their performance.
method Identified and mitigated energy loss during training.
result Significant decrease in training difficulties and notable performance boost.
Estimates neural architecture performance speedily.
problem Accurately evaluating neural architectures' generalization performance.
method Estimates final test performance based on training speed.
result Consistently outperforms other alternatives in correlation with true test performance.
Training deep learning models on mobile devices recently becomes possible, because of increasing computation power on mobile hardware and the advantages of enabling high user experiences. Most of the existing work on machine learning at mobile devices is focused on the inference of deep learning models (particularly co…
New framework shows diverse training data improves subgroup and overall performance.
problem Lack of understanding how diverse training data affects subgroup and overall performance.
method Casts data collection as part of the learning process, analyzes dataset compositions, and guides dataset design.
result Diverse representation in training data improves subgroup and overall performance.
This paper presents by simulation how approximate multipliers can be utilized to enhance the training performance of convolutional neural networks (CNNs). Approximate multipliers have significantly better performance in terms of speed, power, and area compared to exact multipliers. However, approximate multipliers have…
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Improved pre-trained embeddings through effective entropy maximization.
problem Developing high-quality pre-trained embeddings for future tasks.
method E2MC criterion defined in terms of low-dimensional constraints.
result Significant improvement in downstream performance.
Quantum annealer speeds up RBM training for image classification.
problem Training RBM with contrastive divergence (CD) is slow and computationally expensive.
method Used D-Wave 2000Q quantum annealer to calculate model expectation of gradient learning for RBM.
result Quantum training yields similar classification performance to CD but faster.
Improves neural network performance by dynamically adjusting model weights based on source reliability.
problem Training neural networks on data from unreliable sources leads to poor performance.
method Dynamic re-weighting strategy using likelihood tempering to adjust model weights based on estimated source reliability.
result Significant improvement in model performance when trained on mixtures of reliable and unreliable data sources.
TuneUp improves GNN training by focusing on hard-to-learn nodes.
problem Sub-optimal training of GNNs on all nodes equally.
method Two-stage training: base GNN + tail node improvement.
result Significant improvement in tail node prediction performance.
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Paper tackles model collapse in recursive generative models using a weighted training scheme.
problem Model collapse in recursive generative models trained on synthetic data.
method Iteratively trains models on real and synthetic data, evaluates weighted training schemes.
result Optimal weighting scheme for synthetic data follows a unified expression, revealing a trade-off with model performance.
Deep neural networks have shown remarkable performance across a wide range of vision-based tasks, particularly due to the availability of large-scale datasets for training and better architectures. However, data seen in the real world are often affected by distortions that not accounted for by the training datasets. In…
Trained recurrent networks are powerful tools for modeling dynamic neural computations. We present a target-based method for modifying the full connectivity matrix of a recurrent network to train it to perform tasks involving temporally complex input/output transformations. The method introduces a second network during…
Proposes a Siamese NN for algorithm selection focusing on alike performing instances.
problem Lack of effective meta-features for algorithm selection via meta-learning.
method Siamese Neural Network architecture with 'Algorithm-Performance Personas' concept.
result Proposed metric outperforms standard performance metrics in training sample selection.
We train a network to generate mappings between training sets and classification policies (a 'classifier generator') by conditioning on the entire training set via an attentional mechanism. The network is directly optimized for test set performance on an training set of related tasks, which is then transferred to unsee…
FedAUX improves Federated Learning by better using unlabeled data.
problem Improving Federated Learning performance with unlabeled data.
method FedAUX modifies FD training by unsupervised pre-training and private certainty scoring.
result FedAUX outperforms state-of-the-art Federated Learning methods.
Method diagnoses model performance under distribution shifts.
problem Understanding and improving model performance under distribution shifts.
method DIstribution Shift DEcomposition (DISDE) method to attribute performance drop to distribution shifts.
result Shows how model performance can be improved across different distribution shifts.
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
PUMA augments models to remove unique data points without performance loss.
problem Preserving model performance while removing unique training data points.
method Explicitly models data influence, reweights remaining data optimally.
result PUMA effectively removes unique data points without performance degradation.
Sloth predicts LLM performance using latent skills across families.
problem Variations in benchmark performance due to differences in training configurations and data processing across model families.
method Sloth uses publicly available benchmark data and assumes LLM performance is driven by latent skills influenced by model size and training tokens. It exploits correlations across benchmarks to provide accurate predictions.
result Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex tasks.
Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible…
This study explores training Bayesian neural networks at scale using high-performance computing.
problem Challenges in training Bayesian neural networks at scale due to computational overhead.
method High-performance computing with distributed training, network pruning.
result Pruning up to 80% of the network can reduce inference time by 7.0% without significant accuracy loss.
Multi-headed ensembles boost model performance with faster training.
problem Limited computational resources hinder ensemble search performance.
method Extend NES to multi-headed ensembles, leveraging end-to-end training and one-shot NAS methods.
result Multi-headed ensemble search finds robust ensembles 3 times faster with comparable performance.
In training a deep learning system to perform audio transcription, two practical problems may arise. Firstly, most datasets are weakly labelled, having only a list of events present in each recording without any temporal information for training. Secondly, deep neural networks need a very large amount of labelled train…
Study examines how classifier performance is affected by training data quality.
problem How classifier performance is affected by training data quality.
method Extensive numerical experiments with four classifiers (Bayes, neural nets, partition models, random forests) on metagenomic assembly data.
result Classifier performance degrades as training data quality degrades, leading to breakdown-like behavior.
This paper shows RBMs can maintain strong performance even after extreme pruning, but only if done early in training.
problem The computational and environmental costs of large neural networks.
method Investigating the performance of RBMs under extreme pruning conditions, inspired by the Lottery Ticket Hypothesis.
result RBMs can achieve high-quality generative performance even after 80% pruning, but performance degrades sharply above a critical point.
New method improves neural spike train models by minimizing divergence directly, leading to better performance.
problem Poor performance and divergence issues in spike train models using maximum likelihood estimation.
method Directly minimize maximum mean discrepancy using spike train kernels and stochastic optimization.
result The proposed method generates well-behaved models with better control over feature trade-offs.
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…
Steady-state visual evoked potentials (SSVEP) brain-computer interface (BCI) provides reliable responses leading to high accuracy and information throughput. But achieving high accuracy typically requires a relatively long time window of one second or more. Various methods were proposed to improve sub-second response a…
This work theoretically investigates the performance of a composite neural network. A composite neural network is a rooted directed acyclic graph combining a set of pre-trained and non-instantiated neural network models, where a pre-trained neural network model is well-crafted for a specific task and targeted to approx…
Pre-training improves model coverage, crucial for downstream performance.
problem Understanding why pre-training enhances model performance.
method Coverage principle, focusing on next-token prediction and model quality.
result Coverage generalizes faster than cross-entropy, improving downstream performance.
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…
This work improves certifiably robust models by distilling knowledge from adversarially robust teachers.
problem Certifiably robust models suffer from poor standard performance.
method Knowledge distillation from adversarially robust teachers to improve standard performance.
result Distillation from adversarially robust teachers consistently improves certified training performance.
Proposes a new cross-validation method to estimate model performance.
problem The standard cross-validation method does not accurately estimate the performance of the recommended model.
method Develops a new random-effects model framework to improve naive cross-validation estimators.
result Proposed estimators outperform conventional and naive methods in estimating model performance.