Neural network pruning with suitable retraining can yield networks with considerably fewer parameters than the original with comparable degrees of accuracy. Typical pruning methods require large, fully trained networks as a starting point from which they perform a time-intensive iterative pruning and retraining procedu…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “training time reduction”
WeldNet reduces complex dynamics to simpler, manageable segments.
problem Complex, high-dimensional time-dependent datasets from physical processes are costly to simulate.
method Windowed Encoders for Learning Dynamics, splitting time domain into windows for nonlinear dimension reduction and propagator training.
result WeldNet captures nonlinear latent structures and dynamics, outperforming existing methods.
Discrete random variables are natural components of probabilistic clustering models. A number of VAE variants with discrete latent variables have been developed. Training such methods requires marginalizing over the discrete latent variables, causing training time complexity to be linear in the number clusters. By appl…
Testing the implementation of deep learning systems and their training routines is crucial to maintain a reliable code base. Modern software development employs processes, such as Continuous Integration, in which changes to the software are frequently integrated and tested. However, testing the training routines requir…
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
problem Challenges in assessing variance reduction during DNN training using importance sampling.
method Proposes a method for estimating variance reduction using minibatches sampled under importance sampling.
result Demonstrates consistent reduction in variance, improved training efficiency, and enhanced model accuracy.
This work simplifies Gaussian process regression for multiple outputs.
problem Exponential computational complexity in Gaussian process regression.
method Approximating the covariance kernel using eigenvalues and functions.
result Significant reduction in training and regression complexity.
Communication on heterogeneous edge networks is a fundamental bottleneck in Federated Learning (FL), restricting both model capacity and user participation. To address this issue, we introduce two novel strategies to reduce communication costs: (1) the use of lossy compression on the global model sent server-to-client;…
RSAC improves lightweight continuous learning efficiency.
problem Excessive training time and memory usage in continuous learning.
method Regularized subspace approximation classifier with feature reduction and regularization.
result RSAC achieves more efficient continuous learning than prior methods.
K-Nearest neighbor classifier (k-NNC) is simple to use and has little design time like finding k values in k-nearest neighbor classifier, hence these are suitable to work with dynamically varying data-sets. There exists some fundamental improvements over the basic k-NNC, like weighted k-nearest neighbors classifier (wh…
Improved time complexity for parallel stochastic optimization in heterogeneous systems.
problem Time complexity in parallel stochastic optimization for large-scale machine learning models.
method Proposes Rennala MVR, a variance-reduced extension of Rennala SGD based on momentum-based variance reduction.
result Variance reduction improves time complexity in relevant parameter regimes for parallel stochastic optimization in heterogeneous systems.
The diverse world of machine learning applications has given rise to a plethora of algorithms and optimization methods, finely tuned to the specific regression or classification task at hand. We reduce the complexity of algorithm design for machine learning by reductions: we develop reductions that take a method develo…
Balanced Neural ODEs combine VAEs and Neural ODEs for efficient time series modeling.
problem Efficiently modeling systems with time-varying inputs and varying complexity.
method Combines VAEs for dimensionality reduction and Neural ODEs for dynamics, using variational parameters to adaptively learn.
result Balanced Neural ODEs (B-NODE) efficiently approximate Koopman operator without predefined dimensionality.
Automates PDE model reduction with time-scale separation.
problem Computational expense in solving high-dimensional PDEs.
method Combines autoencoder and time-continuous model for latent dynamics.
result Automatically learns independent temporal scales in complex systems.
A scalable graphical method is presented for selecting, and partitioning datasets for the training phase of a classification task. For the heuristic, a clustering algorithm is required to get its computation cost in a reasonable proportion to the task itself. This step is proceeded by construction of an information gra…
Enhances supervised visualization for unseen data using autoencoders and random forest.
problem Lack of generalization to unseen test sets in supervised dimensionality reduction.
method Combines autoencoder and random forest proximities for out-of-sample extension.
result 40% reduction in training time with 10% of training data, achieving consistent quality.
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equ…
The well-known Mori-Zwanzig theory tells us that model reduction leads to memory effect. For a long time, modeling the memory effect accurately and efficiently has been an important but nearly impossible task in developing a good reduced model. In this work, we explore a natural analogy between recurrent neural network…
MARS optimizes large model training by reducing variance, outperforming AdamW.
problem Training large models efficiently and scalably.
method Unified optimization framework MARS combining preconditioned gradient updates and variance reduction.
result MARS outperforms AdamW in training GPT-2 models.
The paper extends Marsden-Weinstein reduction to mechanical presymplectic structures for time-dependent Hamiltonian systems.
problem Limitations of Marsden-Weinstein reduction for cosymplectic structures in time-dependent Hamiltonian systems.
method Developed Marsden-Weinstein reduction for mechanical presymplectic structures.
result Mechanical presymplectic structures provide a more suitable framework for time-dependent Hamiltonian systems than cosymplectic structures.
Paper explores reducing precision in SVM for faster text classification.
problem Efficiency in multi-class text classification training.
method Comparison of SVM trained with reduced precision (16-bit, half) vs original.
result Reduced precision training maintains text classification accuracy.
Paper speeds up large foundation models for time series data.
problem Resource-intensive foundation models limit accessibility.
method Dimensionality reduction techniques, including PCA and neural network adapters.
result Up to 10x speedup and 4.5x more datasets fit on a single GPU.
We present a theoretical analysis and empirical evaluations of a novel set of techniques for computational cost reduction of classifiers that are based on learned transform and soft-threshold. By modifying optimization procedures for dictionary and classifier training, as well as the resulting dictionary entries, our t…
RMDA trains structured neural networks with regularization and variance reduction.
problem Training structured neural networks with desired properties.
method RMDA algorithm for structured NNs with regularization and variance reduction.
result RMDA achieves desired structures identical to regularizer's at stationary points.
The study analyzes sharpness dynamics in neural networks, revealing mechanisms and conditions.
problem Understanding sharpness in neural network training.
method Fixed point analysis and edge of stability analysis in a simplified 2-layer linear network.
result Reveals mechanisms behind sharpness trends, conditions for edge of stability, and a period-doubling route to chaos.
In the covariate shift learning scenario, the training and test covariate distributions differ, so that a predictor's average loss over the training and test distributions also differ. In this work, we explore the potential of extreme dimension reduction, i.e. to very low dimensions, in improving the performance of imp…
We present local discriminative Gaussian (LDG) dimensionality reduction, a supervised dimensionality reduction technique for classification. The LDG objective function is an approximation to the leave-one-out training error of a local quadratic discriminant analysis classifier, and thus acts locally to each training po…
Paper compares Lagrangian reduction methods for rigid body systems.
problem Modeling and reduction of rigid body systems with rotors.
method Euler-Poincaré reduction by the whole group and reduction by stages.
result Equivalence of equations and conservation laws are tracked.
Paper develops an efficient approach to reduce HPO time.
problem Challenges in determining optimal hyperparameters due to large number and training time.
method Nested Latin hypercube design for initialization, truncated additive Gaussian process model for calibration, sequential model-based algorithm for optimization.
result Demonstrates competitive performance on various machine learning models.
New method reduces PDE surrogate model training costs by selectively acquiring time steps.
problem High computational cost of generating training data for PDE surrogate models.
method STAP (Selective Time-Step Acquisition for PDEs) framework that acquires only important time steps.
result Demonstrated effectiveness on several benchmark PDEs, reducing training costs.
Predict training time without training using a linearized model.
problem Predicting the training time of a deep network without actual training.
method Approximate training dynamics with a linearized model and solve a low-dimensional SDE in function space.
result Predict training time within 20% error margin and 30-45x reduction in cost.
To train good supervised and semi-supervised object classifiers, it is critical that we not waste the time of the human experts who are providing the training labels. Existing active learning strategies can have uneven performance, being efficient on some datasets but wasteful on others, or inconsistent just between ru…
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity. Our profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model reveal that fea…
This paper proposes a probabilistic neural network developed on the basis of time-series discriminant component analysis (TSDCA) that can be used to classify high-dimensional time-series patterns. TSDCA involves the compression of high-dimensional time series into a lower-dimensional space using a set of orthogonal tra…
Enhances robustness of AT frameworks to multiple perturbations without increasing training complexity.
problem Defending against the union of multiple perturbations in adversarial training.
method SNAP technique that augments a network with shaped noise to enhance robustness.
result 14%-to-20% improvement in adversarial accuracy for ResNet-18 on CIFAR-10.
A new framework learns clustering and dimensionality reduction together.
problem Challenges in clustering high-dimensional data.
method Gradient-based manifold optimization for joint learning.
result Better performance compared to existing clustering algorithms.
Despite their successes in the field of self-learning AI, Convolutional Neural Networks (CNNs) suffer from having too many trainable parameters, impacting computational performance. Several approaches have been proposed to reduce the number of parameters in the visual domain, the Inception architecture [Szegedy et al.,…
LIIT uses large deviations to train neural networks faster.
problem Training neural networks requires large amounts of data, leading to inefficiencies.
method LIIT uses large deviations principle to generate and update training samples.
result LIIT achieves similar classification performance with smaller, more representative samples.
Spatiotemporal sequence prediction is an important problem in deep learning. We study next-frame(s) video prediction using a deep-learning-based predictive coding framework that uses convolutional, long short-term memory (convLSTM) modules. We introduce a novel reduced-gate convolutional LSTM(rgcLSTM) architecture that…
We propose a novel technique for faster deep neural network training which systematically applies sample-based approximation to the constituent tensor operations, i.e., matrix multiplications and convolutions. We introduce new sampling techniques, study their theoretical properties, and prove that they provide the same…
Proposes a federated learning approach for industrial asset failure prediction.
problem Lack of data and privacy concerns in industrial prognostics.
method Two-stage federated learning: dimension reduction and parameter estimation.
result Validated the approach using simulated and real data.
An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Modelseess.AS
Speaker-independent speech recognition systems trained with data from many users are generally robust against speaker variability and work well for a large population of speakers. However, these systems do not always generalize well for users with very different speech characteristics. This issue can be addressed by bu…
New optimizer MARS-M combines variance reduction with Muon for faster LLM training.
problem Training large-scale neural networks efficiently.
method Integrates MARS variance reduction with Muon optimizer.
result MARS-M converges to a first-order stationary point at a rate of .
New method finds efficient low-rank neural networks during training.
problem High memory and computational demands of neural networks.
method Restricts weight matrices to a low-rank manifold and updates low-rank factors.
result Significantly reduced time and memory resources required for training and evaluation.
Study reduces redundant information in multi-modal datasets.
problem Redundant information increases data size and costs.
method Multimodal framework using entropy, mutual info, Lasso, SHAP, PCA, topic modeling.
result Significant feature space can be pruned with minimal loss in performance.
Pruning improves model generalization in over-parameterized models, contradicting traditional theories.
problem Pruning's effect on generalization in over-parameterized models.
method Empirical study on standard pruning algorithms and additional regularization effects.
result Pruning leads to better training and regularization, improving generalization.
How can we train a statistical mixture model on a massive data set? In this work we show how to construct coresets for mixtures of Gaussians. A coreset is a weighted subset of the data, which guarantees that models fitting the coreset also provide a good fit for the original data set. We show that, perhaps surprisingly…
The paper improves reinforcement learning stability and efficiency with a new theoretical framework.
problem Stability and efficiency in reinforcement learning, especially in data-scarce scenarios.
method Theoretical framework using resampled - and -statistics to model experience replay, applied to policy evaluation and kernel ridge regression.
result Significant improvements in stability and efficiency, particularly in data-scarce scenarios.
Paper identifies reductive MDPs, solving them in polynomial time.
problem Computational hardness of general MDPs and tractability of finite-horizon MDPs.
method Defines reductivity, a new class of SSPs, and develops a polynomial-time solution.
result Optimal policies can be found in polynomial time for reductive SSPs and MDPs.