TentacleNet improves binarized CNNs, reducing accuracy loss and memory usage.
problem Excessive accuracy loss in binarized CNNs.
method Parallelization inspired by ensemble learning theory, end-to-end trainable compact topology.
result Significant memory savings compared to state-of-the-art binary ensemble methods.
Researchers save memory on MCUs by reordering neural network operators.
problem Memory constraints on microcontrollers for neural network inference.
method Operator reordering to save memory, orthogonal to other compression methods.
result Reduced memory footprint of a CNN to deploy on an MCU with 512KB SRAM.
Optimizes memory and computation in wearable systems.
problem Minimizing resource usage in real-time classification for wearable systems.
method Hierarchical SVM classifier structure and memory optimization techniques.
result Up to 56% reduction in memory storage for activity recognition.
Paper proposes low-rank gradient approximation to save memory for deep neural network training.
problem Memory limitation on mobile devices for deep neural network training.
method Approximating gradient matrices using low-rank parameterization.
result Reduces training memory by about 33.0% for Adam optimization and 4.5% relative lower word error rate on ASR personalization task.
Deep learning detects inaccurate smart meters for resource savings.
problem Detecting and replacing inaccurate smart meters to save resources.
method Developed a deep-learning method using LSTM and CNN to predict electricity usage trajectories.
result High accuracy in detecting inaccurate meters for practical usage.
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
problem Exploding inverses in INNs cause numerical non-invertibility, leading to failures in various tasks.
method Derived bi-Lipschitz properties of INN building blocks, proposed regularizers for local invertibility, and stable INN designs for global invertibility.
result Bi-Lipschitz properties and stable INN designs are crucial for addressing numerical non-invertibility.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
TCA algorithm saves up to 40% energy in IoT devices.
problem Energy consumption and user comfort trade-off in IoT devices.
method Incremental learning Temporal Clustering Algorithm (TCA) for IoT devices.
result Achieved up to 40% energy savings with high success rate.
DSG activates only a small amount of neurons for efficient deep learning.
problem Efficient deployment of deep neural networks on embedded devices.
method Dynamic and sparse graph structure with dimension-reduction search and double-mask selection.
result Significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss.
Binary and ternary weights simplify RNNs for mobile devices.
problem Complexity and memory intensity of RNNs on mobile devices.
method Learn binary and ternary weights during training.
result Significant memory saving and inference speedup on ASIC platform.
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a …
The paper investigates how reducing memory supply voltage improves DNN accuracy under bit-cell faults.
problem Reducing energy consumption in deep neural networks by lowering memory supply voltage introduces bit-cell faults.
method The authors explore the robustness of DNN architectures to bit-cell faults and propose a regularizer to mitigate their effects.
result Operating the system in a faulty regime can save energy without significantly reducing accuracy.
NeuZip compresses neural network weights to save memory during training and inference.
problem Memory constraints in neural network training and inference.
method Entropy-based dynamic weight compression.
result Significant reduction in memory usage without performance loss.
PETRA enables parallel training of deep models with reversible architectures.
problem Challenges in parallelizing deep model training.
method Introduces PETRA, a novel approach for parallelizing gradient computations in reversible architectures.
result Achieves competitive accuracies on CIFAR-10, ImageNet32, and ImageNet using ResNet models.
MemNet optimizes neural architectures for memory efficiency.
problem Memory constraints in mobile devices limit the use of large neural networks.
method Augment-trim learning with memory consumption ranking score.
result MemNet finds architectures with 24.17% less memory usage compared to state-of-the-art methods.
SliceOut speeds up deep learning training without sacrificing accuracy.
problem Frequent model re-training and large model training workloads in deep learning.
method SliceOut uses dropout-inspired scheme to drop contiguous sets of units at random, leveraging GPU memory layout.
result 10-40% speedups and memory reduction with minimal accuracy loss.
Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly comput…
EuSN uses Euler discretization for stable, non-dissipative reservoir computing.
problem Designing stable and efficient reservoir computing models.
method Forward Euler discretization and antisymmetric recurrent matrices.
result EuSN outperforms standard RC models in long-term memory tasks and time-series classification.
New method improves grouped convolutions on edge devices.
problem Efficiently implementing grouped convolutions on edge devices.
method Grouped Spatial Pack Convolutions (GSPC) in TVM.
result GSPC outperforms existing implementations by 3.4x, 8x, and 4x on average.
New algorithms find near-stationary points in convex optimization.
problem Finding near-stationary points in convex optimization.
method Memory-saving variant of OGM-G, accelerated SVRG, adaptively regularized accelerated SVRG.
result Schemes achieve fast rates for minimizing gradient norm and function value.
Paper improves DNN accelerator robustness against bit errors with energy savings.
problem Bit errors in quantized DNN weights reduce energy efficiency.
method Combines robust fixed-point quantization, weight clipping, and random bit error training.
result Significantly improves robustness against random bit errors with high energy savings.
Improved memory efficiency for deep neural networks.
problem Increasing model size makes deep learning less accessible.
method Inductive-bias-driven network design and manifold-oriented regularization.
result Improved performance and generalization error compared to traditional methods.
A new memory-efficient Adam variant reduces second moments when feasible.
problem Memory constraints in training machine learning models.
method Signal-to-Noise Ratio (SNR) analysis to identify dimensions where second moments can be replaced by means.
result Memory-efficient Adam variant (SlimAdam) matches performance and stability of Adam while saving up to 98% of second moments.
A new backprop method reduces memory usage for deep neural networks.
problem Memory-intensive backpropagation in deep neural networks.
method Uses approximations of activations in buffers to reduce memory usage.
result Performance close to exact training with 4-bit precision activations.
Node2Grids uncouples GCN training for large graphs, saving memory and computation.
problem GCNs' coupled training framework limits flexibility and scalability for large-scale graphs.
method Node2Grids maps coupled graph data into independent grid-like data for efficient processing.
result Node2Grids achieves comparable results to GCNs while saving memory and computation.
Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…
Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
The multimodal web elements such as text and images are associated with inherent memory costs to store and transfer over the Internet. With the limited network connectivity in developing countries, webpage rendering gets delayed in the presence of high-memory demanding elements such as images (relative to text). To ove…
EnLSTM network improves log generation from small datasets.
problem Generating well logs from small datasets with high accuracy.
method Combining ENN and C-LSTM networks with perturbation methods.
result 34% reduction in mean-square-error compared to existing models.
Count-sketches reduce memory usage for deep learning models without sacrificing performance.
problem Memory constraints in deep learning models as they grow larger.
method Maintain a linear sketch to compress auxiliary variables in optimization methods.
result Compressed optimization methods maintain the same performance as full-sized baselines while using significantly less memory.
Improves Gaussian process models for large datasets.
problem Complexity and memory limitations in Gaussian process models.
method Combines variational sparse approximation and state-space formulation.
result Significant computational and memory savings for large datasets.
The large memory requirements of deep neural networks limit their deployment and adoption on many devices. Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization. In this paper, we present a novel scheme for l…
In this paper we propose a new algorithm for streaming principal component analysis. With limited memory, small devices cannot store all the samples in the high-dimensional regime. Streaming principal component analysis aims to find the k-dimensional subspace which can explain the most variation of the d-dimensiona…
Kernel clustering algorithm improved for large datasets using incomplete Cholesky factorization.
problem Large memory usage in kernel-based clustering for large-scale datasets.
method Approximate the kernel matrix using incomplete Cholesky factorization and apply linear k-means clustering. result The proposed method achieves similar performance to kernel k-means clustering but handles large-scale datasets efficiently. Financial frequency combs emerge from macroeconomic long-range memory.
problem Financial economy's long-run cyclic structure
method Incommensurate fractional-order financial model
result Frequency comb structure in steady-state spectrum
A new method estimates generative model mappings using kernel transfer operators, reducing costs and improving performance.
problem Efficiently estimating mappings between known and unknown distributions in generative models.
method Adapting kernel transfer operators to estimate mappings, reducing computational costs.
result Significant runtime savings and good empirical performance compared to existing methods.
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity. Our profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model reveal that fea…
BatchNorm helps train quantized networks by avoiding gradient explosion.
problem Training quantized neural networks is difficult due to gradient issues.
method Investigated the impact of BatchNorm on both full-precision and quantized networks.
result BatchNorm avoids gradient explosion in quantized networks, contrary to expectations.
AIDEL improves scalability of learned indexes in storage systems.
problem Expensive retraining and heavy inter-model dependency in learned indexes limit scalability.
method Construct different linear regression models based on data distribution, making them independent and easier to partition.
result AIDEL improves insertion performance by about 2x and comparable lookup performance.
CyBeR-0 optimizes federated learning with Byzantine resilience and reduced communication costs.
problem Byzantine attacks and communication inefficiency in federated learning.
method Transformed robust aggregation for zero-order optimization under client heterogeneity.
result CyBeR-0 achieves stable performance with minimal communication costs and reduced memory usage.
Efficient index structures for fast approximate nearest neighbor queries are required in many applications such as recommendation systems. In high-dimensional spaces, many conventional methods suffer from excessive usage of memory and slow response times. We propose a method where multiple random projection trees are c…
Improves memory efficiency for meta-learning with large images.
problem High memory usage in meta-learning for few-shot classification.
method LITE: episodic training scheme that decomposes task gradients and back-propagates only a random subset of images.
result Achieves state-of-the-art accuracy on real-world and challenging benchmarks.
Paper finds sharpness differences in transformer blocks accelerating LLM training.
problem Understanding and accelerating large language model pre-training.
method Uncovering sharpness disparity across transformer blocks and proposing Blockwise Learning Rate.
result Blockwise Learning Rate strategy accelerates LLM pre-training with lower loss and speedup.
Tensor decompositions are powerful tools for large data analytics as they jointly model multiple aspects of data into one framework and enable the discovery of the latent structures and higher-order correlations within the data. One of the most widely studied and used decompositions, especially in data mining and machi…
Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-ini…
SWALP averages SGD iterates for low-precision training, improving scalability and performance.
problem Improving scalability and performance in low-precision training.
method Averages low-precision SGD iterates with a modified learning rate schedule.
result SWALP matches full-precision SGD performance with 8-bit quantization and converges to optimal solutions.
Argentum is a crypto coin for saving and investment in unstable countries.
problem Stable purchasing power for savings in unstable economies.
method Designing a crypto coin backed by investment instruments.
result Provides a stabilization instrument for savings in unstable economies.
Seq-U-Net improves sequence modeling efficiency with dilated U-Net.
problem Efficiently modeling long-term dependencies in sequences.
method Causal U-Net architecture with dilated filters and slow feature hypothesis.
result Seq-U-Net achieves comparable performance with speed-ups of over 4x in audio generation.