A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New techniques improve 16-bit training accuracy without 32-bit units.
Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision ([5..8]-bit). However, majority of studies focus only on DNN inference. In this work, we propose DNN training using posits and compare with the floating point training. We evaluate on both MNIST and F…
The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, un…
Low-precision DNNs have been extensively explored in order to reduce the size of DNN models for edge devices. Recently, the posit numerical format has shown promise for DNN data representation and compute with ultra-low precision in [5..8]-bits. However, previous studies were limited to studying posit for DNN inference…
Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attemp…
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a cruc…
The high computational complexity associated with training deep neural networks limits online and real-time training on edge devices. This paper proposed an end-to-end training and inference scheme that eliminates multiplications by approximate operations in the log-domain which has the potential to significantly reduc…
The FloatSD technology has been shown to have excellent performance on low-complexity convolutional neural networks (CNNs) training and inference. In this paper, we applied FloatSD to recurrent neural networks (RNNs), specifically long short-term memory (LSTM). In addition to FloatSD weight representation, we quantized…
Secure Aggregation protocols allow a collection of mutually distrust parties, each holding a private value, to collaboratively compute the sum of those values without revealing the values themselves. We consider training a deep neural network in the Federated Learning model, using distributed stochastic gradient descen…
Paper explores reducing precision in SVM for faster text classification.
Deep neural networks are commonly developed and trained in 32-bit floating point format. Significant gains in performance and energy efficiency could be realized by training and inference in numerical formats optimized for deep learning. Despite advances in limited precision inference in recent years, training of neura…
We propose a new approach to combine Restricted Boltzmann Machines (RBMs) that can be used to solve combinatorial optimization problems. This allows synthesis of larger models from smaller RBMs that have been pretrained, thus effectively bypassing the problem of learning in large RBMs, and creating a system able to mod…
This paper proposes a learning framework for n-bit quantized neural networks that improves accuracy and speed on FPGAs.
A system for federated learning with private data, adding discrete Gaussian noise and secure aggregation.
Hardware-accelerated RBM solves large combinatorial problems and integer factorization.
We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks…
Quantized-TinyLLaVA reduces communication costs in split learning for multimodal models.
Improves matrix multiplication throughput for asymmetric bit-width operands.
In recent years, Convolutional Neural Network (CNN) based methods have achieved great success in a large number of applications and have been among the most powerful and widely used techniques in computer vision. However, CNN-based methods are computational-intensive and resource-consuming, and thus are hard to be inte…
State of the art deep learning models have made steady progress in the fields of computer vision and natural language processing, at the expense of growing model sizes and computational complexity. Deploying these models on low power and mobile devices poses a challenge due to their limited compute capabilities and str…
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
Un-trained neural networks outperform trained methods in MRI reconstruction.
We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the -th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory …
Meta-learning performance depends on train-validation split type.
New method reveals how training data influence diffusion model outputs.
Paper shows adversarial training can be fooled by new type of noise.
New MIP methods improve training of integer-valued neural networks.
Free adversarial training reduces the generalization gap compared to vanilla method.
Making neural networks robust against adversarial inputs has resulted in an arms race between new defenses and attacks. The most promising defenses, adversarially robust training and verifiably robust training, have limitations that restrict their practical applications. The adversarially robust training only makes the…
learn2mix trains neural nets faster by adjusting class proportions dynamically.
Loss-guided training accelerates node embedding methods on graphs.
Paper explores fast adversarial training to improve robustness with less computation.
Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
New method reconstructs significant parts of training data from neural networks.
The paper studies the asymptotic behavior of adversarial training under -perturbation.
This paper introduces 'General Cyclical Training' for neural networks.
Speeds up training and inference by pruning entire channels before training.
The paper predicts loss scaling across different datasets and compute scales.
Theoretical study shows adversarial training improves robustness in deep learning models.
Training dynamics reveal neural network complexity.
LIIT uses large deviations to train neural networks faster.
Empirical study shows GANs overfit and drop modes when training is deterministic.
Joint training improves model accuracy by selectively using privileged information.
ShadowSync separates background synchronization for scalable distributed training.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
Adversarial training, in which a network is trained on adversarial examples, is one of the few defenses against adversarial attacks that withstands strong attacks. Unfortunately, the high cost of generating strong adversarial examples makes standard adversarial training impractical on large-scale problems like ImageNet…
New method improves adversarial training efficiency and robustness.