Trained DNN models are increasingly adopted as integral parts of software systems, but they often perform deficiently in the field. A particularly damaging problem is that DNN models often give false predictions with high confidence, due to the unavoidable slight divergences between operation data and training data. To…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “DNN operations”
Sparse DNNs face scalability issues; MIT/IEEE/Amazon challenge analyzes best solutions.
problem Scalability issues in Sparse Deep Neural Networks (DNNs).
method Mathematically defined DNN inference computation, community submissions from various fields.
result Sparse DNN execution time, , is strongly dependent on the number of operations, .
Paper improves DNN accelerator robustness against bit errors with energy savings.
problem Bit errors in quantized DNN weights reduce energy efficiency.
method Combines robust fixed-point quantization, weight clipping, and random bit error training.
result Significantly improves robustness against random bit errors with high energy savings.
Neural Operators Meet Energy-based Theory: Operator Learning for Hamiltonian and Dissipative PDEsstat.ML
Proposes ENOs for learning PDE solutions that conserve energy.
problem Learning dynamics that obey physical laws, especially in super-resolution settings.
method Energy-consistent Neural Operators (ENOs) with a novel penalty function inspired by energy-based theory.
result ENOs outperform existing DNN models in predicting solutions from data, especially in super-resolution settings.
Linear algebra approach for parallel deep learning models.
problem Training large DNNs in distributed environments.
method Linear algebraic approach to model parallelism.
result Manual development of backward operators for gradient-based training.
Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networkscs.LG
Deep neural networks are proven universally powerful using Koopman operator.
problem Proving the universality of deep neural networks.
method Formal deep network as a dual voice transform with Koopman operator, using group actions and Schur's lemma.
result Simple proof of the universality of DNNs.
Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet their high computational complexity and memory requirement limit them to applications on high-performance computing platforms. In this paper, we propose a new method to evaluate DNNs trained with 32bit floating point (float32) accura…
Deep neural networks (DNN) are increasingly being accelerated on application-specific hardware such as the Google TPU designed especially for deep learning. Timing speculation is a promising approach to further increase the energy efficiency of DNN accelerators. Architectural exploration for timing speculation requires…
Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…
Deep neural networks (DNNs) may outperform human brains in complex tasks, but the lack of transparency in their decision-making processes makes us question whether we could fully trust DNNs with high stakes problems. As DNNs' operations rely on a massive number of both parallel and sequential linear/nonlinear computati…
Big data is one of the cornerstones to enabling and training deep neural networks (DNNs). Because of the lack of expertise, to gain benefits from their data, average users have to rely on and upload their private data to big data companies they may not trust. Due to the compliance, legal, or privacy constraints, most u…
Pruning improves DNNs against MIA while reducing model size and computation.
problem Vulnerability of DNNs to membership inference attacks (MIA).
method Proposes a pruning algorithm to reduce model size and computational operations.
result Pruned subnetwork prevents privacy leakage from MIA with competitive accuracy.
Reinforcement Learning (RL) has achieved impressive performance in many complex environments due to the integration with Deep Neural Networks (DNNs). At the same time, Genetic Algorithms (GAs), often seen as a competing approach to RL, had limited success in scaling up to the DNNs required to solve challenging tasks. C…
Deep neural network (DNN) quantization converting floating-point (FP) data in the network to integers (INT) is an effective way to shrink the model size for memory saving and simplify the operations for compute acceleration. Recently, researches on DNN quantization develop from inference to training, laying a foundatio…
The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek t…
Deep Neural Networks (DNNs) are increasingly deployed in highly energy-constrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the fir…
This work develops fast and accurate ROMs for AM models using OL methods.
problem Achieving specific material properties in AM by manipulating process parameters increases computational load.
method Operator learning (OL) approach with Fourier neural operator (FNO) and DeepONet.
result OL methods offer comparable performance and outperform DNN in accuracy and generalizability.
This paper optimizes how deep learning models are distributed across different devices.
problem Optimizing how large, complex neural networks are split across multiple devices.
method Identified and solved an optimization problem for device placement of DNN operators.
result Automated algorithms that solve the device placement problem for modern pipelined settings.
Paper proposes a DNN-driven AF framework for improved generalization.
problem Generalization challenge in adaptive filtering.
method Structural embedding of DNN into AF system, using maximum likelihood as implicit cost function.
result Demonstrates improved generalization capability through extensive experiments.
Deep neural networks (DNNs) have achieved excellent performance on several tasks and have been widely applied in both academia and industry. However, DNNs are vulnerable to adversarial machine learning attacks, in which noise is added to the input to change the network output. We have devised an image-processing-based …
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
Many DNN-enabled vision applications constantly operate under severe energy constraints such as unmanned aerial vehicles, Augmented Reality headsets, and smartphones. Designing DNNs that can meet a stringent energy budget is becoming increasingly important. This paper proposes ECC, a framework that compresses DNNs to m…
NetFuse merges different DNN models with varying weights for faster inference.
problem Inference speed of DNN models with different weights cannot be improved using existing techniques.
method NetFuse merges models with the same architecture but different weights and inputs, replacing operations with more general ones.
result NetFuse can speed up DNN inference time up to 3.6x on a NVIDIA V100 GPU.
slimTrain simplifies DNN training by separating features and adapting hyperparameters.
problem Challenges in training deep neural networks, including non-convexity, non-smoothness, and hyperparameter sensitivity.
method slimTrain exploits separability in DNN architectures to reduce hyperparameter sensitivity and improve convergence.
result slimTrain outperforms existing methods with recommended hyperparameters and reduces sensitivity to remaining hyperparameters.
This paper reviews normalization techniques for DNNs.
problem Improving training speed and generalization of DNNs.
method Taxonomy of normalization methods, decomposition of activation methods.
result Insight for designing new normalization techniques.
New theory predicts deep neural networks can operate in an extended critical regime without fine-tuning.
problem Understanding the dynamics and computational principles of deep neural networks.
method Combining theories of heavy-tailed random matrices and non-equilibrium statistical physics.
result Deep neural networks can operate in an extended critical regime without fine-tuning parameters.
Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Netscs.LG
GBML with deep nets converges globally and generalizes well.
problem Theoretical guarantees for few-shot learning with deep nets.
method Proving global convergence and generalization bounds for GBML with over-parameterized DNNs.
result GBML with over-parameterized DNNs converges globally to the optimum at a linear rate and achieves good generalization.
POLICE enforces linear constraints on deep neural networks efficiently.
problem Enforcing constraints on deep neural networks without affecting optimization.
method Provably optimal affine constraint enforcement method that minimally modifies DNNs.
result POLICE ensures DNNs fulfill affine constraints during training and testing.
Winograd convolution is widely used in deep neural networks (DNNs). Existing work for DNNs considers only the subset Winograd algorithms that are equivalent to Toom-Cook convolution. We investigate a wider range of Winograd algorithms for DNNs and show that these additional algorithms can significantly improve floating…
DQA efficiently quantizes deep neural network activations for resource-constrained devices.
problem Efficiently quantizing deep neural network activations for resource-constrained devices.
method DQA uses simple shifting-based operations and Huffman coding for sub-6-bit quantization.
result DQA achieves significantly better accuracy than direct quantization and state-of-the-art methods.
Analyzing biological and artificial neural networks: challenges with opportunities for synergy?q-bio.NC
Deep neural networks (DNNs) transform stimuli across multiple processing stages to produce representations that can be used to solve complex tasks, such as object recognition in images. However, a full understanding of how they achieve this remains elusive. The complexity of biological neural networks substantially exc…
KNF uses Koopman theory to forecast time series with changing dynamics.
problem Temporal distributional shifts in time series data.
method KNF combines DNNs with Koopman theory to learn dynamic operators.
result KNF outperforms alternatives on time series datasets with distributional shifts.
Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Codingcs.IT
Estimates rate-distortion function for large datasets using neural networks.
problem Designing lossy data compression schemes and comparing them with theoretical limits.
method Re-formulate rate-distortion objective and solve using neural networks.
result NERD accurately estimates the rate-distortion function for real-world datasets.
The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and enco…
Resistive crossbars have attracted significant interest in the design of Deep Neural Network (DNN) accelerators due to their ability to natively execute massively parallel vector-matrix multiplications within dense memory arrays. However, crossbar-based computations face a major challenge due to a variety of device and…
Theoretical understanding of deep learning is one of the most important tasks facing the statistics and machine learning communities. While deep neural networks (DNNs) originated as engineering methods and models of biological networks in neuroscience and psychology, they have quickly become a centerpiece of the machin…
Metrics assess uncertainty structure and distribution for regression models.
problem Quantifying uncertainty in high-dimensional and nonlinear regression tasks.
method Two bounded comparison metrics for uncertainty structure and distribution.
result DNNs and DNOs provide encouraging uncertainty metric values in high dimensions.
Because deep neural networks (DNNs) rely on a large number of parameters and computations, their implementation in energy-constrained systems is challenging. In this paper, we investigate the solution of reducing the supply voltage of the memories used in the system, which results in bit-cell faults. We explore the rob…
DNN policies improve stochastic AC OPF for power grid optimization.
problem Optimizing power grid operations under uncertainty.
method Deep neural network (DNN) policies for real-time generator dispatch decisions.
result DNN policies enforce feasibility constraints and produce near optimal solutions.
This paper proposes a hardware-oriented dropout algorithm, which is efficient for field programmable gate array (FPGA) implementation. In deep neural networks (DNNs), overfitting occurs when networks are overtrained and adapt too well to training data. Consequently, they fail in predicting unseen data used as test data…
Contemporary power grids are being challenged by rapid voltage fluctuations that are caused by large-scale deployment of renewable generation, electric vehicles, and demand response programs. In this context, monitoring the grid's operating conditions in real time becomes increasingly critical. With the emergent large …
Expressivity is one of the most significant issues in assessing neural networks. In this paper, we provide a quantitative analysis of the expressivity for the deep neural network (DNN) from its dynamic model, where the Hilbert space is employed to analyze the convergence and criticality. We study the feature mapping of…
Deep Optimisation (DO) combines evolutionary search with Deep Neural Networks (DNNs) in a novel way - not for optimising a learning algorithm, but for finding a solution to an optimisation problem. Deep learning has been successfully applied to classification, regression, decision and generative tasks and in this paper…
Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate fo…
Paper combines regularization and pruning to reduce FLOPs in DNNs.
problem Efficient inference in deep learning models.
method Apply mixup and cutout regularizations and soft filter pruning to the ResNet architecture.
result Combining regularization and pruning reduces FLOPs more effectively than each technique alone.
Deep neural networks (DNNs) frequently contain far more weights, represented at a higher precision, than are required for the specific task which they are trained to perform. Consequently, they can often be compressed using techniques such as weight pruning and quantization that reduce both the model size and inference…
To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To …
Advances in the field of Machine Learning and Deep Neural Networks (DNNs) has enabled rapid development of sophisticated and autonomous systems. However, the inherent complexity to rigorously assure the safe operation of such systems hinders their real-world adoption in safety-critical domains such as aerospace and med…