Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.
A lightweight FPGA-based reinforcement learning approach for edge devices.
problem Resource constraints and inefficiency of DQN on edge devices.
method OS-ELM based training algorithm and L2 regularization for stability.
result 29.77x and 89.40x faster than conventional DQN-based approach for CartPole-v0 task.
A new method prunes neural networks faster and more efficiently.
problem Reducing training time and memory usage for neural networks.
method Set-based Task-Adaptive Meta Pruning (STAMP) that meta-learns a pruning mask.
result Significantly improved compression rates and faster training speed.
On-device federated learning updates edge models by exchanging trained results.
problem Limited training data at edge devices due to model drift.
method OS-ELM for sequential training and autoencoder for anomaly detection, combined with federated learning.
result The proposed approach produces a merged model as accurately as traditional methods with lower costs.
Federated Learning helps IoT devices learn without central servers.
problem Resource constraints in IoT devices hinder traditional ML approaches.
method Local training of IoT devices using global models.
result Federated Learning can be applied to IoT devices with varying resource capabilities.
Survey of knowledge distillation for model compression.
problem Deploying large deep models on resource-limited devices.
method Knowledge distillation from large teacher models to small student models.
result Effective model compression and acceleration.
Proposes a new framework for resource-limited recommendation.
problem Resource constraints affect user choices in recommendation tasks.
method Interest-behavior multiplicative network with MRRNNs and resource-limited branch.
result Framework effectively predicts user interactions considering resource limitations.
MDLdroid improves mobile deep learning for personal sensing with faster training.
problem Continuous local changes and resource constraints in personal mobile sensing affect global model performance.
method ChainSGD-reduce approach to reduce overhead and balance resources.
result 2x to 3.5x faster training on off-the-shelf mobile devices compared to single-device training.
Caratheodory's axiom limits arbitrage in resource-limited systems.
problem Non-arbitrage constraints in resource-limited financial systems.
method Preserving Caratheodory's axiom in resource-limited systems.
result Exponential family is the necessary geometric structure for both thermodynamics and finance.
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
FedDST trains sparse sub-networks to improve efficiency in federated learning.
problem Efficiently train large models on resource-limited edge devices with limited network bandwidth.
method Dynamic sparse training to reduce model size and communication.
result FedDST outperforms existing methods in federated learning, especially in non-i.i.d. settings.
A new algorithm for resource-aware multi-armed bandits minimizes regret.
problem Optimizing resource usage in a multi-armed bandit problem with censored observations.
method UCB-inspired online learning algorithm with theoretical regret analysis.
result The proposed algorithm outperforms standard multi-armed bandit algorithms in simulations.
Mobile edge computing (MEC) emerges recently as a promising solution to relieve resource-limited mobile devices from computation-intensive tasks, which enables devices to offload workloads to nearby MEC servers and improve the quality of computation experience. Nevertheless, by considering a MEC system consisting of mu…
Remarkable achievements have been attained by deep neural networks in various applications. However, the increasing depth and width of such models also lead to explosive growth in both storage and computation, which has restricted the deployment of deep neural networks on resource-limited edge devices. To address this …
OpTorch optimizes deep learning for resource-limited environments.
problem Resource constraints in deep learning training.
method Optimized deep learning pipelines in training time and memory.
result Achieved similar accuracy to existing libraries with reduced memory usage.
Pruning CNNs by removing less important filters based on empirical loss changes.
problem Reducing memory and computation requirements for CNNs on resource-limited devices.
method Developed a novel filter importance norm based on empirical loss changes, and used sampling and ranking to prune filters.
result Reduced 60% of parameters and 64% of FLOPs with less than 0.6% accuracy drop.
Quantizing large Neural Networks (NN) while maintaining the performance is highly desirable for resource-limited devices due to reduced memory and time complexity. It is usually formulated as a constrained optimization problem and optimized via a modified version of gradient descent. In this work, by interpreting the c…
Unified framework for accelerating DNNs on resource-limited platforms.
problem Accelerating DNN execution on resource-limited platforms.
method Block-based pruning framework with reweighted regularization.
result First universal framework for both CNNs and RNNs with real-time acceleration and no accuracy compromise.
The paper examines efficient algorithms for linear regression over resource-limited networks.
problem Efficient communication in distributed learning over resource-limited networks.
method Developed algorithms for communication-efficient learning of linear regression tasks.
result The algorithms enable a tradeoff between communication and learning with theoretical performance guarantees.
Pruning filters in neural networks improves performance and reduces costs.
problem Large neural networks have high inference costs and limited resource usage.
method Batch Bridgeout for efficient pruning of convolutional filters.
result Batch Bridgeout trained networks achieve higher accuracy across various pruning intensities.
We develop model free PAC performance guarantees for multiple concurrent MDPs, extending recent works where a single learner interacts with multiple non-interacting agents in a noise free environment. Our framework allows noisy and resource limited communication between agents, and develops novel PAC guarantees in this…
A new RNN architecture reduces model size and improves performance.
problem Overparameterization and resource limitations in RNNs.
method Jointly encodes weight matrices using tensor-train factorization.
result Reduces model size by several orders of magnitude without sacrificing performance.
Modeling alignment as resource-limited cognitive processes, researchers derive performance bounds.
problem Systematic deviations in feedback-based alignment of large language models.
method Modeling alignment as a two-stage cascade UoHoY given S, with cognitive and total capacities. result Capacity-coupled Alignment Performance Interval derived from Fano and PAC-Bayes bounds.
Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the expon…
This paper investigates compression techniques for deep neural networks to reduce their size without sacrificing performance.
problem Compression of large deep neural networks for resource-limited platforms.
method Weight pruning, quantization, and lossless weight matrix representations based on source coding.
result Achieved up to 165 times compression rate while maintaining or improving model performance.
EEG-TCNet improves MI-BMIs with high accuracy and low resource usage.
problem Improving motor-imagery brain-machine interfaces with high accuracy and low resource usage.
method Proposes EEG-TCNet, a novel TCN for embedded MI-BMIs.
result EEG-TCNet achieves 83.84% classification accuracy on MOABB, outperforming SoA by 0.25.
Convolutional Neural Network (CNN) based Deep Learning (DL) has achieved great progress in many real-life applications. Meanwhile, due to the complex model structures against strict latency and memory restriction, the implementation of CNN models on the resource-limited platforms is becoming more challenging. This work…
FedZKT enables resource-constrained devices to participate in federated learning with heterogeneous models.
problem Inequality in resource allocation hinders participation from resource-constrained devices in federated learning.
method Zero-shot knowledge transfer through a server-assigned distillation process.
result FedZKT effectively transfers knowledge across heterogeneous on-device models without requiring comparable local training efforts.
Deep learning identifies unknown IoT devices in network traffic.
problem Unauthorized IoT devices pose security risks in BYOD environments.
method Deep learning applied to network traffic images for IoT device identification.
result Over 99% accuracy in identifying 10 IoT devices and non-white-listed devices.
A method for trust evaluation of devices in human-device coexistence systems.
problem Efficient trust evaluation of devices in systems with diverse physical and social attributes.
method Canonical correlation analysis-enhanced hypergraph self-supervised learning (HSLCCA).
result The proposed HSLCCA method significantly outperforms baseline algorithms in identifying trusted devices.
Two approaches scale up DNN optimization for diverse edge devices.
problem Optimizing DNNs for edge devices with varying performance requirements.
method Reuse performance predictors on proxy devices and build scalable predictors.
result Optimized DNN designs for many different edge devices without lengthy optimization.
This paper optimizes how deep learning models are distributed across different devices.
problem Optimizing how large, complex neural networks are split across multiple devices.
method Identified and solved an optimization problem for device placement of DNN operators.
result Automated algorithms that solve the device placement problem for modern pipelined settings.
Distributed learning adapts to diverse devices, improving performance.
problem Training neural networks on devices with varying capabilities and resources.
method Each device trains a customized neural network, sharing parameters with others.
result Achieves higher rewards on more powerful devices without sacrificing weaker ones.
Flexible device participation improves federated learning convergence.
problem Strict device participation limits federated learning reach.
method Analytical results and new aggregation scheme for flexible participation.
result Convergence improved with flexible device participation.
The predominant paradigm for using machine learning models on a device is to train a model in the cloud and perform inference using the trained model on the device. However, with increasing number of smart devices and improved hardware, there is interest in performing model training on the device. Given this surge in i…
Semi-decentralized federated learning combines device-to-server and device-to-device communications for faster convergence.
problem Faster convergence in federated learning with decentralized model training.
method Two timescale hybrid federated learning (TT-HF) with cooperative D2D model aggregations.
result Achieves sublinear convergence rate of O(1/t) with adaptive control algorithm.
This paper optimizes AI inference on edge devices with reduced communication and computation costs.
problem Efficiently performing AI inference on resource-constrained edge devices with reduced communication and computation costs.
method A three-step framework for effective inference: model split point selection, communication-aware model compression, and task-oriented encoding of intermediate features.
result Our proposed framework achieves a better trade-off and significantly reduces inference latency compared to baseline methods.
AutoML has become a popular service that is provided by most leading cloud service providers today. In this paper, we focus on the AutoML problem from the \emph{service provider's perspective}, motivated by the following practical consideration: When an AutoML service needs to serve {\em multiple users} with {\em multi…
We propose a real-time context-aware learning system along with the architecture that runs on the mobile devices, provide services to the user and manage the IoT devices. In this system, an application running on mobile devices collected data from the sensors, learned about the user-defined context, made predictions in…
SplitEasy trains ML models on mobile devices without server data transfer.
problem Training complex DL models on resource-limited mobile devices.
method Split learning approach where sensitive layers are trained locally, computationally intensive layers on server.
result SplitEasy trains models on mobile devices with minimal data transfer, near-constant time per sample.
Fog learning distributes ML model training across heterogeneous devices and networks.
problem Challenges with conventional federated learning in heterogeneous networks.
method Intelligent distribution of ML model training across nodes from edge devices to cloud servers.
result Enhanced federated learning with multi-layer hybrid framework considering network, heterogeneity, and proximity.
FedCluster accelerates federated learning convergence by cycling device groups.
problem Federated learning convergence issues with device-level data heterogeneity.
method FedCluster groups devices into clusters that cycle through learning rounds, boosting convergence with meta-updates.
result FedCluster achieves faster convergence in nonconvex optimization compared to FedAvg.
The aim of this work is to address the question of whether we can in principle design rational decision-making agents or artificial intelligences embedded in computable physics such that their decisions are optimal in reasonable mathematical senses. Recent developments in rare event probability estimation, recursive ba…
A new federated learning framework for handling device heterogeneity.
problem Handling device heterogeneity in federated learning.
method Superquantile-based objective with parameterized levels of conformity, optimized using secure aggregation.
result The optimization algorithm converges to a stationary point.
SEFR is a fast, energy-efficient classifier for ultra-low power devices.
problem Running machine learning on battery-powered devices is challenging due to time and energy constraints.
method SEFR is an ultra-low power classifier with linear time complexity for training and testing.
result SEFR is 63 times faster and 70 times more energy efficient than state-of-the-art classifiers.
Automates detection of electric devices in 3D x-ray images of luggage.
problem Detecting electric devices in cluttered 3D baggage images.
method Unpack, Predict, eXtract, Repack (UXPR) algorithm using segmentation and ensemble learning.
result System can accurately detect electric devices in 3D baggage images.
AMS improves video inference on edge devices by adapting a small model with online knowledge distillation.
problem High computation cost of Deep Neural Networks for real-time video inference on edge devices.
method AMS uses a remote server to continually train and adapt a small model on edge devices, using online knowledge distillation from a large model.
result 0.4--17.8 percent mean Intersection-over-Union improvement in video semantic segmentation.