Paper proposes new loss functions for training energy networks.
problem Challenges in computing gradients for training energy networks.
method Proposes generalized Fenchel-Young losses for efficient gradient computation.
result Demonstrates the calibration of excess risk for linear-concave energies.
A neural network model minimizes region-based free energy for faster inference in MRFs.
problem Efficient inference in complex Markov random fields (MRFs).
method Region-based Energy Neural Network (RENN) that directly minimizes region-based free energy.
result RENN outperforms other methods in marginal distribution estimation, partition function estimation, and MRF learning.
Optimizes energy efficiency in wireless sensor networks with limited information.
problem Maximizing energy efficiency in energy harvesting wireless sensor networks with limited channel state information.
method Modeling as a Multi-Armed Bandits problem and developing an Upper Confidence Bound algorithm.
result Significant gains in energy efficiency compared to benchmark schemes.
Paper optimizes neural network layers to reduce energy usage without sacrificing accuracy.
problem Energy consumption in deep neural networks during inference.
method Layerwise noise maximization to optimize reliability of memory elements.
result Reduces memory energy consumption by 3.3 times at equal accuracy.
This work optimizes DNN inference for energy-harvesting devices by compressing and selectively executing neural network exits.
problem Inference delays and energy inefficiency in energy-harvesting devices.
method Developed a power trace-aware and exit-guided network compression algorithm for multi-exit neural networks.
result Superior accuracy and reduced latency compared to state-of-the-art techniques.
Unified perspective on Hopfield networks with attention module.
problem Understanding and optimizing Hopfield networks with attention mechanisms.
method Study of BM counterparts of modern Hopfield networks and their salient properties.
result Introduction of AttnBM with tractable likelihood and gradient.
Graph neural networks are explained through energy gradient flow and framelet decomposition.
problem Understanding and improving graph neural networks.
method Viewing framelet-based models as gradient flows of energy, proposing a generalized energy via framelet decomposition.
result The proposed model leads to more flexible dynamics, enhancing graph neural networks.
New attacks exploit neural network energy and latency, increasing costs by 10-200x.
problem Energy and latency vulnerabilities in neural networks.
method Crafted sponge examples to maximize energy and latency.
result Adversaries can increase energy consumption by 10-200x and delay decisions.
This paper tackles risk-aware energy scheduling for MEC networks with microgrids.
problem Risk in energy demand and supply for MEC networks powered by microgrids.
method Formulated an optimization problem with CVaR for energy consumption and generation, analyzed using a multi-agent stochastic game, derived solution with MADRL-based A3C algorithm.
result Significant performance gain by considering CVaR for high accuracy energy scheduling.
Paper explores sustainable machine learning with energy harvesting.
problem Energy-efficient distributed machine learning in resource-constrained devices.
method Developed a practical learning framework with theoretical guarantees for distributed learning over energy-harvesting devices.
result Demonstrated significant performance improvement over non-harvesting benchmarks.
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
problem Dependency of CNNs with skip connections on the number of parameters.
method Examines the Bayesian free energy of CNNs with and without skip connections.
result The upper bound of free energy of Bayesian CNN with skip connections does not depend on overparametrization.
Deep neural networks improve free energy calculations for peptide conformations.
problem Challenges in developing suitable mappings for free energy perturbation.
method Adapted machine learning approach to train deep neural networks for mapping between Boltzmann distributions.
result Accurate free energy differences calculated between thermodynamic states with spring centers separated by 1 Å and sometimes 2 Å.
Researchers find optimal configurations of complex knots and links.
problem Finding the most efficient configurations of complex knots and links.
method Minimizing Möbius and Minimum Distance energies by describing them with a small number of free parameters.
result Optimal geometries for Hopf links, Borromean rings, and chain links are found.
LNNs learn Lagrangians without canonical coordinates, conserving energy and relativity.
problem Neural networks struggle to learn physical symmetries like conservation laws.
method Lagrangian Neural Networks (LNNs) parameterize arbitrary Lagrangians using neural networks.
result LNNs conserve energy and relativity in complex systems.
In this paper, we analyze energy-harvesting adaptive diffusion networks for a distributed estimation problem. In order to wisely manage the available energy resources, we propose a scheme where a censoring algorithm is jointly applied over the diffusion strategy. An energy-aware variation of a diffusion algorithm is us…
Maximin UCB algorithm optimizes energy harvesting for sensor networks.
problem Optimizing energy harvesting for sensor nodes in varying environments.
method Modeling as Maximin Multi-Armed Bandits and proposing Maximin UCB algorithm.
result Maximin UCB algorithm achieves performance guarantees similar to UCB1.
EDCompress optimizes energy efficiency of CNN models on edge devices.
problem Low energy consumption for edge devices with diverse dataflow types.
method Energy-aware model compression using reinforcement learning.
result Improves energy efficiency by 20X, 17X, 37X in various networks.
Deep learning depends on tuning layers near critical points.
problem Understanding how deep learning architectures depend on tuning parameters.
method Random energy approach to analyze statistical dependence in deep belief networks.
result Statistical dependence can propagate only if layers are tuned near critical points.
TPBS models improve robustness to overfitting with localized Dirichlet energy regularization.
problem Global Dirichlet energy-based regularization fails for TPBS models due to perfect interpolation.
method Propose local Dirichlet energy regularization and two inference estimators.
result TPBS models outperform neural networks in overfitting regimes and maintain competitive performance otherwise.
Adversarial attacks can fool ML energy theft detection models.
problem Vulnerability of ML-based energy theft detection models to adversarial attacks.
method Design of an adversarial measurement generation algorithm.
result ML models can be significantly fooled by adversarial attacks, reducing their detection accuracy.
Many practical machine learning tasks employ very deep convolutional neural networks. Such large depths pose formidable computational challenges in training and operating the network. It is therefore important to understand how fast the energy contained in the propagated signals (a.k.a. feature maps) decays across laye…
Structured prediction energy networks (SPENs; Belanger & McCallum 2016) use neural network architectures to define energy functions that can capture arbitrary dependencies among parts of structured outputs. Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables a…
Structured Prediction Energy Networks (SPENs) are a simple, yet expressive family of structured prediction models (Belanger and McCallum, 2016). An energy function over candidate structured outputs is given by a deep network, and predictions are formed by gradient-based optimization. This paper presents end-to-end lear…
Improves energy efficiency of neuromorphic hardware by optimizing memory organization and encoding schemes.
problem Energy inefficiency in neuromorphic hardware, especially in digital accelerators.
method Synthesized controller and memory for different encoding schemes, introduced functional encoding for structured connectivity.
result Functional encoding offers a 58% reduction in energy for weight updates in convolutional layers.
Modern Hopfield networks help prevent forgetting in generative models after task changes.
problem How to prevent forgetting in generative models after task changes.
method Introduce intrinsic forgetting as an increase in Hopfield energy after task change, analyze memory replay effectiveness, and validate predictions in experiments.
result High-energy, outlier-like samples are more forgettable than cluster-like samples, and energy-based selection of replay samples mitigates forgetting.
We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce con…
New theory shows predictive coding makes learning landscape easier to navigate.
problem Understanding the impact of predictive coding's inference procedure on learning efficiency.
method Analyzed the geometry of the energy landscape of deep linear networks, proving many non-strict saddles become strict in the equilibrated energy.
result All highly degenerate (non-strict) saddles of the loss become strict in the equilibrated energy, suggesting a more robust learning landscape.
The article analyzes the stability of a curve shortening flow for planar networks.
problem Stability analysis of anisotropic curve shortening flow for planar networks.
method Used Lojasiewicz-Simon gradient inequality to derive stability results.
result For initial data close to an energy minimizer, the flow exists globally and converges to a different energy minimum.
"How much energy is consumed for an inference made by a convolutional neural network (CNN)?" With the increased popularity of CNNs deployed on the wide-spectrum of platforms (from mobile devices to workstations), the answer to this question has drawn significant attention. From lengthening battery life of mobile device…
Improved generative models using overparametrized shallow neural networks.
problem Improving generative models for data with hidden low-dimensional structure.
method Using energy-based models with overparametrized shallow neural networks as approximators.
result Models trained in the 'active' regime outperform those in the 'lazy' or kernel regime, leading to better adaptivity to hidden structure.
Designing energy-efficient networks is of critical importance for enabling state-of-the-art deep learning in mobile and edge settings where the computation and energy budgets are highly limited. Recently, Liu et al. (2019) framed the search of efficient neural architectures into a continuous splitting process: it itera…
Dual training method for EBMs with overparametrized neural networks.
problem Training EBMs with non-convex energies is challenging.
method Derive variational principles and dual GDA algorithm for feature-learning regime.
result Dual GDA algorithm performs best with similar time scales for features and particles.
Traditional energy-based learning models associate a single energy metric to each configuration of variables involved in the underlying optimization process. Such models associate the lowest energy state to the optimal configuration of variables under consideration, and are thus inherently dissipative. In this paper we…
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities. The recognition is realized using a two-level hierarchy of Long Short-Term Memory (LSTM) networks, forming a feed-forward deep architecture, which can be tra…
Deep Neural Networks (DNNs) are increasingly deployed in highly energy-constrained environments such as autonomous drones and wearable devices while at the same time must operate in real-time. Therefore, reducing the energy consumption has become a major design consideration in DNN training. This paper proposes the fir…
This paper presents a novel data-driven technique based on the spatiotemporal pattern network (STPN) for energy/power prediction for complex dynamical systems. Built on symbolic dynamic filtering, the STPN framework is used to capture not only the individual system characteristics but also the pair-wise causal dependen…
Paper proposes MAMRL for efficient energy dispatch in self-powered edge computing systems.
problem High energy consumption in self-powered edge computing systems.
method Developed a semi-distributed data-driven MAMRL framework to solve a two-stage linear stochastic programming problem.
result The proposed MAMRL framework reduces up to 11% non-renewable energy usage and 22.4% energy cost.
We explore the energy landscape of a simple neural network. In particular, we expand upon previous work demonstrating that the empirical complexity of fitted neural networks is vastly less than a naive parameter count would suggest and that this implicit regularization is actually beneficial for generalization from fit…
POET enables large neural network training on tiny devices with reduced energy.
problem Training large neural networks on memory-limited edge devices.
method Jointly optimizes rematerialization and paging for memory reduction, formulating an MILP for energy-efficient training.
result POET trains ResNet-18 and BERT within Cortex-M memory constraints, outperforming current methods in energy efficiency.
A learning rule for first-spike times in neural networks reduces energy consumption and reaction times.
problem Energy efficiency and reaction time in neuromorphic systems.
method Derivation of a learning rule for first-spike times in leaky integrate-and-fire neurons, using only input and output spike times.
result Demonstrated that the approach can implement error backpropagation in hierarchical spiking networks and is capable of harnessing neuromorphic system's speed and energy characteristics.
Modeling wind dynamics in Saudi Arabia using deep learning and stochastic PDEs.
problem Accurately modeling spatio-temporal wind patterns in a large, diverse, and understudied region.
method Energy distance-based spatial reduction, sparse stochastic Echo State Network, non-stationary stochastic PDE reconstruction.
result Produces more accurate wind speed and energy forecasts, saving $1 million annually.
Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.
problem Understanding the generalization performance of deep ReLU neural networks.
method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.
result Bayesian free energy is bounded even in overparametrized deep ReLU networks.
Deep learning boosts building energy load forecasting.
problem Short-term load forecasting in buildings.
method Stacked Boosters Network architecture with sparse interactions, parameter sharing, and equivariant representations.
result Outperforms state-of-the-art models in short-term load forecasting tasks.
EB-GFN models discrete data with amortized MCMC sampling.
problem Probabilistic modeling of high-dimensional discrete data.
method EB-GFN combines GFlowNets with energy-based models for efficient sampling.
result EB-GFN effectively models various discrete data tasks.
DeepONet learns operators for PDEs with varying parameters and initial conditions.
problem Learning operators for partial differential equations with different parameters or initial conditions.
method DeepONet uses a Branch net and Trunk net to minimize error between evaluated and expected outputs, incorporating a scalar auxiliary variable approach for energy dissipation.
result DeepONet can accurately approximate operators for PDEs with varying parameters or initial conditions.
A DRL-based strategy improves vehicle tracking accuracy while saving energy.
problem Enhancing vehicle tracking accuracy in WSNs without increasing energy consumption.
method Decentralized strategy with dynamic reinforcement learning to adjust sensing areas.
result Simulation results demonstrate superior performance of DRL-aided design.
NNs accurately predict energy eigenvalues and other physical phenomena in 1D quantum mechanics.
problem Understanding how neural networks interpret physics.
method Training NNs to predict energy eigenvalues from potentials and testing their ability to generalize.
result NNs can predict physical phenomena not learned during training, indicating a new way of understanding physics.
As convolutional neural networks (CNNs) enable state-of-the-art computer vision applications, their high energy consumption has emerged as a key impediment to their deployment on embedded and mobile devices. Towards efficient image classification under hardware constraints, prior work has proposed adaptive CNNs, i.e., …