SDAMI enhances interpretable high-dimensional regression with sparse deep learning and footprint principle.
problem Personalized models for small samples and high-dimensional features with interpretability.
method Sparse Deep Additive Model with Interactions (SDAMI) combining sparsity-driven feature selection and deep subnetworks.
result SDAMI successfully identifies pure interactions with near-zero false positive rates.
Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate fo…
Carbontracker tracks and predicts training DL models' carbon footprint.
problem Exponential growth in energy consumption for training deep learning models.
method Carbontracker tool for tracking and predicting energy and carbon footprint.
result Promotes responsible computing and encourages energy-efficient deep learning.
Sustainability became the most important component of world development, as countries worldwide fight the battle against the climate change. To understand the effects of climate change, the ecological footprint, along with the biocapacity should be observed. The big part of the ecological footprint, the carbon footprin…
This paper analyzes energy and carbon footprints in distributed and federated learning.
problem High energy costs and carbon emissions in centralized AI methods.
method A novel framework quantifying energy and carbon footprints in vanilla and consensus-based FL methods.
result Optimal bounds and operational points for green FL designs and sustainability assessment.
Persistent homology reveals geometric features of metric spaces, especially geodesic circles.
problem Detecting geometric features in metric spaces using persistent homology.
method Analyzing algebraic elements (footprints) in persistent homology of metric spaces and subspace.
result Higher-dimensional persistent homology captures lower-dimensional geometric features.
Compact DNNs increase memory footprint and reduce energy efficiency.
problem Designing compact deep neural networks (DNNs) for improved energy efficiency.
method Evaluation of recently proposed compact DNNs on a Tesla P100 GPU.
result Higher number of activations and memory footprint lead to reduced energy efficiency.
A new Input-Output model, called the Multi-Entity Input-Output (MEIO) model, is introduced to estimate the responsibility of entities of an ecosystem on the footprint of each other. It assumed that the ecosystem is comprised of end users, service providers, and utilities. The proposed MEIO modeling approach can be seen…
New GP-VAE model improves scalability and performance.
problem Inability of conventional VAEs to model correlations between data points.
method Principled sparse inference approaches to improve scalability of GP-VAEs.
result New model outperforms existing approaches in runtime and memory usage.
Study analyzes carbon footprint of 1,417 ML models on Hugging Face.
problem Scarce knowledge on measuring and reporting carbon footprint of ML models.
method Repository mining study on Hugging Face Hub API.
result Stalled carbon emissions-reporting models, slight decrease in carbon footprint over 2 years.
We propose a statistical model to understand people's perception of their carbon footprint. Driven by the observation that few people think of CO2 impact in absolute terms, we design a system to probe people's perception from simple pairwise comparisons of the relative carbon footprint of their actions. The formulation…
We construct Peano curves γ:[0,∞)→R2 whose "footprints" γ([0,t]), t>0, have C∞ boundaries and are tangent to a common continuous line field on the punctured plane R2∖{γ(0)}. Moreover, these boundaries can be taken C∞-close to any prescribed smooth family…
Serenity optimizes neural network execution for edge devices by scheduling with optimal memory footprint.
problem Order of nodes in irregular neural networks affects memory footprint, complicating execution under resource constraints.
method Memory-aware compiler using dynamic programming and graph rewriting to find optimal schedules.
result Achieves optimal peak memory and further improves it with graph rewriting, reducing memory usage by 1.68x-1.86x compared to TensorFlow Lite.
This paper optimizes decarbonized indices for financial tracking, balancing risk and environmental impact.
problem Balancing financial performance with environmental responsibilities in the context of climate risks.
method Develops decarbonized indices using mean-VaR and mean-ES optimization methods.
result Optimized indices reduce financial risk and carbon footprint, providing a balanced investment option.
In this paper we propose solving localized multiple kernel learning (LMKL) using LMKL-Net, a feedforward deep neural network. In contrast to previous works, as a learning principle we propose {\em parameterizing} both the gating function for learning kernel combination weights and the multiclass classifier in LMKL usin…
The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a popular class of machine learning models for analyzing sequential data. Their training on modern GPUs, however, is limited by the GPU memory capacity. Our profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model reveal that fea…
Paper proposes a KGE framework that reduces training time and carbon footprint.
problem Efficient KGE learning with reduced computational cost and environmental impact.
method Full batch learning, Orthogonal Procrustes Analysis, non-negative-sampling training.
result Significant reduction in training time and carbon footprint compared to state-of-the-art approaches.
Automates building structural design with reduced mass and carbon footprint.
problem Time-consuming and laborious manual design process for buildings.
method Formulated building structures as graphs, trained end-to-end pipeline with a differentiable simulator.
result Optimal structural designs comparable to GA, with reduced building mass and carbon footprint.
New method predicts political ideology from online activity.
problem Predicting political ideology from digital footprints.
method Statistical learning approaches applied to reddit data.
result Activity in non-political forums can predict political ideology with high accuracy.
This paper proposes an alternative to E2E training for deep networks, reducing memory footprint.
problem High GPUs memory footprint in end-to-end training of deep networks.
method Locally supervised learning with information propagation loss to avoid information collapse.
result The proposed method achieves competitive performance with less than 40% memory footprint compared to E2E training.
SGQuant reduces GNN memory usage without significant accuracy loss.
problem High memory consumption in GNNs limits their applicability on memory-constrained devices.
method Proposes a specialized GNN quantization scheme (SGQuant) with a quantization algorithm, fine-tuning scheme, and multi-granularity strategy.
result SGQuant reduces GNN memory footprint from 4.25x to 31.9x with minimal accuracy loss.
New method scores DAGs by identifying unobserved confounding.
problem Unobserved confounding complicates causal discovery.
method Score-based causal discovery algorithm that accounts for unobserved confounding.
result Sparse linear Gaussian DAGs can be recovered from observed data.
Estimates Mozambique's population using remote sensing and microcensus data.
problem Lack of frequent population estimation due to censuses lacking spatio-temporal resolution.
method Combines remote sensing, microcensus data, and transfer learning with publicly available datasets.
result Population predictions improve with footprint area estimation using transfer learning.
Quantum-inspired tensor network speeds up financial risk assessment.
problem Efficiently pricing multi-asset derivatives in finance.
method Tensor network algorithms for multi-asset options pricing.
result Tensor network approach yields several orders of magnitude speedup.
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
Improves Gaussian process models for large datasets.
problem Complexity and memory limitations in Gaussian process models.
method Combines variational sparse approximation and state-space formulation.
result Significant computational and memory savings for large datasets.
Neural network models are resource hungry. It is difficult to deploy such deep networks on devices with limited resources, like smart wearables, cellphones, drones, and autonomous vehicles. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary…
In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient a…
Study measures investment funds' climate transition risk, finds moderate losses.
problem Measuring the impact of climate transition on investment portfolios.
method Comprehensive framework using geographical, sectoral, company and ISIN-level data.
result Investment funds suffer a moderate 5.7% loss in high transition risk scenario.
NVIDIA cuDNN is a low-level library that provides GPU kernels frequently used in deep learning. Specifically, cuDNN implements several equivalent convolution algorithms, whose performance and memory footprint may vary considerably, depending on the layer dimensions. When an algorithm is automatically selected by cuDNN,…
GENRE retrieves entities autoregressively, improving efficiency and accuracy.
problem Retrieving entities from queries efficiently and accurately.
method Autoregressive generation of entity names, reducing memory footprint and improving context encoding.
result Significantly improved performance on entity disambiguation, linking, and retrieval tasks.
Localized Multidirectional Correction improves non-refusal target-response behavior in foundation models.
problem Controlled post-training refusal suppression in routed MoE and hybrid-MoE foundation models.
method Introduce Localized Multidirectional Correction (LoMC), a support-gated intervention framework.
result Substantially improves non-refusal target-response behavior while maintaining general capability under a compact intervention footprint.
Optimal design portfolios improve energy efficiency and reduce risk in uncertain reservoirs.
problem Uncertain reservoir conditions lead to unstable gas recovery and low resource efficiency.
method Developed optimal portfolios of well designs based on reservoir conditions and probabilities.
result Remarkable reduction in variation and substantial increase in energy efficiency achieved.
New model captures long-range patterns in sequences efficiently.
problem Efficiently capturing long-range patterns in sequential data.
method Inspired by wavelet multiresolution analysis, introduces MultiresLayer with multiresolution convolution.
result State-of-the-art performance on sequence classification and autoregressive density estimation tasks.
Machine learning predicts greenhouse gas emissions for undisclosed companies.
problem Lack of GHG emissions data for most companies.
method Trained machine learning model on disclosed data to estimate emissions.
result Model accurately predicts emissions for undisclosed companies.
We propose a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining control over their memory footprint and computational time. We show the effectiveness of our approach on the task of character lev…
ActNN reduces neural network training memory by 2-bit quantization.
problem Limited memory in training neural networks.
method Randomly quantizes activations to 2 bits, proving convergence and proposing mixed-precision strategies.
result Reduces activation memory footprint by 12x with negligible accuracy loss.
Paper proposes a new method to select memory data for online class-incremental learning.
problem Selecting which buffered images to replay for online class-incremental learning.
method Adversarial Shapley value scoring method to preserve latent decision boundaries.
result Proposed ASER method provides competitive or improved performance compared to state-of-the-art methods.
Modeling wind dynamics in Saudi Arabia using deep learning and stochastic PDEs.
problem Accurately modeling spatio-temporal wind patterns in a large, diverse, and understudied region.
method Energy distance-based spatial reduction, sparse stochastic Echo State Network, non-stationary stochastic PDE reconstruction.
result Produces more accurate wind speed and energy forecasts, saving $1 million annually.
Recognizing written domain numeric utterances (e.g. I need $1.25.) can be challenging for ASR systems, particularly when numeric sequences are not seen during training. This out-of-vocabulary (OOV) issue is addressed in conventional ASR systems by training part of the model on spoken domain utterances (e.g. I need one …
HD algorithm simulates dynamics on random matrix ensembles without generating full matrices.
problem Simulating dynamics on dense random matrix ensembles with high space and time complexity.
method Householder reflectors for adaptive and recursive construction, deferring decisions.
result Significant reductions in runtime and memory footprint for practical T≪n. Bayesian histograms achieve optimal distribution estimation with minimal memory usage.
problem Efficiently estimating distributions with minimal memory footprint.
method Bayesian histograms for distribution estimation under Wasserstein distance.
result Bayesian histograms require fewer bins to achieve minimax optimality, reducing memory usage by a polynomial factor.
BiQGEMM efficiently multiplies quantized DNN weights using lookup tables.
problem Efficiently multiplying quantized DNN weights on CPUs/GPUs with limited memory.
method BiQGEMM pre-computes and stores redundant intermediate results in lookup tables.
result BiQGEMM achieves lower overall computations and higher performance.
Large-scale deep neural networks (DNNs) are both compute and memory intensive. As the size of DNNs continues to grow, it is critical to improve the energy efficiency and performance while maintaining accuracy. For DNNs, the model size is an important factor affecting performance, scalability and energy efficiency. Weig…
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…
Deep neural networks (DNNs) are shown to be promising solutions in many challenging artificial intelligence tasks. However, it is very hard to figure out whether the low precision of a DNN model is an inevitable result, or caused by defects. This paper aims at addressing this challenging problem. We find that the inter…
The wavelet scattering transform is an invariant signal representation suitable for many signal processing and machine learning applications. We present the Kymatio software package, an easy-to-use, high-performance Python implementation of the scattering transform in 1D, 2D, and 3D that is compatible with modern deep …
The study examines label smoothing to improve confidence calibration in fine-tuned LLMs.
problem Improving confidence calibration in fine-tuned large language models (LLMs) after instruction tuning.
method Examine various open-sourced LLMs, label smoothing, and custom kernel design.
result Label smoothing is effective in maintaining confidence calibration but faces challenges in large vocabulary LLMs.