Researchers enhance EfficientNet models for practical efficiency on Graphcore IPU.
problem Improving practical efficiency of EfficientNet models on high-performance accelerators.
method Group convolutions, proxy-normalized activations, and reduced training resolution.
result Improves practical efficiency for both training and inference on Graphcore IPU.
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performan…
Weather is an important factor affecting transportation and road safety. In this paper, we leverage state-of-the-art convolutional neural networks in labelling images taken by street and highway cameras located across across North America. Road camera snapshots were used in experiments with multiple deep learning frame…
We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, i…
LSQ+ improves quantization of neural nets with Swish activations, achieving state-of-the-art results.
problem Quantization of neural nets with Swish activations, especially negative activations, leads to significant performance loss.
method Introduces learnable scale and offset parameters for asymmetric quantization, and uses MSE-based initialization for quantization parameters.
result Significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations, achieving up to 5.6% gain with W2A2 quantization of EfficientNet-B0.
New method improves ResNet performance without batch normalization.
problem Improving ResNet performance without batch normalization.
method Adapted Weight Standardization to maintain signal propagation.
result Highly performant ResNets achieve state-of-the-art performance on ImageNet.
New algorithms improve neural network generalization by finding flat minima.
problem Finding better generalization in neural networks through flat minima.
method Developed Entropy-SGD and Replicated-SGD algorithms to maximize flatness in the loss function.
result Consistently improved generalization error for various deep learning architectures.
SliceOut speeds up deep learning training without sacrificing accuracy.
problem Frequent model re-training and large model training workloads in deep learning.
method SliceOut uses dropout-inspired scheme to drop contiguous sets of units at random, leveraging GPU memory layout.
result 10-40% speedups and memory reduction with minimal accuracy loss.
CoolMomentum combines momentum and Simulated Annealing for deep learning optimization.
problem Global optimization of non-convex functions in deep learning.
method Discretized Langevin dynamics with Simulated Annealing.
result CoolMomentum achieves high accuracy on Resnet-20 on Cifar-10 and Efficientnet-B0 on Imagenet.
Neural network pruning reduces the computational cost of an over-parameterized network to improve its efficiency. Popular methods vary from ℓ1-norm sparsification to Neural Architecture Search (NAS). In this work, we propose a novel pruning method that optimizes the final accuracy of the pruned network and distil…
New method trains deep ResNets without normalization, achieving state-of-the-art performance.
problem Training deep ResNets without normalization layers leads to instability and lower accuracy.
method Adaptive gradient clipping and Normalizer-Free ResNets design.
result Normalizer-Free ResNets achieve 86.5% top-1 accuracy on ImageNet, matching EfficientNet-B7.
We extend quantization-aware training to extreme model compression.
problem Maximizing model accuracy with minimal model size.
method Quantize a random subset of weights during training, allowing unbiased gradients through other weights.
result Established new state-of-the-art compromises between accuracy and model size.
Despite the success of deep neural networks (DNNs), state-of-the-art models are too large to deploy on low-resource devices or common server configurations in which multiple models are held in memory. Model compression methods address this limitation by reducing the memory footprint, latency, or energy consumption of a…
OUI tool detects optimal Weight Decay for DNNs without validation data.
problem Optimal Weight Decay hyperparameter selection for DNNs.
method Overfitting-Underfitting Indicator (OUI) tool.
result OUI correlates with improved generalization and validation scores.
LTP learns per-layer thresholds for efficient pruning of deep networks.
problem Efficiently pruning deep neural networks to reduce computational cost and size.
method LTP learns thresholds via gradient descent, making pruning computationally efficient and scalable.
result LTP achieves competitive compression rates and maintains high accuracy on ImageNet networks.
DONNA rapidly finds optimal neural networks across diverse spaces.
problem Efficient scaling and handling of diverse architectural search-spaces in NAS.
method Three-phase pipeline: accuracy predictor, rapid evolutionary search, and optimal model finetuning.
result 100x faster than MNasNet in finding state-of-the-art architectures on-device.
E2CM uses class means for efficient early exits in neural networks.
problem Efficient early exits in neural networks with low computational cost.
method Early Exit Class Means (E2CM) based on class means of samples, without gradient-based training. result E2CM achieves higher accuracy with fixed training time budget and boosts existing early exit schemes. Automatically designs normalization and activation layers together.
problem Designing normalization and activation layers separately.
method Unified tensor-to-tensor computation graph, low-level mathematical functions, rejection protocols, multi-objective evolution.
result Discovery of EvoNorms with novel structures.
New method for initializing low-rank neural networks improves performance.
problem Training low-rank neural networks efficiently and accurately.
method Inspired by function approximation, proposes a novel low-rank initialization framework.
result Demonstrates significant gap between spectral and low-rank initialization approaches.
Study uses aerial or satellite imagery to improve land price prediction in Thailand.
problem Limited accuracy of traditional land price prediction methods.
method Proposes a similarity-based asset valuation model using Siamese-inspired Neural Network and ensembles deep learning and tree-based models.
result Achieves an AUC of approximately 0.81, outperforming traditional methods.
New activation networks improve model efficiency and performance.
problem Creating hardware-efficient deep learning models.
method Restructurable Activation Networks (RANs) with RAN-explicit and RAN-implicit methods.
result RANs achieve state-of-the-art results with improved hardware efficiency.
Deep learning identifies precipitation clouds from all-sky camera data.
problem Automating cloud warning systems for observatories.
method Deep learning using EfficientNet network.
result Average accuracy of 99% in identifying rainfall potential and 96% in cloud coverage.
LDA improves image classification accuracy with fewer features.
problem Fine-grained image classification with pretrained features.
method Supervised dimensionality reduction with LDA before linear probing.
result LDA improves accuracy over full features in 11 out of 12 configurations.
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationall…