On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

169,051 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “resnet-50”

Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 secondscs.LG

74.7 seconds to train ResNet-50 on ImageNet with 2,048 GPUs.

problem Training deep learning models on large mini-batches efficiently.

method Optimization methods applied to distributed deep learning.

result Top-1 validation accuracy of 75.08% with training time of 74.7 seconds.

deep learning distributed training mini-batch accelerated SGD ResNet-50

2019-03-29abs ↗pdf ↗

Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to trainstat.ML

For the past 5 years, the ILSVRC competition and the ImageNet dataset have attracted a lot of interest from the Computer Vision community, allowing for state-of-the-art accuracy to grow tremendously. This should be credited to the use of deep artificial neural network designs. As these became more complex, the storage,…

2017-11-12abs ↗pdf ↗

Improving robustness against common corruptions by covariate shift adaptationcs.LG

Improved model robustness against corruptions using online adaptation.

problem Machine vision models' vulnerability to image corruptions like blurring or compression artefacts.

method Using corrupted images' statistics for unsupervised online adaptation to improve robustness.

result ResNet-50 achieves 62.2% mCE on ImageNet-C with adaptation, improving from 76.7% without.

machine vision robustness covariate shift image corruptions unsupervised adaptation

2020-06-30abs ↗pdf ↗

Hyperflux: Pruning Reveals Importancestat.ML

Hyperflux models pruning as a system to reveal weight importance.

problem Pruning large neural networks to reduce latency and power consumption.

method Introduces Hyperflux, a novel

L_0

method that models pruning as flux and pressure.

result Achieves competitive results with ResNet-50, VGG-19, and DeiT-T/S on various datasets.

network pruning neural networks inference latency power consumption ResNet-50

2025-04-06abs ↗pdf ↗

PruneNet: Channel Pruning via Global Importancecs.LG

PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.

problem Improving deep neural network performance and efficiency through channel pruning.

method PruneNet uses a computationally light-weight optimization step to identify and prune channels based on layer redundancy.

result Pruned ResNet models achieve higher accuracy and better performance than non-pruned models.

channel pruning deep learning neural networks model compression accuracy improvement

2020-05-22abs ↗pdf ↗

Selfie: Self-supervised Pretraining for Image Embeddingcs.LG

Selfie pretrains image embeddings without pixel-level predictions.

problem Training image embeddings efficiently with limited labeled data.

method Contrastive Predictive Coding loss for masked patches, using convolutional and attention mechanisms.

result Significant improvement in ResNet-50 accuracy on ImageNet 224 x 224 with 60 examples per class (5%).

image embedding pretraining contrastive predictive coding masked patches resnet

2019-06-07abs ↗pdf ↗

TF-Replicator: Distributed Machine Learning for Researcherscs.LG

TF-Replicator simplifies distributed ML for researchers.

problem Complex distributed ML code for researchers.

method Abstraction over TensorFlow, data-parallel and model-parallel.

result Strong scalability without distributed systems expertise.

distributed machine learning researchers TensorFlow resnet-50 GAN

2019-02-01abs ↗pdf ↗

Vehicle classification using ResNets, localisation and spatially-weighted poolingcs.CV

We investigate whether ResNet architectures can outperform more traditional Convolutional Neural Networks on the task of fine-grained vehicle classification. We train and test ResNet-18, ResNet-34 and ResNet-50 on the Comprehensive Cars dataset without pre-training on other datasets. We then modify the networks to use …

2018-10-15abs ↗pdf ↗

SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognitioncs.LG

Improved speech recognition with faster training and inference.

problem Training very deep CNNs for speech recognition is difficult.

method Proposed SNDCNN using SELU activations instead of RELU and shortcut connections/BN.

result Achieved similar or lower WER with faster training and inference.

speech recognition deep CNNs SELU shortcut connections batch normalization

2019-10-04abs ↗pdf ↗

Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Acceleratorscs.LG

Campfire simplifies sparse training for CNNs, achieving high accuracy with minimal pruning.

problem Training CNNs with sparse weights to maintain high accuracy and robustness.

method Gradual pruning technique with simplified sparsity structure, exploring various tradeoffs.

result Sparse ResNet-50 and ResNet-50 v1.5 achieve 75% top-1 accuracy with 70% sparsity, <1% accuracy loss.

sparse training CNNs pruning robustness ImageNet

2020-01-09abs ↗pdf ↗

Multi-Label Product Categorization Using Multi-Modal Fusion Modelscs.LG

Study improves product categorization on Amazon using multi-modal fusion.

problem Multi-label product categorization in e-commerce.

method Late fusion of image, description, and title modalities using modified CNN and ResNet-50 models.

result Tri-modal late fusion model achieved an

F_1

score of 88.2%, significantly better than single modal models.

e-commerce product categorization multi-modal fusion CNN ResNet-50

2019-06-30abs ↗pdf ↗

Adversarial Defense based on Structure-to-Signal Autoencoderscs.LG

Adversarial attack methods have demonstrated the fragility of deep neural networks. Their imperceptible perturbations are frequently able fool classifiers into potentially dangerous misclassifications. We propose a novel way to interpret adversarial perturbations in terms of the effective input signal that classifiers …

2018-03-21abs ↗pdf ↗

A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscs.LG

Paper develops a statistical framework for quantized training of deep neural networks.

problem Lack of theoretical understanding of gradient quantization in FQT.

method Presented a statistical framework for analyzing FQT algorithms, viewing quantized gradient as a stochastic estimator of QAT gradient.

result Developed two novel gradient quantizers with smaller variance than existing per-tensor quantizer.

deep neural networks quantized training statistical framework gradient quantization ResNet-50

2020-10-27abs ↗pdf ↗

Mish: A Self Regularized Non-Monotonic Activation Functioncs.LG

Mish is a new activation function that improves neural network performance.

problem Improving the performance and training dynamics of neural networks.

method Mish is a self-regularized non-monotonic activation function defined as

f(x)=x anh(softplus(x))

. It outperforms other functions on benchmarks like ImageNet-1k and MS-COCO.

result Mish outperforms Leaky ReLU and ReLU on benchmarks like MS-COCO and ImageNet-1k, respectively, with comparable network parameters.

activation function neural networks Mish YOLOv4 ResNet-50

2019-08-23abs ↗pdf ↗

Trust Region Based Adversarial Attack on Neural Networkscs.LG

Efficient trust region based attacks reduce adversarial perturbation time significantly.

problem Vulnerability of neural networks to adversarial perturbations requiring time-consuming tuning.

method Trust region optimization for efficient adversarial attack computation.

result Significant speed up of up to 37 times for VGG-16 model on a Titan Xp GPU.

adversarial attacks trust region optimization neural networks cifar-10 imagenet

2018-12-16abs ↗pdf ↗

DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learningcs.CV

Robotic weed control has seen increased research of late with its potential for boosting productivity in agriculture. Majority of works focus on developing robotics for croplands, ignoring the weed management problems facing rangeland stock farmers. Perhaps the greatest obstacle to widespread uptake of robotic weed con…

2018-10-09abs ↗pdf ↗

SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processorscs.LG

SparseTrain uses dynamic sparsity in training deep neural networks on CPUs.

problem Training deep neural networks efficiently on general-purpose processors.

method Exploits dynamic zeros introduced by ReLU in feature maps and gradients.

result Significantly speeds up training on CPUs, up to 1.51x.

Deep Learning Training Efficiency General-Purpose SIMD Dynamic Sparsity ReLU

2019-11-22abs ↗pdf ↗

Holistic Filter Pruning for Efficient Deep Neural Networkscs.LG

Holistic Filter Pruning reduces DNN complexity efficiently.

problem Redundant parameters in deep neural networks.

method Holistic Filter Pruning (HFP) for efficient DNN training.

result Achieves state-of-the-art performance with 60% reduction in multiplications on ImageNet.

Deep Neural Networks Filter Pruning Efficient Training CIFAR-10 ImageNet

2020-09-17abs ↗pdf ↗

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factorscs.LG

Rank-1 BNNs improve efficiency and scalability of Bayesian neural nets.

problem Underfitting and lack of scalability in Bayesian neural networks.

method Propose a rank-1 parameterization of BNNs and use mixture approximate posteriors.

result Rank-1 BNNs achieve state-of-the-art performance across various datasets.

Bayesian neural networks rank-1 parameterization deep ensembles mixture approximate posteriors scalability

2020-05-14abs ↗pdf ↗

Can Biases in ImageNet Models Explain Generalization?cs.CV

Study reveals biases in ImageNet models are not sufficient for generalization.

problem Understanding and improving generalization of image classification models.

method Large-scale study of 48 ImageNet models trained via various methods.

result Biases identified in ImageNet models do not fully explain generalization.

ImageNet Biases Generalization Neural Networks ResNet-50

2024-04-01abs ↗pdf ↗

Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrumcs.CV

New method uses eigenspectrum to optimize object detector architectures.

problem Understanding the effects of ImageNet pre-training on object detectors.

method Analysis of eigenspectrum dynamics of feature maps in object detectors.

result Object detectors trained from scratch and ImageNet pre-trained models behave differently.

object detection pre-training eigenspectrum Faster R-CNN Mask R-CNN

2019-09-09abs ↗pdf ↗

Real-time Person Re-identification at the Edge: A Mixed Precision Approachcs.LG

MobileNet-v2 improves re-ID on edge devices with mixed precision.

problem Real-time person re-identification in edge devices with limited resources.

method Mixed precision training (FP32 and FP16) for MobileNet-v2 on edge nodes.

result Mixed precision training improves inference throughput and reduces power consumption.

person re-identification MobileNet-v2 mixed precision real-time edge devices

2019-08-19abs ↗pdf ↗

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescs.LG

Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely used in training large-scale deep neural networks. Although using larger mini-batch sizes can improve the system scalability by reducing the communication-to-computation ratio, it may hurt the generalization ability of the models…

2018-07-30abs ↗pdf ↗

Adversarial AutoAugmentcs.CV

Adversarial AutoAugment improves image classification with less computation.

problem Improving deep neural network generalization through data augmentation.

method Adversarial method to optimize augmentation policies without retraining the target network.

result Significant performance improvements on ImageNet and CIFAR-10.

data augmentation adversarial training deep learning image classification generalization

2019-12-24abs ↗pdf ↗

Automatic image-based identification and biomass estimation of invertebratescs.CV

Automated system identifies and counts insects from images.

problem Manual sorting and identification of insect samples is time-consuming and limits biodiversity mapping.

method Robot-enabled image-based identification machine using CNNs.

result Classification accuracy of

0.980

for initial dataset.

invertebrates image-based identification biomonitoring CNNs

2020-02-05abs ↗pdf ↗

Adjoined Networks: A Training Paradigm with Applications to Network Compressioncs.LG

Adjoined Networks trains both base and compressed networks together for efficient model compression.

problem Efficiently compressing deep neural networks while maintaining accuracy.

method Adjoined Networks (AN) trains both a base network and a smaller compressed network simultaneously, sharing parameters.

result AN achieves 71.8% top-1 accuracy with 1.8M parameters and 1.6 GFLOPs on ImageNet.

model compression neural networks knowledge distillation CNN computer vision

2020-06-10abs ↗pdf ↗

Big Self-Supervised Models are Strong Semi-Supervised Learnerscs.LG

Big models pretrain and fine-tune for semi-supervised learning on ImageNet.

problem Learning from few labeled examples with a large amount of unlabeled data.

method Unsupervised pretraining of a big ResNet model followed by supervised fine-tuning and distillation.

result 73.9% ImageNet top-1 accuracy with just 1% of labels (

\le

13 labeled images per class).

semi-supervised learning ImageNet big models ResNet pretraining

2020-06-17abs ↗pdf ↗

Understanding intermediate layers using linear classifier probesstat.ML

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself. This helps us better understand …

2016-10-05abs ↗pdf ↗

Image Classification at Supercomputer Scalecs.LG

Paper discusses optimizing deep learning models for supercomputers.

problem Training deep learning models at high petaFLOPS scale.

method Three systems-related optimizations: distributed batch normalization, input pipeline, and 2-D torus all-reduce.

result Trained ResNet-50 on ImageNet to 76.3% accuracy in 2.2 minutes.

deep learning supercomputing TPU batch normalization input pipeline

2018-11-16abs ↗pdf ↗

Compressing the Input for CNNs with the First-Order Scattering Transformcs.CV

We study the first-order scattering transform as a candidate for reducing the signal processed by a convolutional neural network (CNN). We show theoretical and empirical evidence that in the case of natural images and sufficiently small translation invariance, this transform preserves most of the signal information nee…

2018-09-27abs ↗pdf ↗

Back to the Basics: Revisiting Out-of-Distribution Detection Baselinescs.LG

A simple method flags images as out-of-distribution based on their distance to nearest neighbors.

problem Detecting images not aligned with a trained model's in-distribution data.

method Flag images as OOD if their average distance to K nearest neighbors is large in the classifier's representation space.

result Simple methods can outperform more complex ones when considering learned representations.

out-of-distribution image detection representation learning nearest neighbors classifier

2022-07-07abs ↗pdf ↗

Flexible Deep Neural Network Processingcs.NE

The recent success of Deep Neural Networks (DNNs) has drastically improved the state of the art for many application domains. While achieving high accuracy performance, deploying state-of-the-art DNNs is a challenge since they typically require billions of expensive arithmetic computations. In addition, DNNs are typica…

2018-01-23abs ↗pdf ↗

Linear Mode Connectivity and the Lottery Ticket Hypothesiscs.LG

Study how neural networks optimize to stable linearly connected regions.

problem Understanding how neural networks converge to stable solutions under different training conditions.

method Investigate the stability of neural networks to SGD noise and apply it to iterative magnitude pruning.

result Subnetworks that reach full accuracy must be stable to SGD noise, either at initialization or early in training.

neural networks stability SGD noise lottery ticket hypothesis iterative magnitude pruning

2019-12-11abs ↗pdf ↗

Stochastic Gradient Push for Distributed Deep Learningcs.LG

SGP combines PushSum with stochastic gradient updates for robust distributed deep learning.

problem Synchronization issues in distributed deep learning.

method Stochastic Gradient Push (SGP) using PushSum for approximate distributed averaging.

result SGP converges to a stationary point at the same rate as SGD and achieves consensus.

distributed deep learning SGP PushSum stochastic gradient consensus

2018-11-27abs ↗pdf ↗

26ms Inference Time for ResNet-50: Towards Real-Time Execution of all DNNs on Smartphonecs.LG

CADNN optimizes DNN execution on smartphones for real-time inference.

problem Executing Deep Neural Networks on mobile devices with low latency and high accuracy.

method Advanced model compression and architecture-aware optimization.

result CADNN outperforms state-of-the-art frameworks in DNN execution on mobile devices.

model compression architecture optimization Deep Neural Networks mobile devices real-time inference

2019-05-02abs ↗pdf ↗

EdgeFool: An Adversarial Image Enhancement Filtercs.LG

EdgeFool generates adversarial images to mislead classifiers.

problem Misleading classifiers with adversarial images.

method Trains a fully convolutional neural network to generate perturbations that enhance image details and mislead classifiers.

result EdgeFool outperforms other adversarial methods on various classifiers and datasets.

adversarial examples image enhancement deep learning classification perturbation

2019-10-27abs ↗pdf ↗

Structured Probabilistic Pruning for Convolutional Neural Network Accelerationcs.LG

In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…

2017-09-20abs ↗pdf ↗

Don't Decay the Learning Rate, Increase the Batch Sizecs.LG

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reache…

2017-11-01abs ↗pdf ↗

Momentum Improves Normalized SGDcs.LG

Improved analysis shows momentum in SGD reduces batch size needs for non-convex objectives.

problem Reducing batch size requirements for SGD in non-convex optimization.

method Normalized SGD with momentum, adaptive method for small gradient variance.

result Normalized SGD with momentum achieves

ε

-critical points in

O(1/ε^{3.5})

iterations.

SGD momentum non-convex optimization normalized adaptive methods

2020-02-09abs ↗pdf ↗

Bombus Species Image Classificationcs.CV

Image classification system identifies bumble bee species from images.

problem Manual identification of bumble bee species is time-consuming and requires expert knowledge.

method Transfer learning using Inception, VGG16, VGG19, and ResNet models.

result Inception and VGG classifiers achieved up to 23% accuracy for single species identification.

bumble bees image classification transfer learning accuracy data limitations

2020-06-09abs ↗pdf ↗

The Difficulty of Training Sparse Neural Networkscs.LG

Sparse neural networks training is difficult due to optimization failures and energy landscape issues.

problem Training sparse neural networks leads to suboptimal solutions and optimization failures.

method Investigated optimization dynamics and energy landscape in sparse neural networks.

result Sparse neural networks have a linear path with a monotonically decreasing objective from initialization to a good solution, but not from a bad solution.

sparse neural networks optimization failures energy landscape training difficulties ImageNet-2012

2019-06-25abs ↗pdf ↗

PETRA: Parallel End-to-end Training with Reversible Architecturescs.LG

PETRA enables parallel training of deep models with reversible architectures.

problem Challenges in parallelizing deep model training.

method Introduces PETRA, a novel approach for parallelizing gradient computations in reversible architectures.

result Achieves competitive accuracies on CIFAR-10, ImageNet32, and ImageNet using ResNet models.

reversible architectures parallel training deep learning memory savings gradient computations

2024-06-04abs ↗pdf ↗

Prodigy: An Expeditiously Adaptive Parameter-Free Learnercs.LG

Prodigy estimates learning rate without tuning, improving convergence.

problem Estimating optimal learning rate in adaptive methods.

method Prodigy modifies D-Adaptation to estimate distance to solution

D

result Prodigy improves convergence rate by a factor of

O(\sqrt{\log(D/d_0)})

learning rate adaptive methods AdaGrad Adam convergence

2023-06-09abs ↗pdf ↗

An Automated Deep Learning Approach for Bacterial Image Classificationq-bio.QM

Deep learning automates bacterial image classification.

problem Manual bacterial classification is time-consuming and error-prone.

method ResNet-50 pre-trained CNN architecture with transfer learning.

result Average classification accuracy of 99.2%.

deep learning bacterial classification image recognition transfer learning CNN

2019-12-04abs ↗pdf ↗

FRODO: Free rejection of out-of-distribution samples: application to chest x-ray analysiscs.LG

FRODO method rejects out-of-distribution chest x-ray images with high accuracy.

problem Rejecting out-of-distribution samples in medical image analysis.

method Uses feature activations and Mahalanobis distance to measure distribution mismatch.

result Achieves an AUC score of 0.99 in classifying chest x-ray images as in or out of distribution.

out-of-distribution chest x-rays medical imaging feature activations Mahalanobis distance

2019-07-02abs ↗pdf ↗

Dissecting Pruned Neural Networkscs.LG

Pruning neural networks reduces parameters without sacrificing interpretability.

problem Reducing unnecessary structure in neural networks to improve efficiency.

method Examined the effect of pruning on the number of hidden units learning disentangled representations.

result Pruning does not harm interpretability until a significant portion of parameters are removed.

neural networks pruning interpretability disentangled representations hidden units

2019-06-29abs ↗pdf ↗

Restricting the Flow: Information Bottlenecks for Attributionstat.ML

New method uses information bottleneck to quantify variable importance in neural networks.

problem Quantifying the importance of individual input variables in neural networks.

method Restricting the flow of information by adding noise to intermediate feature maps.

result Our method outperforms all baselines in five out of six settings.

information bottleneck neural networks attribution machine learning feature importance

2020-01-02abs ↗pdf ↗

Leveraging Filter Correlations for Deep Model Compressioncs.CV

Compression method reduces model size while maintaining accuracy.

problem Deep neural networks are large and resource-intensive.

method Iteratively drops correlated filters, re-optimizes, and finetunes.

result State-of-the-art FLOPs compression rates with excellent predictive performance.

model compression deep learning filter correlation neural networks re-optimization

2018-11-26abs ↗pdf ↗