arXiv research

We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks…

Improved CNN training speed with K-FAC on large datasets.

The study shows removing fully connected output layers improves efficiency without sacrificing performance.

Optimal quantization improves dataset distillation for faster training.

ST-STORM separates semantic and appearance features for robust representation learning.

One epoch training yields certifiably robust models.

We present a novel high frequency residual learning framework, which leads to a highly efficient multi-scale network (MSNet) architecture for mobile and embedded vision problems. The architecture utilizes two networks: a low resolution network to efficiently approximate low frequency components and a high resolution ne…

AB dynamically scales gradients to mitigate asynchronous training delays.

Adversarial attacks for image classification are small perturbations to images that are designed to cause misclassification by a model. Adversarial attacks formally correspond to an optimization problem: find a minimum norm image perturbation, constrained to cause misclassification. A number of effective attacks have b…

Unified framework interprets SSL models, revealing biases.

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy o…

In this paper, we propose a novel learning method for image classification called Between-Class learning (BC learning). We generate between-class images by mixing two images belonging to different classes with a random ratio. We then input the mixed image to the model and train the model to output the mixing ratio. BC …

We propose orthogonality as a necessary condition for disentangling aleatoric and epistemic uncertainty.

This paper presents a novel adaptively connected neural network (ACNet) to improve the traditional convolutional neural networks (CNNs) {in} two aspects. First, ACNet employs a flexible way to switch global and local inference in processing the internal feature representations by adaptively determining the connection s…

Deep neural networks perform well on real world data but are prone to adversarial perturbations: small changes in the input easily lead to misclassification. In this work, we propose an attack methodology not only for cases where the perturbations are measured by $\ell_p$ norms, but in fact any adversarial dissimilarit…

Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a gen…

Kernel PCA explains self-attention mechanisms in deep learning models.

OUI tool detects optimal Weight Decay for DNNs without validation data.

Reduced precision computation for deep neural networks is one of the key areas addressing the widening compute gap driven by an exponential growth in model size. In recent years, deep learning training has largely migrated to 16-bit precision, with significant gains in performance and energy efficiency. However, attemp…

Quantized Neural Networks (QNNs) are often used to improve network efficiency during the inference phase, i.e. after the network has been trained. Extensive research in the field suggests many different quantization schemes. Still, the number of bits required, as well as the best quantization scheme, are yet unknown. O…

This paper improves SSL methods using ensemble techniques with data-dependent weighted losses.

Mobile V-MoEs scale down ViTs for resource-constrained vision tasks.

VJE learns latent representations without contrastive learning, providing probabilistic semantics.

Resource is an important constraint when deploying Deep Neural Networks (DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based search approach, which limits the flexibility of network patterns in learned cell structures. Moreover, due to the topology-agnostic nature of existing works, including…

Novel NAS method balances performance and hardware metrics efficiently.

Develops an efficient method for large-scale deep learning problems.

New algorithm PRACTISE accelerates networks with tiny sets, reducing latency by 22%.

PLD distills knowledge using choice-theoretic Plackett-Luce model.

Enhances SSL methods with depth cues for better image understanding.

Integrates momentum into SMoE for improved stability and robustness.

SogCLR uses small batch sizes for global contrastive learning, achieving similar performance to SimCLR.

iREPA shows spatial structure, not global semantic, drives generation performance in REPA.

LeJEPA provides a scalable, theory-driven approach to self-supervised learning.

For the past 5 years, the ILSVRC competition and the ImageNet dataset have attracted a lot of interest from the Computer Vision community, allowing for state-of-the-art accuracy to grow tremendously. This should be credited to the use of deep artificial neural network designs. As these became more complex, the storage,…

DiSK improves DP optimizers by simplifying Kalman filtering for better performance.

E2GC optimizes energy efficiency in DNNs by balancing computational and data movement costs.