arXiv research

Paper combines regularization and pruning to reduce FLOPs in DNNs.

sGPO optimizes RLVR training by balancing inference FLOPs and training efficiency.

There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporti…

Speeds up training and inference by pruning entire channels before training.

Finet uses FBN for efficient, lightweight neural networks.

This paper optimizes caching and model multiplexing for large model inference.

New method grows deep networks efficiently by dynamically pruning and growing layers.

Deep neural networks with more parameters and FLOPs have higher capacity and generalize better to diverse domains. But to be deployed on edge devices, the model's complexity has to be constrained due to limited compute resource. In this work, we propose a method to improve the model capacity without increasing inferenc…

In this paper, we study the behavior of Ricci-flat Kähler metrics on Calabi-Yau manifolds under algebraic geometric surgeries: extremal transitions or flops. We prove a version of Candelas and de la Ossa's conjecture: Ricci-flat Calabi-Yau manifolds related by extremal transitions and flops can be connected by a path c…

Ensembling smaller models can outperform larger models in terms of accuracy and efficiency.

Memory-augmented neural networks (MANNs) are designed for question-answering tasks. It is difficult to run a MANN effectively on accelerators designed for other neural networks (NNs), in particular on mobile devices, because MANNs require recurrent data paths and various types of operations related to external memory a…

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the …

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy o…

We define relative Gromov-Witten invariants and establish a general gluing theory of pseudo-holomorphic curves for symplectic cutting and contact surgery. Then, we use our general gluing theory to study the change of GW-invariants of Calabi-Yau 3-folds tranform under flops and extremal transitions. We prove a complete …

This article is concerned with an observation for proving non-existence of canonical Kahler metrics. The idea is to use a rather explicit type of degeneration that applies in many situations. Namely, in a variation on a theme introduced by Ross-Thomas, we consider flops of the deformation to the normal cone. This yield…

The deployment of deep neural networks in real-world applications is mostly restricted by their high inference costs. Extensive efforts have been made to improve the accuracy with expert-designed or algorithm-searched architectures. However, the incremental improvement is typically achieved with increasingly more expen…

This paper introduces channel gating, a dynamic, fine-grained, and hardware-efficient pruning scheme to reduce the computation cost for convolutional neural networks (CNNs). Channel gating identifies regions in the features that contribute less to the classification result, and skips the computation on a subset of the …

We exhibit many examples of closed complex surfaces whose diffeomorphism groups are not simply-connected and contain loops that are not homotopic to loops of symplectomorphisms.

VMoER improves uncertainty quantification in MoE layers for scalable foundation models.

RicciNets prunes neural networks by removing edges of low importance based on Ricci curvature, reducing FLOPs by 35%.

Funnel-Transformer reduces computation by compressing sequence data.

FLOP algorithm speeds up causal structure learning for linear models.

Recurrent neural networks (RNNs) can model natural language by sequentially 'reading' input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to…

New method infers graph from dependent matrix data.

This paper gives a description of the full space of Bridgeland stability conditions on the bounded derived category of a contraction algebra associated to a 3-fold flop. The main result is that the stability manifold is the universal cover of a naturally associated hyperplane arrangement, which is known to be simplicia…

LLoCa makes any network Lorentz-equivariant, achieving high accuracy and efficiency.

SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structure…

We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $ε$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{κd})\log(\frac{1}ε))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $κ$ is the condition number. As long as $n > d$, the proposed method…

Researchers enhance EfficientNet models for practical efficiency on Graphcore IPU.

A new framework compresses neural networks using sparse optimization.

Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we int…

Estimates MLP expected output without sampling, using fewer FLOPs.

Neural networks offer high-accuracy solutions to a range of problems, but are costly to run in production systems because of computational and memory requirements during a forward pass. Given a trained network, we propose a techique called Deep Learning Approximation to build a faster network in a tiny fraction of the …

We prove that the metric completion of a canonical Ricci-flat Kahler metric on the nonsingular part of a projective Calabi-Yau variety $X$ with ordinary double point singularities, is a compact metric length space homeomorphic to the projective variety $X$ itself. As an application, we prove a conjecture of Candelas an…

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic…

Mobile V-MoEs scale down ViTs for resource-constrained vision tasks.

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has address…

The random subspace method, known as the pillar of random forests, is good at making precise and robust predictions. However, there is not a straightforward way yet to combine it with deep learning. In this paper, we therefore propose Neural Random Subspace (NRS), a novel deep learning based random subspace method. In …

DNAS disentangles neural architecture search for better interpretability and performance.

This work proposes a method to learn sparse representations that are more efficient for large-scale data retrieval.

NPAS trains neural networks with a fixed parameter budget, improving performance and compactness.

We introduce the fibred toric varieties as equivariant $\mathbb{C}P^r$ bundles over lower dimensional toric varieties. An equivalent characterization is that the natural morphisms on them degenerate to bundle projections in the context of variation of toric varieties as GIT quotients. Our main observation is that these…

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank…

AlphaNet improves supernets training with alpha-divergence.

In 1995 D. Joyce explicitly constructed a series of self-dual metrics with torus action on the connected sums of complex projective planes. In this paper we explicitly construct the twistor spaces of some of Joyce's self-dual metrics. Starting from a fiber space whose fibers are compact singular toric surfaces, we appl…

Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…

Despite the promising results of convolutional neural networks (CNNs), their application on devices with limited resources is still a big challenge; this is mainly due to the huge memory and computation requirements of the CNN. To counter the limitation imposed by the network size, we use pruning to reduce the network …