arXiv research

sGPO optimizes RLVR training by balancing inference FLOPs and training efficiency.

Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the …

New method grows deep networks efficiently by dynamically pruning and growing layers.

There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporti…

This paper introduces channel gating, a dynamic, fine-grained, and hardware-efficient pruning scheme to reduce the computation cost for convolutional neural networks (CNNs). Channel gating identifies regions in the features that contribute less to the classification result, and skips the computation on a subset of the …

RicciNets prunes neural networks by removing edges of low importance based on Ricci curvature, reducing FLOPs by 35%.

Funnel-Transformer reduces computation by compressing sequence data.

In this paper we propose the macroblock scaling (MBS) algorithm, which can be applied to various CNN architectures to reduce their model size. MBS adaptively reduces each CNN macroblock depending on its information redundancy measured by our proposed effective flops. Empirical studies conducted with ImageNet and CIFAR-…

Paper combines regularization and pruning to reduce FLOPs in DNNs.

Neural networks offer high-accuracy solutions to a range of problems, but are costly to run in production systems because of computational and memory requirements during a forward pass. Given a trained network, we propose a techique called Deep Learning Approximation to build a faster network in a tiny fraction of the …

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structure…

In this paper, we study the behavior of Ricci-flat Kähler metrics on Calabi-Yau manifolds under algebraic geometric surgeries: extremal transitions or flops. We prove a version of Candelas and de la Ossa's conjecture: Ricci-flat Calabi-Yau manifolds related by extremal transitions and flops can be connected by a path c…

Ensembling smaller models can outperform larger models in terms of accuracy and efficiency.

We define relative Gromov-Witten invariants and establish a general gluing theory of pseudo-holomorphic curves for symplectic cutting and contact surgery. Then, we use our general gluing theory to study the change of GW-invariants of Calabi-Yau 3-folds tranform under flops and extremal transitions. We prove a complete …

Active subspace is a model reduction method widely used in the uncertainty quantification community. In this paper, we propose analyzing the internal structure and vulnerability and deep neural networks using active subspace. Firstly, we employ the active subspace to measure the number of "active neurons" at each inter…

This article is concerned with an observation for proving non-existence of canonical Kahler metrics. The idea is to use a rather explicit type of degeneration that applies in many situations. Namely, in a variation on a theme introduced by Ross-Thomas, we consider flops of the deformation to the normal cone. This yield…

Model-based compression is an effective, facilitating, and expanded model of neural network models with limited computing and low power. However, conventional models of compression techniques utilize crafted features [2,3,12] and explore specialized areas for exploration and design of large spaces in terms of size, spe…

We exhibit many examples of closed complex surfaces whose diffeomorphism groups are not simply-connected and contain loops that are not homotopic to loops of symplectomorphisms.

A new framework compresses neural networks using sparse optimization.

FLOP algorithm speeds up causal structure learning for linear models.

Speeds up training and inference by pruning entire channels before training.

This paper gives a description of the full space of Bridgeland stability conditions on the bounded derived category of a contraction algebra associated to a 3-fold flop. The main result is that the stability manifold is the universal cover of a naturally associated hyperplane arrangement, which is known to be simplicia…

Paper reduces neural network complexity for image classification.

Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic…

LLoCa makes any network Lorentz-equivariant, achieving high accuracy and efficiency.

We propose a fast proximal Newton-type algorithm for minimizing regularized finite sums that returns an $ε$-suboptimal point in $\tilde{\mathcal{O}}(d(n + \sqrt{κd})\log(\frac{1}ε))$ FLOPS, where $n$ is number of samples, $d$ is feature dimension, and $κ$ is the condition number. As long as $n > d$, the proposed method…

Finet uses FBN for efficient, lightweight neural networks.

Neural network training is computationally and memory intensive. Sparse training can reduce the burden on emerging hardware platforms designed to accelerate sparse computations, but it can affect network convergence. In this work, we propose a novel CNN training algorithm Sparse Weight Activation Training (SWAT). SWAT …

Recurrent neural networks (RNNs) can model natural language by sequentially 'reading' input tokens and outputting a distributed representation of each token. Due to the sequential nature of RNNs, inference time is linearly dependent on the input length, and all inputs are read regardless of their importance. Efforts to…

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has address…

Estimates MLP expected output without sampling, using fewer FLOPs.

We prove that the metric completion of a canonical Ricci-flat Kahler metric on the nonsingular part of a projective Calabi-Yau variety $X$ with ordinary double point singularities, is a compact metric length space homeomorphic to the projective variety $X$ itself. As an application, we prove a conjecture of Candelas an…

Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise…

Deep neural networks with more parameters and FLOPs have higher capacity and generalize better to diverse domains. But to be deployed on edge devices, the model's complexity has to be constrained due to limited compute resource. In this work, we propose a method to improve the model capacity without increasing inferenc…

STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.

Gradual pruning reduces inference cost by pruning least important channels during training.

DNAS disentangles neural architecture search for better interpretability and performance.

This work proposes a method to learn sparse representations that are more efficient for large-scale data retrieval.

We introduce the fibred toric varieties as equivariant $\mathbb{C}P^r$ bundles over lower dimensional toric varieties. An equivalent characterization is that the natural morphisms on them degenerate to bundle projections in the context of variation of toric varieties as GIT quotients. Our main observation is that these…

AlphaNet improves supernets training with alpha-divergence.

In 1995 D. Joyce explicitly constructed a series of self-dual metrics with torus action on the connected sums of complex projective planes. In this paper we explicitly construct the twistor spaces of some of Joyce's self-dual metrics. Starting from a fiber space whose fibers are compact singular toric surfaces, we appl…

This paper optimizes caching and model multiplexing for large model inference.

Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…

Despite the promising results of convolutional neural networks (CNNs), their application on devices with limited resources is still a big challenge; this is mainly due to the huge memory and computation requirements of the CNN. To counter the limitation imposed by the network size, we use pruning to reduce the network …

In recent years, deep neural networks have achieved great success in the field of computer vision. However, it is still a big challenge to deploy these deep models on resource-constrained embedded devices such as mobile robots, smart phones and so on. Therefore, network compression for such platforms is a reasonable so…

SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.

PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.

A new method quantifies feature-map discriminativeness for efficient pruning of deep neural networks.