In this paper, a simple, general method of adding auxiliary stochastic neurons to a multi-layer perceptron is proposed. It is shown that the proposed method is a generalization of recently successful methods of dropout (Hinton et al., 2012), explicit noise injection (Vincent et al., 2010; Bishop, 1995) and semantic has…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,341 papers · 148 categories
Trend · papers per month
48 results for “Neuron Dropping”
Simplifies deep neural networks by dropping neurons during training.
problem Training deep neural networks is computationally expensive and large.
method Regularisation of network architecture to simplify during training.
result Simpler deep neural networks achieve comparable performance.
AGF explains feature learning in neural networks through alternating steps.
problem Understanding what features neural networks learn and how they learn them.
method AGF is an algorithmic framework that approximates the dynamics of feature learning in two-layer networks.
result AGF provides a unified framework to understand feature learning in neural networks, matching experimental results across various architectures.
This work proposes an attention mechanism for structured sparsity in neural networks.
problem Uncontrolled sparsity in neural networks leads to significant accuracy drops.
method An attention mechanism that controls sparsity intensity and network pruning.
result The proposed method outperforms baseline methods by 6% on CIFAR-10.
BSF algorithm reduces neural network size and selects features efficiently.
problem Neural network size and feature selection optimization.
method Binary Stochastic Filtering (BSF) layer that penalizes information, stochastically passes or drops features.
result Multifold decrease in neural network size and optimal feature selection.
Neuron-specific dropout reduces overfitting and data needs for neural networks.
problem Overfitting and insufficient training data for deep neural networks.
method Compares training and validation passes of a layer, drops targeted neurons based on feature analysis.
result Achieves similar or better testing accuracy with less data, reducing overfitting.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
Adaptive network sparsification improves model compactness and accuracy.
problem Suboptimal network sparsification due to input-independent dropout.
method Dependent variational beta-Bernoulli dropout.
result Significantly more compact networks with consistent accuracy improvements.
A new neural network architecture reduces parameters by 94% while maintaining performance.
problem Reduction of trainable parameters in neural networks.
method Spatially-coupled sparse construction to allocate trainable parameters efficiently.
result Performance comparable to traditional neural networks with 94% fewer parameters.
Tabu Dropout improves performance of standard Dropout by generating more diverse neural network architectures.
problem Preventing co-adaptation of neurons in deep neural networks.
method Integrates a diversification strategy into dropout, marking units from the last forward propagation for re-selection in the current forward propagation.
result Improves performance of standard Dropout on MNIST and Fashion-MNIST datasets.
Theoretical framework for neural network compression using sparsity norms.
problem Understanding and quantifying compressibility and accuracy trade-offs in neural networks.
method Using sparsity-sensitive ℓ_q-norm to characterize compressibility and developing adaptive pruning algorithms.
result Theoretical relationship between network sparsity and compressibility with controlled accuracy degradation.
Deep learning methods have shown great promise in many practical applications, ranging from speech recognition, visual object recognition, to text processing. However, most of the current deep learning methods suffer from scalability problems for large-scale applications, forcing researchers or users to focus on small-…
Greedy selection finds smaller, more accurate subnetworks.
problem Finding smaller, accurate subnetworks in large neural networks.
method Greedy forward selection starting from an empty network.
result Theoretical guarantee of finding subnetworks with lower loss.
Direct feedback alignment reduces data movement in neural networks.
problem Efficiency and energy-efficiency in training large neural networks.
method Sparse feedback matrix for local learning, reducing data movement and compute.
result Orders of magnitude improvement in data movement and 2x improvement in multiply-and-accumulate operations.
MsIGN tackles high-dimensional Bayesian inference using multiscale structure.
problem High-dimensional Bayesian inference challenges due to the curse of dimensionality.
method MsIGN generates samples from coarse to fine scale, minimizing Jeffreys divergence.
result MsIGN outperforms previous approaches in posterior approximation and mode capture.
Regularizing for or against class selectivity in DNNs improves test accuracy.
problem The necessity and sufficiency of class selectivity in DNNs.
method Direct regularization of class selectivity in convolutional neural networks.
result Reducing class selectivity improves test accuracy, while increasing it decreases it.
Round balls minimize liquid drop model volumes ≤ 1.
problem Minimizing volumes in liquid drop models.
method Proved uniqueness of minimizers for small volumes.
result Round balls uniquely minimize volumes ≤ 1.
Drop-Activation reduces overfitting by randomly setting activations to identity.
problem Overfitting in deep learning models.
method Randomly sets activations to identity during training and uses a deterministic network during testing.
result Improves generalization and performance of neural networks.
Empirical study shows GANs overfit and drop modes when training is deterministic.
problem Understanding overfitting and mode drop in GAN training.
method Empirical analysis of GAN training with and without stochasticity.
result GANs overfit and drop modes when training is deterministic.
The study examines mass drop and multiplicity in mean curvature flow.
problem Analyzing mass drop and multiplicity in mean curvature flow.
method Defined Brakke flow with variational inequality, proved mass drop conditions.
result Mass drop and multiplicity one conjecture are equivalent for Brakke flows.
Drop Pruning uses stochastic optimization to prune and recover weights, reducing model size and improving performance.
problem Complexity and inefficiency in pruning deep neural networks.
method Introduces stochastic optimization with 'drop away' and 'drop back' strategies to prune and recover weights.
result Achieves competitive compression performance and accuracy compared to state-of-the-art approaches.
This paper analyzes the configurations of shapes that shows a spacelike liquid drop in Minkowski space deposited over a spacelike plane . We assume the presence of a uniform gravity field directed toward and that the volume of the drop is prescribed. Our interest are the liquid drops that are critical points of …
Neuron Shapley identifies key neurons in deep networks, improving model accuracy and fairness.
problem Identifying responsible neurons in deep networks for better model performance and fairness.
method Neuron Shapley framework quantifies neuron contributions, accounting for interactions.
result Removing just 30 critical filters can destroy model accuracy, revealing network function.
New -neurons improve neural network performance.
problem Improving neural network activation functions.
method Introducing -neurons based on Jackson's -derivatives with stochastic parameters.
result Consistently improved performance over state-of-the-art activation functions.
Describes explaining neurons in deep representations using compositional logical concepts.
problem Interpreting neuron behavior in deep neural networks.
method Identifying compositional logical concepts that closely approximate neuron behavior.
result Compositional explanations provide insights into model performance and allow for adversarial example creation.
SeReNe prunes neurons with low sensitivity to reduce network size.
problem Large neural networks consume too many resources on resource-constrained devices.
method Exploits neural sensitivity as a regularizer to prune neurons with low sensitivity.
result Pruning neurons with low sensitivity achieves competitive compression ratios.
Under-parameterized networks can either copy or average teacher weights, leading to universal optimal solutions.
problem Approximating a teacher network with an under-parameterized student network.
method Analyzing shallow neural networks with erf activation function and unitary teacher weights, proving copy-average configurations are critical points and finding the optimal solution.
result The optimal solution for under-parameterized networks has a universal structure, whether copying or averaging teacher neurons.
Paper proposes HRS to improve neural network robustness without significant accuracy loss.
problem Vulnerability of neural networks to adversarial attacks and performance degradation.
method Hierarchical Random Switching (HRS) for robustness without sacrificing accuracy.
result HRS significantly improves adversarial robustness with minimal accuracy loss.
This research investigates selectively pruning hyper and hypo neurons to improve neural network generalization.
problem Improving neural network generalization to unseen data.
method Investigates pruning hyper and hypo neurons selectively in fully connected layers of CNNs.
result Selective pruning of hyper and hypo neurons improves model performance on out-of-domain data.
SGAD reduces DNN FLOPs by 77% on CIFAR-10 with minimal accuracy loss.
problem Redundancies in DNNs and varying difficulty of input samples.
method Soft-guided adaptive dropping mechanism to inform DNNs of input sample difficulties.
result SGAD reduces FLOPs by 77% on CIFAR-10 with <1% accuracy drop.
Modeling hidden neurons in SNNs using mesoscopic approximations.
problem Underconstrained problem of modeling unobserved neurons in SNNs.
method Coarse-graining and mean-field approximations to derive neuLVM.
result neuLVM can efficiently model large SNNs and recover connectivity parameters.
New dropout technique reduces training time by 20-77%.
problem Efficient training of deep neural networks consumes excessive time and energy.
method Approximate Random Dropout replaces random neuron/synapse dropout with regular patterns to reduce computation and data access.
result Reduces training time by 20-77% with minimal accuracy loss.
Detection and skeletonization of single neurons and tracer injections using topological methodscs.CV
Topological methods improve neuron analysis and tracer injection summary.
problem Traditional methods fail to capture the tree-like structure of neurons.
method Discrete Morse (DM) Theory for neuron skeletonization and consensus tree summarization.
result Significant performance improvements over non-topological methods.
This paper improves neural network compression by using robust low-rank approximations.
problem Neural network compression sensitivity to outliers.
method Introduces robust low-rank approximations using norms (for ) and provides efficient algorithms.
result Achieves up to 28% compression with minimal accuracy loss compared to existing methods.
Developed a BP algorithm for training neural networks with 2nd order neurons.
problem Training neural networks with nonlinear quadratic operations.
method Created a general backpropagation algorithm.
result Validated the generalized BP algorithm through numerical studies.
Quadratic autoencoder improves low-dose CT image denoising.
problem Low-dose CT image denoising.
method Quadratic autoencoder architecture applied to CT denoising.
result Quadratic autoencoder achieves superior denoising performance and efficiency.
Solves internal covariate shift and dying neurons with linked neurons.
problem Internal covariate shift and dying neurons in deep learning.
method Defining linked neurons with two constraints: shared operating point and non-zero gradient.
result Linked neurons effectively solve internal covariate shift and improve training efficiency.
When Neurons Failstat.ML
Neural network robustness evaluated in the presence of failing neurons.
problem Evaluating robustness of neural networks in the face of neuron failures.
method Leveraging Lipschitz continuity of activation functions, calculating Forward Error Propagation.
result Tight bounds on the number of failing neurons before network accuracy is compromised.
We developed a faster method for calculating neuron importance in neural networks.
problem Assigning importance to individual neurons in deep learning models.
method We developed Neuron Integrated Gradients, a scalable implementation of Total Conductance.
result Neuron Integrated Gradients is faster and empirically stronger than DeepLIFT.
BEAN models neuronal correlations to create interpretable representations.
problem Hard interpretation of dense-layer representations in DNNs.
method Inspired by neuroscience, BEAN models neuronal correlations and dependencies.
result BEAN enables formation of interpretable neuronal clusters without sacrificing model performance.
SpaRCe optimizes reservoir computing by learning neuron thresholds to improve performance and prevent forgetting.
problem Improving performance and preventing forgetting in reservoir computing networks.
method Integrates neuron-specific learnable thresholds to optimize sparsity without altering dynamics, learning read-out weights and thresholds via gradient rule.
result Threshold learning improves performance and alleviates catastrophic forgetting.
A Note on Topology Preservation in Classification, and the Construction of a Universal Neuron Gridcs.NE
It will be shown that according to theorems of K. Menger, every neuron grid if identified with a curve is able to preserve the adopted qualitative structure of a data space. Furthermore, if this identification is made, the neuron grid structure can always be mapped to a subset of a universal neuron grid which is constr…
Paper resolves Huisken's conjecture without strict genus drop theorem.
problem Huisken's genericity conjecture in mean curvature flow in R^3.
method Short density-drop theorem + Bamler-Kleiner multiplicity-one theorem for tangent flows.
result Fully resolves Huisken's conjecture without strict genus drop theorem.
Study on liquidation games with market drop-out, proving unique equilibria.
problem Analyzing portfolio liquidation with market drop-out constraints.
method Proves existence and uniqueness of equilibria using integral equations.
result Existence and uniqueness of equilibria in both mean-field and finite-player games.
This work improves DNN interpretability by reducing neuron ambiguity.
problem Lack of interpretability in DNNs, especially in healthcare applications.
method Developed a metric to evaluate neuron consistency, used adversarial examples to identify ambiguous features, and proposed adversarial training to improve consistency.
result Reduced ambiguity of neurons in DNNs, improving interpretability.
A new method to understand neural networks by sampling the 'inverse set' of a neuron.
problem Understanding the internal representation of neurons in neural networks.
method Optimization-based sampling approach to characterize the input space that excites a neuron.
result Inspection of samples reveals regularities that help understand the neuron's representation.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
A new method reduces rank in Frank-Wolfe steps for nuclear norm problems.
problem High rank intermediate iterates in Frank-Wolfe algorithm for nuclear norm problems.
method Rank-drop steps to ensure rank decreases and feasibility.
result Reduced rank of solutions compared to Frank-Wolfe and variants.