On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

12 results for “SiLU”

A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Functioncs.LG

This work interprets GELU and related activations via a first-order loss function.

problem Understanding and optimizing activation functions in neural networks.

method Complementary interpretation using the Gaussian first-order loss function.

result Calibrated or learned uniform-threshold gates are competitive and often outperform GELU, ReLU, and SiLU/Swish.

activation functions GELU ReLU SiLU first-order loss

2026-07-04abs ↗pdf ↗

The limitation of neural nets for approximation and optimizationcs.LG

Study assesses neural nets for optimization problems, highlighting SiLU's effectiveness.

problem Using neural nets for optimization problems, especially for accurate approximations.

method Determined best activation function (SiLU) for nonlinear optimization problems. Analyzed function approximations using neural networks and interpolation/regression models.

result Neural nets can deliver competitive zero- and first-order approximations but underperform on second-order approximations.

neural networks optimization activation functions SiLU approximation

2023-11-21abs ↗pdf ↗

Learning a Single Neuron for Non-monotonic Activation Functionsstat.ML

Study learns a neuron with non-monotonic activation functions.

problem Learning a single neuron with non-monotonic activation functions.

method Gradient descent (GD) with conditions on activation function and input distribution.

result Learnability of non-monotonic activation functions is established without monotonicity assumption.

neuron learning non-monotonic activation gradient descent polynomial time Gaussian distribution

2022-02-16abs ↗pdf ↗

Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscs.LG

Deep networks can approximate various activation functions with modest adjustments.

problem Expressive power of deep neural networks with diverse activation functions.

method Approximation of any activation function in set A by ReLU networks with specific scaling factors.

result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.

Deep Learning Neural Networks Activation Functions ReLU Approximation

2023-07-13abs ↗pdf ↗

Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycs.LG

This paper studies activation sparsity in large language models, finding key trends and implications.

problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.

method Proposes PPL-

p\%

sparsity, analyzes trends with training data, width-depth ratio, and parameter scale.

result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.

activation sparsity large language models Transformer training data width-depth ratio

2024-11-04abs ↗pdf ↗

AGGLIO: Global Optimization for Locally Convex Functionsmath.OC

AGGLIO optimizes non-convex functions with local convexity guarantees.

problem Optimizing non-convex functions with local convexity.

method Stage-wise, graduated optimization technique for locally convex functions.

result Global convergence to the global optimum for non-convex and locally convex objectives.

optimization non-convex functions local convexity stage-wise optimization graduated optimization

2021-11-06abs ↗pdf ↗

Demystifying Oversmoothing in Attention-Based Graph Neural Networkscs.LG

Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.

problem The issue of oversmoothing in attention-based GNNs.

method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.

result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.

oversmoothing attention-based GNNs graph neural networks graph convolutional networks nonlinear dynamical systems

2023-05-25abs ↗pdf ↗

Tight Robustness Certificates and Wasserstein Distributional Attacks for Deep Neural Networkscs.LG

Improved robustness for deep neural networks with tighter bounds and attacks.

problem Loose upper bounds and prohibitive computation in existing adversarial robustness methods.

method Primal approach with exact Lipschitz certificates for ReLU networks and modern architectures, and novel Wasserstein Distributional Attacks.

result Tighter upper bounds and greater flexibility in attack points compared to existing methods.

Adversarial robustness Deep neural networks Wasserstein distributionally robust optimization Exact Lipschitz certificates Wasserstein Distributional Attacks

2025-10-11abs ↗pdf ↗

Temporal Functional Circuits: From Spline Plots to Faithful Explanations in KAN Forecastingcs.LG

Temporal Functional Circuits explain KAN forecasts with interpretable edge functions.

problem Lack of mechanistic explanations in KAN forecasting.

method Transform KAN edge functions into faithful, temporally grounded explanations using a gated residual KAN.

result Gated KAN achieves lower MSE than linear-only models on regime-switching signals.

Kolmogorov-Arnold Networks temporal forecasting interpretable edge functions gate mechanism signal complexity

2026-05-07abs ↗pdf ↗

On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmeticcs.LG

Quantized neural networks can represent all fixed-point functions under certain conditions.

problem Expressive power of quantized neural networks under fixed-point arithmetic.

method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.

result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.

quantized networks fixed-point arithmetic expressive power activation functions round-off errors

2024-08-30abs ↗pdf ↗

Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplesstat.ML

This paper explores adversarial training limits and improves model robustness against norm-bounded perturbations.

problem Understanding and improving adversarial robustness of deep neural networks.

method Systematic study of adversarial training with various factors, including model size, activation functions, and unlabeled data.

result Training robust models that go beyond state-of-the-art results by combining larger models, Swish/SiLU activations, and model weight averaging.

adversarial training robustness norm-bounded perturbations CIFAR-10 CIFAR-100

2020-10-07abs ↗pdf ↗

A Unified and Constructive Framework for the Universality of Neural Networkscs.LG

Unified framework proves neural networks' ability to mimic complex tasks.

problem Lack of a single constructive framework for neural network universality.

method Introduces neural network approximate identity (nAI) and proves it leads to universality.

result Any nAI activation function is universal.

neural networks universality activation functions approximate identity functional analysis

2021-12-30abs ↗pdf ↗