In this work, a method of random parameters generation for randomized learning of a single-hidden-layer feedforward neural network is proposed. The method firstly, randomly selects the slope angles of the hidden neurons activation functions from an interval adjusted to the target function, then randomly rotates the act…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “random activation functions”
Recent seminal work at the intersection of deep neural networks practice and random matrix theory has linked the convergence speed and robustness of these networks with the combination of random weight initialization and nonlinear activation function in use. Building on those principles, we introduce a process to trans…
Optimal AFs minimize RFR test error and sensitivity.
problem Finding optimal AFs for RFR to minimize test error and sensitivity.
method Closed-form solution for AFs minimizing test error and sensitivity under different functional parsimony.
result Optimal AFs can be linear, saturated linear, or Hermite polynomial expressions.
New algorithm trains deep neural networks without global optimization.
problem Training deep neural networks efficiently and without global optimization.
method Uses random complex exponential activation functions and Markov Chain Monte Carlo sampling.
result Consistently attains theoretical approximation rate for residual networks.
Echo state networks with random weights can approximate any continuous system.
problem Approximating continuous dynamical systems using echo state networks.
method Randomly generated internal weights and a sampling procedure for activation functions.
result Echo state networks with random weights can approximate any continuous casual time-invariant operators with high probability.
Study on critical points in random neural networks, revealing three regimes based on activation function.
problem Investigating the expected number of critical points in random neural networks.
method Deriving asymptotic formulas for critical points under infinite-width limit and suitable regularity conditions.
result Three distinct regimes of critical points behavior depending on activation function.
RAFs Ensemble improves neural network uncertainty quantification.
problem Inaccurate predictions in out-of-distribution data.
method Proposes RAFs Ensemble, using random activation functions in neural networks.
result RAFs Ensemble outperforms state-of-the-art methods in uncertainty quantification.
Mean field theory has been successfully used to analyze deep neural networks (DNN) in the infinite size limit. Given the finite size of realistic DNN, we utilize the large deviation theory and path integral analysis to study the deviation of functions represented by DNN from their typical mean field solutions. The para…
This paper characterizes how randomized neural networks generalize well in multi-dimensional tasks.
problem Understanding the generalization of randomized neural networks in multi-dimensional tasks.
method Characterizes RSNs as an IGAM formalized by an optimization problem with a regularization functional and loss.
result RSNs generalize well in multi-dimensional tasks, akin to spline regression under certain conditions.
Study on neural network initialization with shaped infinite depth-and-width networks.
problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
Bounds on Gaussian approximation for neural networks with novel smoothing techniques.
problem Approximating the distribution of wide random neural networks.
method Stein's method, Gaussian smoothing, Laplacian operators, Cameron-Martin space.
result First bounds on Gaussian approximation of wide random neural networks.
Random feature maps improve forecasting of chaotic dynamical systems.
problem Forecasting chaotic dynamical systems with high accuracy.
method Data-driven random feature maps with tanh activation, skip connections, and localization.
result Effective forecasting skill for dynamical systems with dimensions up to 512.
We describe a framework for designing efficient active learning algorithms that are tolerant to random classification noise and are differentially-private. The framework is based on active learning algorithms that are statistical in the sense that they rely on estimates of expectations of functions of filtered random e…
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evalu…
RVFL networks can efficiently approximate Lipschitz functions in L∞ norm.
problem Efficiently approximating Lipschitz continuous functions in L∞ norm.
method Random Vector Functional Link (RVFL) network with ReLU activation functions, proving approximation in L∞ norm.
result An RVFL with ReLU activation functions can approximate Lipschitz continuous functions in L∞ norm.
Study on deep neural networks using branching processes and Mehler's formula.
problem Understanding the mathematical role of activation functions in compositional neural networks.
method Connection between compositional kernels and branching processes via Mehler's formula; new random features algorithm.
result Explicit formulas for eigenvalues of compositional kernels quantify complexity.
Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functionstat.ML
We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in…
Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightsstat.ML
Study Gaussian approximation for deep neural networks with random weights.
problem Understanding the distribution of deep neural networks with random weights.
method Established Gaussian approximation bounds in Wasserstein-1 norm.
result Convergence rates of order for deep networks with proportional layer widths.
The weight initialization and the activation function of deep neural networks have a crucial impact on the performance of the training procedure. An inappropriate selection can lead to the loss of information of the input during forward propagation and the exponential vanishing/exploding of gradients during back-propag…
We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is in . We show that, if the activation function sat…
Hyperplane hashing aims at rapidly searching nearest points to a hyperplane, and has shown practical impact in scaling up active learning with SVMs. Unfortunately, the existing randomized methods need long hash codes to achieve reasonable search accuracy and thus suffer from reduced search speed and large memory overhe…
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Wide neural networks can be closely approximated by Gaussian processes, with rates depending on the activation function's properties.
problem Approximating the behavior of wide neural networks using Gaussian processes.
method Established convergence rates for the central limit theorem in an infinite-dimensional functional space, using a transportation distance metric.
result Explicit convergence rates for neural networks approximated by Gaussian processes, varying based on the activation function's properties.
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcs.LG
DFRot improves LLMs by reducing outlier and massive activation effects.
problem Reducing outlier and massive activation effects in rotated LLMs.
method Weighted loss function and orthogonal Procrustes transforms for rotation matrix refinement.
result DFRot achieves dual free (Outlier-Free and Massive Activation-Free) with significant improvements in perplexity.
The standard method of generating random weights and biases in feedforward neural networks with random hidden nodes, selects them both from the uniform distribution over the same fixed interval. In this work, we show the drawbacks of this approach and propose a new method of generating random parameters. This method en…
REALITrees uses a Rashomon ensemble approach for active learning in sparse decision trees.
problem Active learning reduces labeling costs by selecting informative samples, but current methods often sacrifice model diversity and direct characterization of the hypothesis space.
method REALITrees constructs a committee of all near-optimal sparse decision tree models using a Rashomon Set and a Gibbs posterior to weight them by empirical risk.
result REALITrees outperforms randomized ensembles, especially in noisy environments, by leveraging expanded model multiplicity.
Optimized nonlinearities enhance generalization in random feature models.
problem Improving generalization performance in random feature models.
method Analyzed and defined optimal nonlinearities from Gaussian model parameters.
result Optimized nonlinearities achieve better generalization performance than ReLU.
Study fractal and regular geometry in deep neural networks.
problem Investigate geometric properties of neural networks.
method Analyze boundary volumes of excursion sets for different activations.
result Hausdorff dimension increases with depth for non-regular activations.
FMMNN combines sine activations with multi-component, multi-layer structure for high-frequency function approximation.
problem Effective representation and learning of high-frequency features in neural networks.
method Introduces FMMNN with sine-type activations and multi-component, multi-layer structure.
result FMMNN achieves strong accuracy and favorable convergence on oscillatory function-approximation benchmarks.
Neural networks with random hidden nodes have gained increasing interest from researchers and practical applications. This is due to their unique features such as very fast training and universal approximation property. In these networks the weights and biases of hidden nodes determining the nonlinear feature mapping a…
Dirichlet Active Learningstat.ML
DiAL uses Bayesian Dirichlet random fields for active learning with sparse labels.
problem Active learning with limited labeled data.
method Bayesian Dirichlet random field for feature-conditional class probabilities, calibrating with graph Laplacian.
result Competitive performance in low-label rate graph learning tasks.
Random neural networks with ReLU activations are non-Gaussian processes.
problem Understanding the behavior of neural networks with random initialization and rectified linear units.
method Proving these networks are non-Gaussian processes and deriving their properties.
result These networks can converge to non-Gaussian processes under certain conditions.
Standard ChIP-seq peak calling pipelines seek to differentiate biochemically reproducible signals of individual genomic elements from background noise. However, reproducibility alone does not imply functional regulation (e.g., enhancer activation, alternative splicing). Here we present a general-purpose, interpretable …
Many neural networks use the tanh activation function, however when given a probability distribution as input, the problem of computing the output distribution in neural networks with tanh activation has not yet been addressed. One important example is the initialization of the echo state network in reservoir computing…
The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tra…
A spectral-based analysis of the separation between two-layer neural networks and linear methodsstat.ML
Spectral analysis shows neural networks separate from linear methods in approximating functions.
problem Separating two-layer neural networks from linear methods in function approximation.
method Spectral-based approach using Kolmogorov width and kernel spectrum.
result Upper and lower bounds on separation, explicit hard functions identified.
AL-SPCE improves reliability analysis for complex systems with active learning and SPCE.
problem Efficiently analyzing reliability of complex, computationally expensive models with intrinsic randomness.
method Active learning framework using stochastic polynomial chaos expansions (SPCE) to reduce computational burden.
result AL-SPCE maintains high accuracy in reliability estimates while significantly improving efficiency.
Randomized methods of neural network learning suffer from a problem with the generation of random parameters as they are difficult to set optimally to obtain a good projection space. The standard method draws the parameters from a fixed interval which is independent of the data scope and activation function type. This …
PALS extends PAL for optimizing stochastic simulators efficiently.
problem Optimizing stochastic simulators with high output variance and expensive evaluations.
method Bayesian optimization with probabilistic models, extending PAL for stochastic settings.
result PALS outperforms other methods in optimizing stochastic simulators.
Logistic regression is by far the most widely used classifier in real-world applications. In this paper, we benchmark the state-of-the-art active learning methods for logistic regression and discuss and illustrate their underlying characteristics. Experiments are carried out on three synthetic datasets and 44 real-worl…
Study shows RFRR's effectiveness with nearly orthogonal data in overparameterized settings.
problem Understanding the effectiveness of random feature regression with nearly orthogonal data.
method Investigates RFRR with nearly orthogonal deterministic unit-length input data vectors in the overparameterized regime.
result Shows high-probability non-asymptotic concentration results for RFRR's training, cross-validation, and generalization errors.
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
INP accelerates stochastic simulations using deep Bayesian active learning.
problem Computational expense of stochastic simulations at fine-grained resolution.
method Interactive Neural Process (INP) framework combining spatiotemporal surrogate model and active learning acquisition function.
result STNP outperforms baselines in accelerating stochastic simulations and LIG achieves state-of-the-art for Bayesian active learning.
We propose a new active learning algorithm for parametric linear regression with random design. We provide finite sample convergence guarantees for general distributions in the misspecified model. This is the first active learner for this setting that provably can improve over passive learning. Unlike other learning se…
It is well-known that overparametrized neural networks trained using gradient-based methods quickly achieve small training error with appropriate hyperparameter settings. Recent papers have proved this statement theoretically for highly overparametrized networks under reasonable assumptions. These results either assume…
New method explains GNNs using power iteration clustering.
problem Mysterious mechanism of message passing in GNNs.
method Subspace power iteration clustering (SPIC) models.
result Message passing in GNNs can be understood through power iteration.
Gradient descent dynamics in neural networks show quenching and activation phases.
problem Understanding training dynamics in neural networks.
method Numerical and phenomenological study of gradient descent algorithm for two-layer neural networks.
result Gradient descent dynamics exhibit quenching and activation phases in under-parametrized networks.