On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “ReLu neural networks”

Toric geometry of ReLU neural networksmath.AG

This study connects ReLU neural networks to toric geometry to analyze function realization.

problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.

method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.

result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.

ReLU neural networks toric geometry function realization algebraic geometry tropical geometry

2025-09-07abs ↗pdf ↗

Large Deviations of Gaussian Neural Networks with ReLU activationstat.ML

Large deviation principle for deep neural networks with ReLU activation.

problem Understanding the behavior of deep neural networks with ReLU activation.

method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.

result Simplified expressions and power-series expansions for the ReLU case.

large deviations neural networks ReLU activation Gaussian weights rate function

2024-05-27abs ↗pdf ↗

Poisson Hyperplane Processes with Rectified Linear Unitscs.LG

PHP connects to ReLU neural networks for scalable Bayesian inference.

problem Scalability and Bayesian inference in two-layer ReLU neural networks.

method PHP with Gaussian prior, decomposition propositions, annealed sequential Monte Carlo.

result PHP provides an alternative scalable representation for two-layer ReLU neural networks.

Poisson hyperplane processes ReLU neural networks Bayesian inference Monte Carlo methods scalability

2026-01-09abs ↗pdf ↗

Optimal rates of approximation by shallow ReLU

^k

neural networks and applications to nonparametric regressionstat.ML

Optimal rates for shallow ReLU networks in nonparametric regression.

problem Approximating smooth and non-smooth functions with shallow ReLU networks.

method Analysis of shallow ReLU

^k

neural networks, using variation norms and deep learning theory.

result Optimal approximation rates for shallow ReLU networks in nonparametric regression.

ReLU networks nonparametric regression shallow neural networks convolutional neural networks Hölder functions

2023-04-04abs ↗pdf ↗

Approximation Properties of Deep ReLU CNNscs.LG

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

problem Establishing

L^2

approximation properties for deep ReLU CNNs.

method Analysis based on decomposition theorem for convolutional kernels, properties of ReLU activation, and connections with one-hidden-layer ReLU NNs.

result Universal approximation theorem for deep ReLU CNNs with classic structure.

Deep Learning ReLU CNNs Approximation Properties Convolutional Kernels Neural Networks

2021-09-01abs ↗pdf ↗

Stochastic Bandits with ReLU Neural Networkscs.LG

Study shows efficient neural network approach for stochastic bandits.

problem Optimizing decisions in uncertain environments with neural network models.

method OFU-ReLU algorithm that balances exploration and exploitation, using a transformed feature space.

result Achieves

ilde{O}(\sqrt{T})

regret guarantee for stochastic bandits with ReLU neural networks.

stochastic bandits ReLU neural networks regret minimization OFU-ReLU algorithm piecewise linear structure

2024-05-12abs ↗pdf ↗

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacs.LG

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.

method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.

result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.

gradient descent ReLU networks implicit bias nearly-orthogonal data stable rank

2023-10-29abs ↗pdf ↗

On the existence of optimal shallow feedforward networks with ReLU activationcs.LG

Proves existence of optimal shallow neural networks with ReLU activation.

problem Proving the existence of optimal shallow feedforward networks with ReLU activation.

method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.

result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.

ReLU activation shallow neural networks global minima loss landscape continuous target functions

2023-03-06abs ↗pdf ↗

One Simple Trick to Fix Your Bayesian Neural Networkstat.ML

Leaky ReLU activations improve the calibration of Bayesian neural networks.

problem Bayesian neural networks struggle with mean-field variational inference for ReLU activations.

method Investigated the effect of activation functions on the calibration of Bayesian neural networks.

result Leaky ReLU activations lead to more Gaussian-like weight posteriors and lower expected calibration error.

Bayesian neural networks Leaky ReLU activation function calibration variational inference

2022-07-26abs ↗pdf ↗

Topological Signatures of ReLU Neural Network Activation Patternscs.LG

The paper examines topological features of ReLU networks and their relation to decision boundaries and training loss.

problem Understanding the topological structure of ReLU neural network activation patterns.

method Polytope decomposition of feature space, Fiedler partition of dual graph, homology computation of cellular decomposition.

result The Fiedler partition of the dual graph correlates with decision boundaries in binary classification tasks, and similar patterns in training loss and polyhedral cell-count emerge in regression tasks.

ReLU networks topology neural networks polytope Fiedler partition

2025-10-14abs ↗pdf ↗

Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weightscs.LG

The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.

problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.

method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.

result Obtained an approximation formula for functions represented by 2-layer neural networks.

Fisher information neural tangent kernels ReLU networks random weights spectral decomposition

2025-07-24abs ↗pdf ↗

How do noise tails impact on deep ReLU networks?math.ST

This paper examines how noise affects deep neural networks and improves their performance.

problem The impact of noise on the stability of deep ReLU neural networks for nonparametric regression.

method Investigates the optimal rate of convergence for deep ReLU neural networks under Huber loss, considering the p-th moment of noise and the smoothness of the function.

result The optimal rate of convergence cannot be achieved by ordinary least squares but can be by Huber loss with a properly chosen parameter.

deep neural networks ReLU networks nonparametric regression Huber loss noise impact

2022-03-20abs ↗pdf ↗

On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural Networkscs.LG

The paper proves neural networks with ReLU and softmax can approximate any function.

problem Approximating functions and class labels in neural networks.

method Extended universal approximator theory to neural networks with ReLU and softmax.

result Neural networks with ReLU and softmax can approximate any function and class labels.

Neural Networks ReLU Softmax Universal Approximator Pattern Classification

2020-02-10abs ↗pdf ↗

On the Expressive Power of Neural Networksmath.CA

Improved bounds on neural network expressivity.

problem Understanding neural network expressivity and approximation capabilities.

method Improved bounds on the maximal number of linear regions of ReLU-networks.

result New insights into the expressivity of neural networks.

Neural Networks Expressivity ReLU Approximation Linear Regions

2023-05-31abs ↗pdf ↗

Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performancecs.LG

The study examines if ReLU activation function is optimal for modularity in neural networks.

problem Finding the best activation function for modularity in neural networks.

method Comparing ReLU with other activation functions for modularity and performance.

result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.

activation functions neural networks modularity ReLU performance

2020-09-16abs ↗pdf ↗

Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscs.LG

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.

problem Learning depth-2 neural networks with non-zero bias terms and general ReLU activations.

method Robust tensor decomposition of Hermite expansions.

result Polynomial time and sample efficient learning of depth-2 networks with ReLU activations.

Neural Networks ReLU Activations Polynomial Time Tensor Decomposition Hermite Expansion

2021-07-21abs ↗pdf ↗

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spacesstat.ML

New bounds on ReLU networks for low-regular functions.

problem Bounding approximation error for ReLU networks on low-regular functions.

method Complexity analysis of Fourier features residual networks to ReLU networks.

result Approximation error bound proportional to target function norm and inversely proportional to network width and depth.

ReLU networks approximation error low-regular functions Fourier features neural networks

2024-05-10abs ↗pdf ↗

Spurious Local Minima of Deep ReLU Neural Networks in the Neural Tangent Kernel Regimestat.ML

In this paper, we theoretically prove that the deep ReLU neural networks do not lie in spurious local minima in the loss landscape under the Neural Tangent Kernel (NTK) regime, that is, in the gradient descent training dynamics of the deep ReLU neural networks whose parameters are initialized by a normal distribution i…

2018-06-13abs ↗pdf ↗

Approximation capabilities of neural networks on unbounded domainscs.LG

In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]^n. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU act…

2019-10-21abs ↗pdf ↗

Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscs.LG

Two-layer ReLU networks can overfit without harm, study finds.

problem Understanding when and how two-layer ReLU networks can overfit without harming generalization.

method Established algorithm-dependent risk bounds for two-layer ReLU convolutional neural networks with label-flipping noise.

result Gradient descent-trained ReLU networks can achieve near-zero training loss and Bayes optimal test risk.

ReLU networks benign overfitting two-layer neural networks gradient descent label-flipping noise

2023-03-07abs ↗pdf ↗

Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networksstat.ML

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

problem Generalization of gradient descent methods for deep neural networks

method Establishing minimax-optimal rates for GD and SGD with deep ReLU networks

result Gradient descent methods for deep ReLU networks achieve optimal generalization rates

generalization gradient descent deep neural networks ReLU networks minimax-optimal

2026-06-04abs ↗pdf ↗

Constraining the outputs of ReLU neural networksmath.AG

We study algebraic varieties of ReLU networks to understand their representable functions.

problem Understanding the functions that ReLU neural networks can represent.

method We introduce algebraic varieties associated with ReLU networks and derive polynomial equations to characterize representable functions.

result Conditions under which ReLU networks attain their expected dimension, providing insight into their structural properties.

ReLU networks piecewise linear piecewise multilinear algebraic varieties rank constraints

2025-08-05abs ↗pdf ↗

Explicit neural network classifiers for non-separable datacs.LG

Neural networks can separate non-separable data using feature maps.

problem Non-separable data in neural networks.

method Characterization of feedforward neural networks and use of feature maps.

result ReLU neural networks can separate concentric data.

neural networks non-separable data feature maps ReLU

2025-04-25abs ↗pdf ↗

How many samples are needed to train a deep neural network?math.ST

New study shows neural networks need many samples for training.

problem How much data is needed to train a ReLU feed-forward neural network?

method Theoretical and empirical analysis of ReLU feed-forward neural networks.

result Generalization error scales at

1/\sqrt{n}

in sample size

n

neural networks generalization error ReLU networks sample size

2024-05-26abs ↗pdf ↗

Generalization and Estimation Error Bounds for Model-based Neural Networkscs.LG

Model-based neural networks generalize better than ReLU networks for sparse recovery.

problem Understanding and quantifying the superior generalization of model-based neural networks.

method Using complexity measures like global and local Rademacher complexities, the paper provides theoretical bounds on generalization and estimation errors.

result Model-based neural networks exhibit higher generalization capabilities for sparse recovery problems compared to ReLU networks.

model-based neural networks generalization sparse recovery Rademacher complexity ReLU networks

2023-04-19abs ↗pdf ↗

Functional dimension of feedforward ReLU neural networksmath.MG

Functional dimension varies in ReLU networks, with implications for symmetry and connectivity.

problem Understanding the functional dimension of ReLU neural networks.

method Careful definition and analysis of functional dimension, study of quotient space and fibers.

result Functional dimension is inhomogeneous and can be non-constant, with implications for symmetry and connectivity.

ReLU neural networks functional dimension symmetry quotient space fiber

2022-09-08abs ↗pdf ↗

On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networksmath.CO

New insights into how neural networks classify data.

problem Understanding the topological structure of decision regions in ReLU networks.

method Defining generic and transversal ReLU networks, and using linear complexes to identify obstructions.

result Generic, transversal ReLU networks have at most one bounded connected component in their decision regions.

ReLU neural networks transversality topological expressiveness decision regions binary classification

2020-08-20abs ↗pdf ↗

The effect of Target Normalization and Momentum on Dying ReLUcs.LG

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

problem Understanding and mitigating the dying ReLU problem in neural networks.

method Empirical analysis and theoretical modeling of a discrete-time linear autonomous system.

result Target variance plays a crucial role in the dying ReLU phenomenon, and momentum exacerbates this issue.

dying ReLU neural networks momentum target normalization rectified linear units

2020-05-13abs ↗pdf ↗

Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Casescs.LG

Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.

problem Understanding the generalization performance of deep ReLU neural networks.

method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.

result Bayesian free energy is bounded even in overparametrized deep ReLU networks.

Bayesian free energy deep ReLU neural networks overparametrization generalization error singular learning theory

2023-03-28abs ↗pdf ↗

Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networkscs.LG

Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…

2019-08-10abs ↗pdf ↗

Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficientstat.ML

Hamiltonian Monte Carlo on ReLU networks is inefficient due to large local error.

problem Inefficiency of Hamiltonian Monte Carlo on ReLU neural networks.

method Analysis of Hamiltonian Monte Carlo with leapfrog integrator for Bayesian neural network inference.

result Leapfrog HMC for ReLU networks has a large local error rate of

Ω(ε)

, leading to inefficiency.

Hamiltonian Monte Carlo ReLU networks Bayesian inference Efficiency Local error

2024-10-29abs ↗pdf ↗

Path-conditioned training: a principled way to rescale ReLU neural networksstat.ML

A new method to rescale ReLU neural networks based on path-lifting.

problem Lack of principled ways to leverage rescaling symmetries in ReLU neural networks.

method Introduces a geometrically motivated criterion to rescale neural network parameters, aligning a kernel in the path-lifting space with a chosen reference.

result Proposed method can speed up training and aligns a kernel in the path-lifting space with a chosen reference.

ReLU neural networks rescaling symmetries path-lifting training dynamics kernel alignment

2026-02-23abs ↗pdf ↗

Local Identifiability of Deep ReLU Neural Networks: the Theorymath.ST

New theory for local parameterization of deep ReLU networks.

problem Determining local parameters of deep ReLU neural networks.

method Introducing local lifting operators and charts of a manifold, deriving necessary and sufficient conditions for local identifiability.

result Sharp and testable conditions for local identifiability of deep ReLU networks.

Deep Learning Neural Networks Local Identifiability ReLU Networks Geometric Conditions

2022-06-15abs ↗pdf ↗

Nonparametric regression using over-parameterized shallow ReLU neural networksstat.ML

Deep neural networks can learn smooth functions without parameters.

problem Learning smooth functions from shallow ReLU neural networks.

method Using over-parameterized shallow ReLU neural networks with norm constraints.

result Least squares estimators based on shallow neural networks are minimax optimal.

deep learning ReLU networks nonparametric regression minimax optimality shallow neural networks

2023-06-14abs ↗pdf ↗

Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscs.LG

GD-trained shallow ReLU nets learn Lipschitz functions with noise.

problem Learning Lipschitz functions with additive noise in overparameterized neural networks.

method Gradient Descent (GD) with early stopping, focusing on the Neural Tangent Kernel (NTK).

result Early-stopped GD achieves minimax optimal rates for learning Lipschitz functions.

Gradient Descent ReLU neural networks Lipschitz functions Noise Early stopping

2022-12-28abs ↗pdf ↗

Complexity of One-Dimensional ReLU DNNscs.LG

Study on the complexity of 1D ReLU neural networks, proving growth in linear regions.

problem Understanding the complexity and expressivity of 1D ReLU neural networks.

method Analyzing the number of linear regions in randomly initialized, fully connected 1D ReLU networks in the infinite-width limit.

result The expected number of linear regions grows as a function of the number of neurons in each layer.

ReLU neural networks one-dimensional networks expressivity linear regions infinite-width limit

2025-12-08abs ↗pdf ↗

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscs.LG

Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.

problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.

method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.

result GmP resolves instability, leading to better optimization, convergence, and generalization.

ReLU networks training dynamics stochastic optimization activation boundaries Geometric Parameterization

2023-05-25abs ↗pdf ↗

Any Deep ReLU Network is Shallowcs.LG

Deep ReLU networks can be simplified to a three-layer model.

problem Understanding the behavior of deep neural networks.

method Constructive proof and algorithm to transform deep networks into shallow ones.

result Deep ReLU networks can be represented by a simpler three-layer structure.

deep learning ReLU networks shallow networks neural networks algorithm

2023-06-20abs ↗pdf ↗

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networksstat.ML

Neural networks minimize error with shallow ReLU models for function estimation.

problem Estimating unknown functions from noisy data.

method Minimizing squared errors plus weight decay regularization.

result Neural network estimators are minimax optimal up to logarithmic factors.

Neural Networks ReLU Estimation Minimax Function Space

2021-09-18abs ↗pdf ↗

Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Datacs.LG

Deep ReLU networks need Ω(N) parameters to interpolate at irregularly spaced points.

problem Interpolating at irregularly spaced data points with deep ReLU networks.

method Analyzing the number of parameters required for interpolation.

result Ω(N) parameters are necessary for interpolation when δ is exponentially small in N.

deep learning ReLU networks interpolation irregular data VC dimension

2023-02-02abs ↗pdf ↗

Learning Prescriptive ReLU Networkscs.LG

A new neural network model for optimal treatment assignment.

problem Learning optimal treatment assignment from observational data.

method Prescriptive ReLU network (P-ReLU) model that balances performance and interpretability.

result P-ReLU can be converted into an equivalent prescriptive tree with hyperplane splits for interpretability.

treatment assignment neural networks prescriptive models interpretability observational data

2023-06-01abs ↗pdf ↗

Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verificationcs.LG

Unified approach to verify NN properties using ReLU's unique polytope structure.

problem Lack of robustness and interpretability in ReLU NNs for risk-sensitive applications.

method Identifying and traversing the local polytopes of ReLU NNs, developing an algorithm to verify properties.

result Unified approach to examine network behavior in risk-sensitive settings.

ReLU neural networks polytopes robustness interpretability network verification

2021-11-17abs ↗pdf ↗

Rates of Approximation by ReLU Shallow Neural Networkscs.LG

Paper studies shallow ReLU networks' approximation rates for Hölder functions.

problem Understanding shallow ReLU networks' efficiency in approximating Hölder functions.

method Analyzes rates of uniform approximation by ReLU shallow neural networks with

m

hidden neurons.

result Shows ReLU shallow neural networks can uniformly approximate Hölder functions with rates close to optimal.

ReLU networks Hölder spaces shallow neural networks approximation rates uniform approximation

2023-07-24abs ↗pdf ↗

A global convergence theory for deep ReLU implicit networks via over-parameterizationcs.LG

This paper proves SGD converges to global minimum for over-parameterized ReLU networks.

problem Theoretical understanding of implicit neural networks is limited.

method Gradient flow analysis of ReLU activated implicit neural networks.

result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.

deep learning implicit networks ReLU gradient descent convergence

2021-10-11abs ↗pdf ↗

Shapley Interpretation and Activation in Neural Networksstat.ML

We propose a novel Shapley value approach to help address neural networks' interpretability and "vanishing gradient" problems. Our method is based on an accurate analytical approximation to the Shapley value of a neuron with ReLU activation. This analytical approximation admits a linear propagation of relevance across …

2019-09-13abs ↗pdf ↗

Expressivity of Shallow and Deep Neural Networks for Polynomial Approximationcs.LG

Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.

problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.

method Established lower bounds on shallow network complexity for polynomial approximation.

result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.

ReLU networks polynomial approximation curse of dimensionality Lipschitz functions

2023-03-06abs ↗pdf ↗

Optimizing Objective Functions from Trained ReLU Neural Networks via Samplingmath.OC

New algorithms optimize neural networks with ReLU activations using sampling.

problem Optimizing trained neural networks with ReLU activations.

method Iterative algorithm and enhanced algorithm using sampling and neighborhood search.

result The methods reduce the initial MIP problem into smaller LP or MIP problems.

neural networks ReLU activations optimization sampling piecewise linear

2022-05-27abs ↗pdf ↗

Limitations of neural network training due to numerical instability of backpropagationcs.LG

Deep neural networks struggle with numerical instability during training.

problem Numerical instability in gradient descent training of deep neural networks.

method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.

result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.

numerical instability gradient descent ReLU networks deep learning training stability

2022-10-03abs ↗pdf ↗