On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “ReLU networks”

Toric geometry of ReLU neural networksmath.AG

This study connects ReLU neural networks to toric geometry to analyze function realization.

problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.

method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.

result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.

ReLU neural networks toric geometry function realization algebraic geometry tropical geometry

2025-09-07abs ↗pdf ↗

Any Deep ReLU Network is Shallowcs.LG

Deep ReLU networks can be simplified to a three-layer model.

problem Understanding the behavior of deep neural networks.

method Constructive proof and algorithm to transform deep networks into shallow ones.

result Deep ReLU networks can be represented by a simpler three-layer structure.

deep learning ReLU networks shallow networks neural networks algorithm

2023-06-20abs ↗pdf ↗

Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networkscs.LG

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

problem Understanding the generalization capability of ReLU networks.

method Examined gradient flow dynamics and support vectors in single ReLU neuron training.

result Support vectors play a crucial role in the generalization of ReLU networks.

ReLU networks gradient dynamics support vectors generalization single neuron

2022-02-11abs ↗pdf ↗

Large Deviations of Gaussian Neural Networks with ReLU activationstat.ML

Large deviation principle for deep neural networks with ReLU activation.

problem Understanding the behavior of deep neural networks with ReLU activation.

method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.

result Simplified expressions and power-series expansions for the ReLU case.

large deviations neural networks ReLU activation Gaussian weights rate function

2024-05-27abs ↗pdf ↗

On the existence of optimal shallow feedforward networks with ReLU activationcs.LG

Proves existence of optimal shallow neural networks with ReLU activation.

problem Proving the existence of optimal shallow feedforward networks with ReLU activation.

method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.

result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.

ReLU activation shallow neural networks global minima loss landscape continuous target functions

2023-03-06abs ↗pdf ↗

Nonparametric regression with modified ReLU networksstat.ML

Modified ReLU networks improve regression estimation rates.

problem Regression estimation with smooth functions.

method Using modified ReLU neural networks with specific weight modifications.

result Empirical risk minimizers achieve minimax rate of prediction.

ReLU networks Regression Minimax rate Smooth functions

2022-07-17abs ↗pdf ↗

Approximation Properties of Deep ReLU CNNscs.LG

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

problem Establishing

L^2

approximation properties for deep ReLU CNNs.

method Analysis based on decomposition theorem for convolutional kernels, properties of ReLU activation, and connections with one-hidden-layer ReLU NNs.

result Universal approximation theorem for deep ReLU CNNs with classic structure.

Deep Learning ReLU CNNs Approximation Properties Convolutional Kernels Neural Networks

2021-09-01abs ↗pdf ↗

On the Expressive Power of Neural Networksmath.CA

Improved bounds on neural network expressivity.

problem Understanding neural network expressivity and approximation capabilities.

method Improved bounds on the maximal number of linear regions of ReLU-networks.

result New insights into the expressivity of neural networks.

Neural Networks Expressivity ReLU Approximation Linear Regions

2023-05-31abs ↗pdf ↗

Stochastic Bandits with ReLU Neural Networkscs.LG

Study shows efficient neural network approach for stochastic bandits.

problem Optimizing decisions in uncertain environments with neural network models.

method OFU-ReLU algorithm that balances exploration and exploitation, using a transformed feature space.

result Achieves

ilde{O}(\sqrt{T})

regret guarantee for stochastic bandits with ReLU neural networks.

stochastic bandits ReLU neural networks regret minimization OFU-ReLU algorithm piecewise linear structure

2024-05-12abs ↗pdf ↗

Poisson Hyperplane Processes with Rectified Linear Unitscs.LG

PHP connects to ReLU neural networks for scalable Bayesian inference.

problem Scalability and Bayesian inference in two-layer ReLU neural networks.

method PHP with Gaussian prior, decomposition propositions, annealed sequential Monte Carlo.

result PHP provides an alternative scalable representation for two-layer ReLU neural networks.

Poisson hyperplane processes ReLU neural networks Bayesian inference Monte Carlo methods scalability

2026-01-09abs ↗pdf ↗

The effect of Target Normalization and Momentum on Dying ReLUcs.LG

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

problem Understanding and mitigating the dying ReLU problem in neural networks.

method Empirical analysis and theoretical modeling of a discrete-time linear autonomous system.

result Target variance plays a crucial role in the dying ReLU phenomenon, and momentum exacerbates this issue.

dying ReLU neural networks momentum target normalization rectified linear units

2020-05-13abs ↗pdf ↗

Constraining the outputs of ReLU neural networksmath.AG

We study algebraic varieties of ReLU networks to understand their representable functions.

problem Understanding the functions that ReLU neural networks can represent.

method We introduce algebraic varieties associated with ReLU networks and derive polynomial equations to characterize representable functions.

result Conditions under which ReLU networks attain their expected dimension, providing insight into their structural properties.

ReLU networks piecewise linear piecewise multilinear algebraic varieties rank constraints

2025-08-05abs ↗pdf ↗

On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimemath.AP

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.

problem Training two-layer ReLU networks using gradient descent in the mean field regime.

method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.

result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.

gradient descent two-layer ReLU networks mean field regime minimum Bayes risk ReLU activation

2020-05-27abs ↗pdf ↗

Analytical bounds on the local Lipschitz constants of ReLU networkscs.LG

The paper calculates upper bounds on ReLU network Lipschitz constants.

problem Determining the maximum perturbation size for robustness of neural networks.

method Analyzing ReLU, affine-ReLU, and max pooling functions; combining results; tracking zero elements; using a computational approach.

result The method produces the largest known bounds on minimum adversarial perturbations for large networks.

ReLU networks Lipschitz constants Adversarial perturbations Neural networks Robustness

2021-04-29abs ↗pdf ↗

Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacs.LG

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.

method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.

result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.

gradient descent ReLU networks implicit bias nearly-orthogonal data stable rank

2023-10-29abs ↗pdf ↗

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spacesstat.ML

New bounds on ReLU networks for low-regular functions.

problem Bounding approximation error for ReLU networks on low-regular functions.

method Complexity analysis of Fourier features residual networks to ReLU networks.

result Approximation error bound proportional to target function norm and inversely proportional to network width and depth.

ReLU networks approximation error low-regular functions Fourier features neural networks

2024-05-10abs ↗pdf ↗

Topological Signatures of ReLU Neural Network Activation Patternscs.LG

The paper examines topological features of ReLU networks and their relation to decision boundaries and training loss.

problem Understanding the topological structure of ReLU neural network activation patterns.

method Polytope decomposition of feature space, Fiedler partition of dual graph, homology computation of cellular decomposition.

result The Fiedler partition of the dual graph correlates with decision boundaries in binary classification tasks, and similar patterns in training loss and polyhedral cell-count emerge in regression tasks.

ReLU networks topology neural networks polytope Fiedler partition

2025-10-14abs ↗pdf ↗

Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalizationstat.ML

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size an…

2018-08-14abs ↗pdf ↗

Implicit Regularization Towards Rank Minimization in ReLU Networkscs.LG

ReLU networks implicitly favor low-rank solutions, but not as strongly as linear networks.

problem Understanding implicit regularization in ReLU networks for rank minimization.

method Analysis of gradient flow on ReLU networks, empirical testing.

result Gradient flow on ReLU networks does not necessarily minimize ranks, unlike in linear networks.

ReLU networks rank minimization implicit regularization gradient flow deep learning

2022-01-30abs ↗pdf ↗

Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activationscs.LG

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.

problem Learning depth-2 neural networks with non-zero bias terms and general ReLU activations.

method Robust tensor decomposition of Hermite expansions.

result Polynomial time and sample efficient learning of depth-2 networks with ReLU activations.

Neural Networks ReLU Activations Polynomial Time Tensor Decomposition Hermite Expansion

2021-07-21abs ↗pdf ↗

Clustering-Based Interpretation of Deep ReLU Networkstat.ML

The paper reveals how clustering in ReLU networks enhances interpretability.

problem Improving interpretability of deep learning models.

method Clustering of ReLU neuron patterns to reveal underlying affine maps.

result The network's predictions can be explained by feature importance within clusters.

Deep Learning ReLU Networks Interpretability Clustering Feature Importance

2021-10-13abs ↗pdf ↗

How do noise tails impact on deep ReLU networks?math.ST

This paper examines how noise affects deep neural networks and improves their performance.

problem The impact of noise on the stability of deep ReLU neural networks for nonparametric regression.

method Investigates the optimal rate of convergence for deep ReLU neural networks under Huber loss, considering the p-th moment of noise and the smoothness of the function.

result The optimal rate of convergence cannot be achieved by ordinary least squares but can be by Huber loss with a properly chosen parameter.

deep neural networks ReLU networks nonparametric regression Huber loss noise impact

2022-03-20abs ↗pdf ↗

Approximation of Nonlinear Functionals Using Deep ReLU Networksstat.ML

Study approximates nonlinear functionals using deep ReLU networks.

problem Approximating nonlinear continuous functionals with neural networks.

method Constructs continuous piecewise linear interpolation under simple triangulation, analyzes rates of approximation.

result Established rates of approximation for functional deep ReLU networks.

Deep ReLU networks Functional approximation Nonlinear functionals Rectified linear unit Triangulation

2023-04-10abs ↗pdf ↗

Traversing the Local Polytopes of ReLU Neural Networks: A Unified Approach for Network Verificationcs.LG

Unified approach to verify NN properties using ReLU's unique polytope structure.

problem Lack of robustness and interpretability in ReLU NNs for risk-sensitive applications.

method Identifying and traversing the local polytopes of ReLU NNs, developing an algorithm to verify properties.

result Unified approach to examine network behavior in risk-sensitive settings.

ReLU neural networks polytopes robustness interpretability network verification

2021-11-17abs ↗pdf ↗

Approximation capabilities of neural networks on unbounded domainscs.LG

In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]^n. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU act…

2019-10-21abs ↗pdf ↗

One Simple Trick to Fix Your Bayesian Neural Networkstat.ML

Leaky ReLU activations improve the calibration of Bayesian neural networks.

problem Bayesian neural networks struggle with mean-field variational inference for ReLU activations.

method Investigated the effect of activation functions on the calibration of Bayesian neural networks.

result Leaky ReLU activations lead to more Gaussian-like weight posteriors and lower expected calibration error.

Bayesian neural networks Leaky ReLU activation function calibration variational inference

2022-07-26abs ↗pdf ↗

Optimal rates of approximation by shallow ReLU

^k

neural networks and applications to nonparametric regressionstat.ML

Optimal rates for shallow ReLU networks in nonparametric regression.

problem Approximating smooth and non-smooth functions with shallow ReLU networks.

method Analysis of shallow ReLU

^k

neural networks, using variation norms and deep learning theory.

result Optimal approximation rates for shallow ReLU networks in nonparametric regression.

ReLU networks nonparametric regression shallow neural networks convolutional neural networks Hölder functions

2023-04-04abs ↗pdf ↗

When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?stat.ML

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

problem Conditions for gradient descent to drive logistic loss to zero in deep networks with smoothed ReLU activations.

method Gradient descent applied to fixed-width deep networks with smoothed ReLU approximations (e.g., Swish, Huberized ReLU).

result Gradient descent can drive logistic loss to zero under specific conditions, providing bounds on convergence rate.

gradient descent logistic loss deep networks smoothed ReLU convergence

2021-02-09abs ↗pdf ↗

Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficientstat.ML

Hamiltonian Monte Carlo on ReLU networks is inefficient due to large local error.

problem Inefficiency of Hamiltonian Monte Carlo on ReLU neural networks.

method Analysis of Hamiltonian Monte Carlo with leapfrog integrator for Bayesian neural network inference.

result Leapfrog HMC for ReLU networks has a large local error rate of

Ω(ε)

, leading to inefficiency.

Hamiltonian Monte Carlo ReLU networks Bayesian inference Efficiency Local error

2024-10-29abs ↗pdf ↗

Quantile regression with deep ReLU Networks: Estimators and minimax ratesmath.ST

Quantile regression with ReLU networks achieves minimax rates for various function types.

problem Estimating quantiles from covariates with neural networks.

method Quantile regression with rectified linear unit (ReLU) neural networks.

result ReLU networks achieve minimax rates for broad collections of function types.

quantile regression ReLU networks minimax rates rectified linear units function approximation

2020-10-16abs ↗pdf ↗

Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Datacs.LG

Deep ReLU networks need Ω(N) parameters to interpolate at irregularly spaced points.

problem Interpolating at irregularly spaced data points with deep ReLU networks.

method Analyzing the number of parameters required for interpolation.

result Ω(N) parameters are necessary for interpolation when δ is exponentially small in N.

deep learning ReLU networks interpolation irregular data VC dimension

2023-02-02abs ↗pdf ↗

Learning Prescriptive ReLU Networkscs.LG

A new neural network model for optimal treatment assignment.

problem Learning optimal treatment assignment from observational data.

method Prescriptive ReLU network (P-ReLU) model that balances performance and interpretability.

result P-ReLU can be converted into an equivalent prescriptive tree with hyperplane splits for interpretability.

treatment assignment neural networks prescriptive models interpretability observational data

2023-06-01abs ↗pdf ↗

Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weightscs.LG

The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.

problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.

method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.

result Obtained an approximation formula for functions represented by 2-layer neural networks.

Fisher information neural tangent kernels ReLU networks random weights spectral decomposition

2025-07-24abs ↗pdf ↗

Complexity of One-Dimensional ReLU DNNscs.LG

Study on the complexity of 1D ReLU neural networks, proving growth in linear regions.

problem Understanding the complexity and expressivity of 1D ReLU neural networks.

method Analyzing the number of linear regions in randomly initialized, fully connected 1D ReLU networks in the infinite-width limit.

result The expected number of linear regions grows as a function of the number of neurons in each layer.

ReLU neural networks one-dimensional networks expressivity linear regions infinite-width limit

2025-12-08abs ↗pdf ↗

Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscs.LG

Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.

problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.

method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.

result GmP resolves instability, leading to better optimization, convergence, and generalization.

ReLU networks training dynamics stochastic optimization activation boundaries Geometric Parameterization

2023-05-25abs ↗pdf ↗

On Size-Independent Sample Complexity of ReLU Networkscs.LG

Improved sample complexity for ReLU networks with norm constraints.

problem Estimating sample complexity for ReLU networks under norm constraints.

method Refined Rademacher complexity analysis for function class.

result Often no explicit depth-dependence in sample complexity bound.

ReLU networks sample complexity Rademacher complexity norm constraints generalization

2023-06-03abs ↗pdf ↗

Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Boundsstat.ML

Adam achieves optimal convergence in deep ReLU networks via novel Kakeya bounds.

problem Training deep ReLU networks using Adam in non-smooth settings.

method Stratified Morse theory and Kakeya bounds to analyze region crossings and convergence.

result First global-optimal convergence for Adam in non-smooth, non-convex ReLU landscapes.

Adam ReLU networks Morse theory Kakeya sets Generalization bound

2025-05-21abs ↗pdf ↗

Singular Values for ReLU Layerscs.LG

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

2018-12-06abs ↗pdf ↗

Confidence Interval Construction and Conditional Variance Estimation with Dense ReLU Networksstat.ML

Paper improves confidence intervals and variance estimation for deep learning models.

problem Improving confidence intervals and variance estimation in deep learning models.

method Residual-based framework for conditional variance estimation; robust bootstrap procedure for confidence intervals.

result First non-asymptotic bounds for variance estimation using ReLU networks.

Confidence intervals Conditional variance Deep learning ReLU networks Bootstrap

2024-12-29abs ↗pdf ↗

Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performancecs.LG

The study examines if ReLU activation function is optimal for modularity in neural networks.

problem Finding the best activation function for modularity in neural networks.

method Comparing ReLU with other activation functions for modularity and performance.

result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.

activation functions neural networks modularity ReLU performance

2020-09-16abs ↗pdf ↗

On the Approximation Properties of Random ReLU Featuresstat.ML

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

2018-10-10abs ↗pdf ↗

Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Casescs.LG

Bayesian free energy remains bounded for deep ReLU networks in overparametrized cases.

problem Understanding the generalization performance of deep ReLU neural networks.

method Analyzes Bayesian free energy in overparametrized deep ReLU neural networks.

result Bayesian free energy is bounded even in overparametrized deep ReLU networks.

Bayesian free energy deep ReLU neural networks overparametrization generalization error singular learning theory

2023-03-28abs ↗pdf ↗

Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networksstat.ML

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

problem Generalization of gradient descent methods for deep neural networks

method Establishing minimax-optimal rates for GD and SGD with deep ReLU networks

result Gradient descent methods for deep ReLU networks achieve optimal generalization rates

generalization gradient descent deep neural networks ReLU networks minimax-optimal

2026-06-04abs ↗pdf ↗

On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcs.LG

Deep neural networks can approximate complex functions through repeated compositions of a fixed-size ReLU network.

problem Understanding the expressive power of deep neural networks through function compositions.

method Demonstrated the surprising expressive power of repeated compositions of a single fixed-size ReLU network.

result Repeated compositions of a single fixed-size ReLU network can approximate 1-Lipschitz continuous functions on

[0,1]^d

with an error

\mathcal{O}(r^{-1/d})

deep neural networks ReLU networks expressive power function compositions approximation theory

2023-01-29abs ↗pdf ↗

On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networksmath.CO

New insights into how neural networks classify data.

problem Understanding the topological structure of decision regions in ReLU networks.

method Defining generic and transversal ReLU networks, and using linear complexes to identify obstructions.

result Generic, transversal ReLU networks have at most one bounded connected component in their decision regions.

ReLU neural networks transversality topological expressiveness decision regions binary classification

2020-08-20abs ↗pdf ↗

Functional dimension of feedforward ReLU neural networksmath.MG

Functional dimension varies in ReLU networks, with implications for symmetry and connectivity.

problem Understanding the functional dimension of ReLU neural networks.

method Careful definition and analysis of functional dimension, study of quotient space and fibers.

result Functional dimension is inhomogeneous and can be non-constant, with implications for symmetry and connectivity.

ReLU neural networks functional dimension symmetry quotient space fiber

2022-09-08abs ↗pdf ↗

Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcs.LG

New insights into how linear classifiers and leaky ReLU networks can overfit without harming generalization.

problem Understanding conditions for benign overfitting in linear classifiers and leaky ReLU networks.

method Utilizing Karush--Kuhn--Tucker (KKT) conditions for margin maximization.

result Satisfaction of KKT conditions leads to benign overfitting in linear classifiers and leaky ReLU networks.

benign overfitting linear classifiers leaky ReLU networks Karush--Kuhn--Tucker conditions margin maximization

2023-03-02abs ↗pdf ↗

Reverse-Engineering Deep ReLU Networkscs.LG

It has been widely assumed that a neural network cannot be recovered from its outputs, as the network depends on its parameters in a highly nonlinear way. Here, we prove that in fact it is often possible to identify the architecture, weights, and biases of an unknown deep ReLU network by observing only its output. Ever…

2019-10-02abs ↗pdf ↗