Paper transforms deep rectifier networks into shallow ones for analysis.
problem Understanding the complexity of deep neural networks.
method Transformation of deep rectifier networks into shallow ones.
result Shallow networks can represent deep networks with fewer functions.
Neural networks can approximate rectifiable measures with small error.
problem Approximating complex rectifiable measures using neural networks.
method Using ReLU neural networks to approximate (countably) m-rectifiable measures as push-forwards of the Lebesgue measure.
result The approximation error in terms of Wasserstein distance can be made arbitrarily small.
Paper provides convergence rates for rectifier convnets.
problem Understanding why rectifier networks perform well empirically.
method Introduces gated games to capture rectifier units' gating function.
result Gradient descent on rectifier convnets converges to a critical point.
Deep rectifier networks can approximate high resolution boundaries with fewer parameters.
problem Classifying high-dimensional data with high resolution boundaries.
method Theoretical justification of deep rectifier networks' superior performance using PWL classifier boundaries.
result Deep rectifier networks can approximate high resolution boundaries with fewer parameters.
In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evalu…
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
CRITS improves time series classification with interpretable local explanations.
problem Lack of detailed explanations in time series classification models.
method CRITS uses convolutional kernels, max-pooling, and rectified linear units to extract feature weights.
result CRITS provides intrinsically interpretable local explanations without requiring gradients or random perturbations.
Paper provides convergence guarantees for rectifier networks using neural Taylor approximations.
problem Smoothness and convexity issues in modern convolutional networks.
method Neural Taylor approximations and Taylor loss for optimization.
result Guarantees match lower bounds for convex nonsmooth functions and accurately capture optimization dynamics.
New neural network model with rectified edges for classification.
problem Classifying data with monotone properties.
method Introduces rectified wire networks with a novel learning algorithm.
result Demonstrates SDA algorithm achieving zero loss for correct classification.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
Rectified flows achieve optimal sample complexity for generating data.
problem Generating high-quality data samples efficiently.
method Rectified flows constrain transport trajectories to be linear, enabling efficient sampling.
result Achieve sample complexity of ildeO(ε−2), matching optimal rate for mean estimation. Study on the optimization landscape of half-rectified networks without simplifying assumptions.
problem Understanding the optimization landscape of deep neural networks, focusing on half-rectified networks.
method Theoretical analysis and empirical study of gradient descent on half-rectified networks.
result Proves that half-rectified single layer networks are asymptotically connected and provides bounds on the interplay between data distribution and model over-parametrization.
This paper controls the capacity of weight-normalized deep neural networks using rectified linear units.
problem Capacity control of weight-normalized deep neural networks.
method Establishes upper bounds on Rademacher complexities and analyzes approximation properties of Lp,q weight normalized networks. result For L1,∞ weight normalized networks, the approximation error is controlled by the L1 norm of the output layer, and generalization error depends on the square root of depth. Improved texture synthesis using wavelet-based statistics with rectifier non-linearity.
problem Improving texture synthesis quality using wavelet representations.
method Proposes a family of statistics based on non-linear wavelet representations with a generalized rectifier non-linearity.
result Significantly improves visual quality of texture synthesis compared to classical wavelet-based models.
Improved activation function NLReLU boosts neural network performance.
problem Performance issues with ReLU activation function.
method NLReLU uses parametric natural logarithmic transform to improve ReLU.
result NLReLU provides higher accuracy than ReLU in various neural networks.
Upper bound on knots in neural networks simplifies complexity analysis.
problem Understanding the complexity of neural networks with rectified linear units.
method Inductive analysis on the number of layers and neurons, proving an upper bound.
result A tight upper bound on the number of knots in neural networks is derived.
QuickNet is a faster, more accurate deep learning model.
problem Improving deep learning model efficiency and accuracy.
method Modified Darknet model with depthwise separable convolutions and parametric rectified linear units.
result QuickNet achieves 95.7% accuracy on CIFAR-10 dataset, faster than other models.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
New insights into neural network kernels for rectified MLPs.
problem Understanding the behavior of weight distributions in rectified MLPs.
method Deriving equivalent kernels for MLPs with ReLU or Leaky ReLU activations under various weight distributions.
result Kernels corresponding to layers with certain weight distributions are asymptotically universal and well approximated by Gaussian kernels.
Empirical bounds estimate the number of linear regions in deep ReLU networks.
problem Estimating the number of linear regions in deep neural networks.
method Empirical bounds based on activation patterns and probabilistic inference.
result Fast proxy for the number of linear regions of deep neural networks.
R3Net improves neural network robustness to input perturbations.
problem Improving neural network robustness to input perturbations.
method Randomized features, sign-splitter, and ReLU layers.
result Network output maintains distance between non-arbitrarily close inputs.
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
New method trains deep vanilla networks as fast as ResNets without shortcut connections.
problem Training very deep neural networks is challenging.
method Developed a new type of transformation compatible with Leaky ReLUs.
result Validation accuracies with deep vanilla networks are competitive with ResNets and significantly higher.
Rectifying submanifolds are characterized by their tangential position vector field component.
problem Characterizing rectifying submanifolds in Euclidean spaces.
method Introducing rectifying submanifolds and proving their properties.
result Rectifying submanifolds are identified by a specific tangential vector field property.
Scattering networks maximize separation on low-dimensional data.
problem Maximizing separation capacity on low-dimensional datasets.
method Characterize and bound separation capacity for feature extractors, then apply to scattering networks with specific criteria.
result Design criteria for scattering networks to maximize separation on low-dimensional data.
Study rectifying curves in 3D multiplicative Euclidean space.
problem Investigate rectifying curves in a non-Newtonian geometry setting.
method Apply multiplicative differential-geometric concepts to rectifying curves.
result Classify multiplicative rectifying curves using spherical curves.
Investigates Darboux rectifying curves on smooth surfaces.
problem Characterizing Darboux rectifying curves on smooth surfaces.
method Analyzes the position vector under isometry and finds conformal invariance conditions.
result Identifies sufficient conditions for conformal invariance of Darboux rectifying curves.
Study characterizes k-rectifiable sets in homogeneous groups.
problem Characterizing k-rectifiable sets in arbitrary homogeneous groups. method Proves characterizations using (k,G)-approximate tangent groups. result Existence of (k,G)-approximate tangent groups implies k-rectifiability. New curves generalize helix and rectifying curves.
problem Generalizing helix and rectifying curves.
method Introducing f-rectifying curves with f-position vector in rectifying plane.
result Classification and characterization of f-rectifying curves.
This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.
problem Exact data interpolation using sparse, infinitely wide neural networks.
method Atomic norm framework to derive convex hulls and equivalent convex formulations.
result Simple characterizations of convex hulls for different constraints on network weights and biases.
New capacity measure based on Fisher-Rao norm for neural networks.
problem Understanding the complexity and capacity of neural networks.
method Introducing Fisher-Rao norm and studying its invariance properties.
result The Fisher-Rao norm serves as an umbrella for existing norm-based complexity measures.
Reduced parameter space improves neural network training efficiency.
problem Training efficiency and optimization in neural networks.
method Training weights on a unit sphere and thresholds in a bounded interval.
result Equivalent performance with reduced parameter space.
Recalls and refines the concept of algebraically rectifiable curves.
problem Classical notion of algebraically rectifiable plane curves.
method Provides new criteria, relates to quadratic differentials, and generalizes to higher order differentials.
result Generalization and new criteria for algebraic rectifiability.
Harmonic maps to Euclidean buildings have rectifiable singular strata.
problem Understanding the structure of singular points for harmonic maps.
method Defining singular strata and proving rectifiability using the rectifiable Reifenberg program.
result Rectifiability of singular strata for harmonic maps into F-connected complexes. Uniform rectifiability proven for sets with Poincaré inequalities.
problem Uniform rectifiability of sets with Poincaré inequalities.
method Weak (1,d)-Poincaré inequality and surface measure. result Uniform rectifiability achieved for sets supporting such inequalities.
We prove (without using Federer's structure theorem) that a finite-mass flat chain over any coefficient group is rectifiable if and only if almost all of its 0-dimensional slices are rectifiable. This implies that every flat chain of finite mass and finite size is rectifiable. It also leads to a simple necessary and su…
New rectified flow method improves image generation and converges to optimal transport.
problem Improving computational and statistical guarantees of rectified flow for image generation.
method Introducing c-rectified flow, which projects velocity fields onto a gradient class while preserving marginals.
result Iterative c-rectified flow always converges to the optimal transport coupling under suitable assumptions.
The paper analyzes deep neural networks using rectified linear units.
problem Understanding the individual affine linear representations of deep neural networks.
method Signal processing perspective, atomic decompositions, Lipschitz regularity estimation.
result Conditions for stabilizing learning in deep neural networks without network depth constraints.
Model stores many more patterns than neurons, improving pattern recognition.
problem Storing and retrieving many more patterns than neurons in a network.
method Constructs a family of models interpolating between feature-matching and prototype modes, corresponding to neural networks with various activation functions.
result Higher rectified polynomials can be used in neural networks for improved pattern recognition.
Estimates parameters of a rectified Gaussian distribution using ReLU networks.
problem Estimating parameters of a rectified Gaussian distribution from i.i.d. samples.
method Simple algorithm using O(1/ε2) samples and O(d2/ε2) time. result Estimates distribution up to ε in total variation distance. Geometrically explains deep learning's manifold structure and probability distribution.
problem Lack of understanding how deep learning works and its theoretical foundation.
method Geometric view of deep learning, focusing on manifold structure and probability distribution.
result Deep learning learns the manifold and probability distribution in high-dimensional data.
New methods for solving hydrodynamic-type equations using quasi-rectifiable Lie algebras.
problem Solving systems of hydrodynamic-type equations.
method Introducing and analyzing quasi-rectifiable Lie algebras and vector fields.
result New methods for solving hydrodynamic-type equations.
Study approximates nonlinear functionals using deep ReLU networks.
problem Approximating nonlinear continuous functionals with neural networks.
method Constructs continuous piecewise linear interpolation under simple triangulation, analyzes rates of approximation.
result Established rates of approximation for functional deep ReLU networks.
Rectifies singular set of harmonic maps into complex.
problem Regularity of harmonic maps into complex manifolds.
method Proves (m−2)-rectifiability of singular set. result Singular set is (m−2)-rectifiable. The paper generalizes rectifying and normal curves in Lorentzian n-space.
problem Characterizing and classifying g−rectifying and g−normal curves in Lorentzian n-space. method Introducing a g−position vector field and defining g−rectifying and g−normal curves based on this field. result Comprehensive characterization and classification of g−rectifying and g−normal curves. Paper introduces a new GGM variant for efficient inference and unsupervised learning.
problem Limited modeling abilities of traditional Gaussian graphical models.
method Introduces a novel variant of Gaussian graphical models with truncated normal distributions and bipartite structure.
result Efficient inference and unsupervised learning capabilities demonstrated.
New algorithm reveals piecewise affine structure of neural networks.
problem Lack of strong guarantees on deep neural networks' behavior in safety-critical applications.
method Developed a novel algorithm to compute the piecewise affine form of neural networks.
result Computed piecewise affine representations of neural networks with rectified linear unit activations.