Rectified flows achieve optimal sample complexity for generating data.
problem Generating high-quality data samples efficiently.
method Rectified flows constrain transport trajectories to be linear, enabling efficient sampling.
result Achieve sample complexity of ildeO(ε−2), matching optimal rate for mean estimation. Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
Upper bound on knots in neural networks simplifies complexity analysis.
problem Understanding the complexity of neural networks with rectified linear units.
method Inductive analysis on the number of layers and neurons, proving an upper bound.
result A tight upper bound on the number of knots in neural networks is derived.
This paper controls the capacity of weight-normalized deep neural networks using rectified linear units.
problem Capacity control of weight-normalized deep neural networks.
method Establishes upper bounds on Rademacher complexities and analyzes approximation properties of Lp,q weight normalized networks. result For L1,∞ weight normalized networks, the approximation error is controlled by the L1 norm of the output layer, and generalization error depends on the square root of depth. In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evalu…
Study on the complexity of deep neural networks and their linear regions.
problem Understanding the complexity and structure of deep neural networks.
method Leveraging the dimension of the space defining each linear region, the study presents bounds and methods for counting linear regions.
result Tighter upper and lower bounds for the maximum number of linear regions on rectifier networks, and a method for exact enumeration.
Geometrically explains deep learning's manifold structure and probability distribution.
problem Lack of understanding how deep learning works and its theoretical foundation.
method Geometric view of deep learning, focusing on manifold structure and probability distribution.
result Deep learning learns the manifold and probability distribution in high-dimensional data.
Rectifies singular set of harmonic maps into complex.
problem Regularity of harmonic maps into complex manifolds.
method Proves (m−2)-rectifiability of singular set. result Singular set is (m−2)-rectifiable. Paper transforms deep rectifier networks into shallow ones for analysis.
problem Understanding the complexity of deep neural networks.
method Transformation of deep rectifier networks into shallow ones.
result Shallow networks can represent deep networks with fewer functions.
CRITS improves time series classification with interpretable local explanations.
problem Lack of detailed explanations in time series classification models.
method CRITS uses convolutional kernels, max-pooling, and rectified linear units to extract feature weights.
result CRITS provides intrinsically interpretable local explanations without requiring gradients or random perturbations.
Improved texture synthesis using wavelet-based statistics with rectifier non-linearity.
problem Improving texture synthesis quality using wavelet representations.
method Proposes a family of statistics based on non-linear wavelet representations with a generalized rectifier non-linearity.
result Significantly improves visual quality of texture synthesis compared to classical wavelet-based models.
Gradient descent learns ReLUs in high dimensions with optimal sample complexity.
problem Learning Rectified Linear Units (ReLUs) in high-dimensional settings.
method Projected gradient descent initialized at zero, with inputs from Gaussian distribution and labels from planted weights.
result Projected gradient descent converges linearly to the planted model with optimal sample complexity.
Harmonic maps to Euclidean buildings have rectifiable singular strata.
problem Understanding the structure of singular points for harmonic maps.
method Defining singular strata and proving rectifiability using the rectifiable Reifenberg program.
result Rectifiability of singular strata for harmonic maps into F-connected complexes. New geometric definition for subsets of Euclidean space, proving second-order rectifiability.
problem Defining and proving second-order rectifiability for closed subsets of Euclidean space.
method Defining strata based on linear independence of touching directions and proving rectifiability using a new criterion.
result The m-th stratum of a closed subset of an n-dimensional Euclidean space is second-order rectifiable of dimension m. We consider the class of integer rectifiable currents without boundary satisfying a positivity condition. We establish that these currents can be written as a linear superposition of graphs of finitely many functions with bounded variation.
Develops a new method for solving sparse non-negative least squares problems.
problem Sparse non-negative least squares problem.
method Bayesian evidence maximization framework using Rectified Gaussian Scale Mixture (R-GSM) prior and Expectation-Maximization (EM) algorithm.
result Proposed R-SBL method outperforms existing solvers in signal and support recovery.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
Polynomial-time reachability for LTI systems with TLL NN controllers is achieved.
problem Bounding the reachable set of LTI systems controlled by TLL NN controllers.
method Polynomial-time computation of exact one-step reachable set and tight bounding box via two methods.
result Exact reachability computation in polynomial time for TLL NN controllers.
Empirical bounds estimate the number of linear regions in deep ReLU networks.
problem Estimating the number of linear regions in deep neural networks.
method Empirical bounds based on activation patterns and probabilistic inference.
result Fast proxy for the number of linear regions of deep neural networks.
A new method to measure neural network expressiveness using tighter upper bounds.
problem Measuring the expressiveness of deep neural networks (DNNs).
method Proposes a new tighter upper bound for the number of linear regions in rectifier networks, using matrix computation.
result The proposed upper bound is tighter than existing ones and explains the performance improvements of skip connections and residual structures.
Equivariant neural networks use symmetry to interpret complex data.
problem Interpreting and understanding the behavior of equivariant neural networks.
method Decompose layers into simple representations and analyze nonlinear activation functions.
result Equivariant neural networks can be interpreted using a filtration generalizing Fourier series.
L*ReLU improves deep learning for fine-grained image classification.
problem Fine-grained image classification requires specific AFs.
method Proposes L*ReLU, piece-wise linear AFs for deep learning.
result L*ReLU achieves superior results on FGVC datasets.
QuickNet is a faster, more accurate deep learning model.
problem Improving deep learning model efficiency and accuracy.
method Modified Darknet model with depthwise separable convolutions and parametric rectified linear units.
result QuickNet achieves 95.7% accuracy on CIFAR-10 dataset, faster than other models.
Paper provides convergence rates for rectifier convnets.
problem Understanding why rectifier networks perform well empirically.
method Introduces gated games to capture rectifier units' gating function.
result Gradient descent on rectifier convnets converges to a critical point.
AReLU uses attention-based rectification to improve neural network performance.
problem Improving neural network performance through better activation functions.
method Integrates attention mechanism with rectified linear unit (ReLU) to learn and scale feature maps.
result AReLU significantly boosts performance of most network architectures with minimal changes.
The paper analyzes deep neural networks using rectified linear units.
problem Understanding the individual affine linear representations of deep neural networks.
method Signal processing perspective, atomic decompositions, Lipschitz regularity estimation.
result Conditions for stabilizing learning in deep neural networks without network depth constraints.
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…
Convex relaxations improve CNNs with fixed weights.
problem Improving CNNs with fixed weights.
method Convex relaxations for CNNs with fixed weights using second order cone programs.
result The relaxation recovers the global minimum under a planted model assumption.
Language Rectified Flow improves diffusion language generation by simplifying complex steps.
problem Complexity in diffusion language models limits their implementation in NLP applications.
method Reformulates probabilistic flow models to learn neural ODE models for efficient domain transfer.
result Consistently outperforms baselines on fine-grained control tasks and text editing.
Apollo improves nonconvex stochastic optimization efficiency.
problem Nonconvex stochastic optimization challenges.
method Adaptive parameter-wise diagonal quasi-Newton method approximating Hessian.
result Significant improvements in convergence speed and generalization over SGD and Adam.
Improved activation function NLReLU boosts neural network performance.
problem Performance issues with ReLU activation function.
method NLReLU uses parametric natural logarithmic transform to improve ReLU.
result NLReLU provides higher accuracy than ReLU in various neural networks.
Paper proposes efficient image inversion and editing using rectified stochastic differential equations.
problem Inversion and editing of real images using generative models.
method Proposes RF inversion using dynamic optimal control and a linear quadratic regulator, extending to stochastic sampler for Flux.
result Allows state-of-the-art performance in zero-shot inversion and editing, outperforming prior works.
Deep rectifier networks can approximate high resolution boundaries with fewer parameters.
problem Classifying high-dimensional data with high resolution boundaries.
method Theoretical justification of deep rectifier networks' superior performance using PWL classifier boundaries.
result Deep rectifier networks can approximate high resolution boundaries with fewer parameters.
New algorithm reveals piecewise affine structure of neural networks.
problem Lack of strong guarantees on deep neural networks' behavior in safety-critical applications.
method Developed a novel algorithm to compute the piecewise affine form of neural networks.
result Computed piecewise affine representations of neural networks with rectified linear unit activations.
We discuss approximation of functions using deep neural nets. Given a function f on a d-dimensional manifold Γ⊂Rm, we construct a sparsely-connected depth-4 neural network and bound its error in approximating f. The size of the network depends on dimension and curvature of the manifold Γ, the…
Proposes a non-crossing deep neural network quantile regression method.
problem Quantile crossing in nonparametric quantile regression.
method Non-crossing constraints via rectified linear unit penalty function.
result Established non-asymptotic upper bounds for excess risk.
Study on the optimization landscape of half-rectified networks without simplifying assumptions.
problem Understanding the optimization landscape of deep neural networks, focusing on half-rectified networks.
method Theoretical analysis and empirical study of gradient descent on half-rectified networks.
result Proves that half-rectified single layer networks are asymptotically connected and provides bounds on the interplay between data distribution and model over-parametrization.
Improved optimal regularity for harmonic almost complex structures.
problem Establishing optimal regularity for harmonic almost complex structures.
method Quantitative stratification method and rectifiability of singular strata.
result Optimal regularity theory for energy minimizing harmonic almost complex structures.
R3Net improves neural network robustness to input perturbations.
problem Improving neural network robustness to input perturbations.
method Randomized features, sign-splitter, and ReLU layers.
result Network output maintains distance between non-arbitrarily close inputs.
Study of spatial curves in generalized Minkowski spaces.
problem Characterizing and invariants of spatial curves in non-Euclidean spaces.
method Derive Frenet-type results and invariants for spatial curves in generalized Minkowski spaces.
result Characterization of cylindrical helices and rectifying curves in generalized Minkowski spaces.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
Study approximates nonlinear functionals using deep ReLU networks.
problem Approximating nonlinear continuous functionals with neural networks.
method Constructs continuous piecewise linear interpolation under simple triangulation, analyzes rates of approximation.
result Established rates of approximation for functional deep ReLU networks.
Rectifying submanifolds are characterized by their tangential position vector field component.
problem Characterizing rectifying submanifolds in Euclidean spaces.
method Introducing rectifying submanifolds and proving their properties.
result Rectifying submanifolds are identified by a specific tangential vector field property.
The study provides bounds for geodesic diameter in Euclidean space.
problem Finding bounds for geodesic diameter in Euclidean space.
method Develops a geometric approach using locally rectifiable chains and complete normed commutative group bundles.
result Provides a new method for calculating geodesic diameter bounds.
Reduced parameter space improves neural network training efficiency.
problem Training efficiency and optimization in neural networks.
method Training weights on a unit sphere and thresholds in a bounded interval.
result Equivalent performance with reduced parameter space.
Paper introduces a new GGM variant for efficient inference and unsupervised learning.
problem Limited modeling abilities of traditional Gaussian graphical models.
method Introduces a novel variant of Gaussian graphical models with truncated normal distributions and bipartite structure.
result Efficient inference and unsupervised learning capabilities demonstrated.
Study rectifying curves in 3D multiplicative Euclidean space.
problem Investigate rectifying curves in a non-Newtonian geometry setting.
method Apply multiplicative differential-geometric concepts to rectifying curves.
result Classify multiplicative rectifying curves using spherical curves.