Softplus normalization improves uncertainty estimation in Bayesian CNNs.
problem Improving uncertainty estimation in Bayesian CNNs.
method Softplus normalization applied to the final layer of Bayesian CNNs with variational inference.
result The method achieves equivalent performance to frequentist inference while naturally incorporating uncertainty and regularization.
A new method lifts training of input-convex neural networks to avoid dead weights and plateaued loss.
problem Training input-convex neural networks with non-negative weights.
method Introduces a hypernetwork that emits non-negative weights from a summary of the input batch, adding stochasticity to soften the loss landscape.
result The lift method achieves lower test loss than projected gradient descent and direct softplus reparametrization.
To construct flexible nonlinear predictive distributions, the paper introduces a family of softplus function based regression models that convolve, stack, or combine both operations by convolving countably infinite stacked gamma distributions, whose scales depend on the covariates. Generalizing logistic regression that…
Neural networks can approximate any L^p functions on R^n.
problem Approximating functions on unbounded domains with neural networks.
method Monotone sigmoid, ReLU, ELU, Softplus, LeakyReLU activation functions.
result Shallow neural networks can arbitrarily well approximate L^p functions on R^n.
New method improves GANs by estimating density ratios in feature space with SP loss.
problem Filtering out unrealistic images from GANs trained with suboptimal discriminators.
method Develops DRE-F-SP method based on Softplus loss for density ratio estimation in feature space, and proposes three subsampling methods.
result Empirically shows substantial improvement over existing methods on synthetic and CIFAR-10 datasets.
To model categorical response variables given their covariates, we propose a permuted and augmented stick-breaking (paSB) construction that one-to-one maps the observed categories to randomly permuted latent sticks. This new construction transforms multinomial regression into regression analysis of stick-specific binar…
Improved variational inference for logistic regression and classification.
problem Intractability of Evidence Lower Bound in variational logistic regression.
method Introducing a new bound for the expectation of softplus function, applied to variational logistic regression and Gaussian process classification.
result The new bound results in a tighter posterior and faster computation compared to Monte-Carlo methods.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
New method calibrates adaptive learning rate to improve convergence of Adam.
problem Inconsistent adaptive learning rate across dimensions leads to convergence issues.
method Proposes a new AGM by calibrating the adaptive learning rate with a softplus function.
result Calibrated methods outperform existing AGMs and generalize better.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
New algorithm reduces sample complexity for learning CNNs.
problem Learning one-hidden-layer CNNs with various activation functions.
method Approximate gradient descent algorithm for training CNNs.
result Sample complexity matches information-theoretic lower bound for linear activation functions.
A neural network derived from first principles using MaxEnt.
problem Developing a neural network from first principles.
method Derived a neural network using the principle of Maximum Entropy, with linear dimension-reducing transformations and conditional mean estimators.
result Unified theoretical justification for activation functions like sigmoid, softplus, and relu.
AGGLIO optimizes non-convex functions with local convexity guarantees.
problem Optimizing non-convex functions with local convexity.
method Stage-wise, graduated optimization technique for locally convex functions.
result Global convergence to the global optimum for non-convex and locally convex objectives.
Global convergence of SGD proven for two-layer neural nets with regularization.
problem Proving global convergence of SGD for two-layer neural nets.
method Regularized empirical risk, SGD iterates, Villani functions.
result Global convergence of SGD for a special class of initializations.
LRA trains deep networks robustly with less sensitivity to initial weights.
problem Training deep networks is challenging due to issues like exploding and vanishing gradients.
method Local Representation Alignment (LRA) is a training procedure less sensitive to initial weights.
result LRA can train networks robustly, even with null initial weights, and outperforms other methods.
Gradient flows of neural networks converge to optimal values or diverge, with thresholds and asymptotic behaviors.
problem Understanding the convergence and divergence of gradient flows in neural networks.
method Analysis of gradient flows on loss landscapes of neural networks using o-minimal structures.
result Gradient flows either converge to optimal values or diverge to infinity, with thresholds and asymptotic behaviors.
Mish is a new activation function that improves neural network performance.
problem Improving the performance and training dynamics of neural networks.
method Mish is a self-regularized non-monotonic activation function defined as f(x)=xanh(softplus(x)). It outperforms other functions on benchmarks like ImageNet-1k and MS-COCO. result Mish outperforms Leaky ReLU and ReLU on benchmarks like MS-COCO and ImageNet-1k, respectively, with comparable network parameters.
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
Lower bound proves ridgeless regression performs poorly near interpolation threshold.
problem Proving performance of ridgeless regression near interpolation threshold.
method Distribution-independent lower bound for mean squared error in noisy ridgeless linear regression.
result Lower bound implies ridgeless regression performs poorly near interpolation threshold.
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
Minimum width for ReLU networks on compact domain is exactly max{d_x, d_y, 2}
problem Characterizing the minimum width for ReLU networks to approximate functions on compact domains
method Analyzing the minimum width for Lp approximation of Lp functions from [0,1]d to Rdy using ReLU-like activation functions result The minimum width for Lp approximation on a compact domain is exactly max{d_x, d_y, 2} for ReLU-like activation functions IGNIS uses neural networks to estimate copula parameters robustly.
problem Pathological properties of Archimedean copulas make traditional estimators brittle.
method Unified neural estimation framework with multi-input architecture and softplus output layer.
result Accurate and stable estimates for real-world datasets.
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
This work refines imitation learning to approximate optimal state-feedback policies for a quadcopter model.
problem Designing optimal control policies for complex systems like quadcopters.
method Supervised imitation learning using deep neural networks trained on optimal trajectories.
result Deep neural networks can approximate optimal state-feedback policies with high accuracy, even with two layers.
A new method normalizes activations to match batch normalization without batch dependence.
problem Performance degradation with batch-independent normalization techniques.
method Proxy-Normalizing Activations
result Proxy-Normalization technique emulates batch normalization's behavior and performance.
Online Normalization normalizes neural network activations without batching for better accuracy.
problem Theoretical limitations of Batch Normalization and its inapplicability to certain network types.
method Introduces an unbiased gradient computation technique for normalized activations without using batches.
result Equivalent accuracy to Batch Normalization without batch usage.
This work introduces an efficient method to sample high-quality images from conditional GANs.
problem Efficient subsampling of images from conditional GANs (cGANs) is challenging.
method Developed a novel conditional density ratio estimation method (cDRE-F-cSP) and rejection sampling scheme (cDR-RS).
result cDR-RS outperforms state-of-the-art methods in both effectiveness and efficiency.
This paper analyzes how normalization layers improve neural network training.
problem Improving generalization performance and training speed of neural networks.
method Global convergence analysis of two-layer neural networks with ReLU activations and Weight Normalization.
result Introduction of normalization layers changes the optimization landscape, enabling faster convergence.
ILM-Norm normalizes instances individually for better performance.
problem Learning to normalize parameters for improved model performance.
method ILM-Norm learns normalization parameters via feature feed-forward and gradient back-propagation.
result ILM-Norm consistently improves model performance across different architectures and tasks.
Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural n…
The paper analyzes various normalization methods in deep learning.
problem Lack of mathematical tools to analyze normalization methods.
method Proposed a lemma to define tools, analyzed BN, LN, WN, GN.
result Normalization methods can be unified on a sphere, improving training stability and weight norm.
Distance, normals, and double normals for real plane curves with singularities
problem Relation between normals and double normals and critical points of the squared distance function for real algebraic curves with singularities
method Investigate the topological discriminant of the distance function
result The topological discriminant consists of the evolute and distinguished normal lines at algebraic singular points
Normalizes pseudo-Einstein contact forms for easier analysis.
problem Understanding pseudo-Einstein contact forms.
method Constructing intrinsic CR normal coordinates using parabolic normal coordinates.
result Normal form for pseudo-Einstein contact forms.
This paper reviews normalization techniques for DNNs.
problem Improving training speed and generalization of DNNs.
method Taxonomy of normalization methods, decomposition of activation methods.
result Insight for designing new normalization techniques.
Optimized normalization layers improve domain generalization.
problem Improving model generalization across different domains.
method Learning separate normalization parameters per domain using multiple normalization methods (batch and instance).
result State-of-the-art accuracy on domain generalization benchmarks.
We consider the Laplace normal vector field of relatively normalized ruled surfaces with non-vanishing Gaussian curvature in the three-dimensional Euclidean space R3. We determine all ruled surfaces and all relative normalizations for which the Laplace normal image degenerates into a point or into a curve…
Monotonicity of normalized implied-volatility coordinates under no-arbitrage
problem Monotonicity of normalized implied-volatility coordinates under no-arbitrage
method Elementary discrete no-arbitrage proof
result Monotonicity principle extended to Bachelier implied volatility
The paper studies polar normalizations of skew ruled surfaces in 3D space.
problem Understanding the properties and invariants of polar normalized skew ruled surfaces.
method Determination of invariants and analysis of Tchebychev and support vector fields.
result Special polar normalizations lead to degenerate curves.
Proposes adversarial normalization for multi-domain image segmentation.
problem Current image normalization is per-dataset, limiting multi-domain segmentation.
method Adversarial training to learn common normalizing functions across multiple datasets.
result Optimal normalizer improves segmentation accuracy and realism.
Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute…
Proposes a flexible normalization method to handle multi-modal data.
problem Reduced effectiveness of batch normalization in multi-modal distributions.
method Extends normalization to multiple means and variances, detecting data modes on-the-fly.
result Outperforms batch normalization and other methods in various experiments.
The paper defines normal forms for rational 3-tangles and shows a sequence of moves to transform one form to another.
problem Understanding and manipulating rational 3-tangles.
method Definition of normal forms and sequence of normal jump moves.
result There is a sequence of normal jump moves leading to equivalent normal forms of rational 3-tangles.
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
Developed criteria for selecting non-normalized models using NCE and score matching.
problem No information criteria for non-normalized models estimated by NCE or score matching.
method Developed information criteria based on discrepancy measures for non-normalized models estimated by NCE or score matching.
result The proposed criteria enable selection of the appropriate non-normalized model in a data-driven manner.
This paper investigates the impact of normalization on deep neural networks for click-through rate prediction.
problem The effect of normalization on deep neural network models for CTR estimation.
method Systematic study of various normalization approaches applied to feature embedding and MLP part of DNN models.
result Correct normalization significantly enhances model performance, as demonstrated by extensive experiments on real-world datasets.
The study finds abundant normal generators for mapping class groups.
problem Understanding normal generation in mapping class groups.
method Analyzing restrictions on invariant subsurfaces and Teichmüller spaces.
result Reducible mapping classes can normally generate mapping class groups based on their asymptotic translation lengths.
Spectral normalization stabilizes GAN training for better image generation.
problem Stability issues in GAN training.
method Spectral normalization technique to stabilize discriminator training.
result Spectrally normalized GANs (SN-GANs) generate better quality images.
New method trains normalizing flows using entropy-regularized transport.
problem Training continuous normalizing flows efficiently.
method Formulates flows as gradients of scalar potentials, training only these potentials.
result Trains normalizing flows without explicit flow computation during training.