Softplus regressions use multiple hyperplanes to classify data.
problem Classifying data with flexible nonlinear decision boundaries.
method Softplus function based regression models convolving gamma distributions.
result Softplus regressions achieve comparable classification accuracy to SVM but with less computation.
Softplus normalization improves uncertainty estimation in Bayesian CNNs.
problem Improving uncertainty estimation in Bayesian CNNs.
method Softplus normalization applied to the final layer of Bayesian CNNs with variational inference.
result The method achieves equivalent performance to frequentist inference while naturally incorporating uncertainty and regularization.
A new Bayesian multinomial regression model using permuted and augmented stick-breaking.
problem Modeling categorical response variables given covariates.
method Permuted and augmented stick-breaking (paSB) construction.
result Transforms multinomial regression into regression of stick-specific binary variables.
Neural networks can approximate any L^p functions on R^n.
problem Approximating functions on unbounded domains with neural networks.
method Monotone sigmoid, ReLU, ELU, Softplus, LeakyReLU activation functions.
result Shallow neural networks can arbitrarily well approximate L^p functions on R^n.
New method improves GANs by estimating density ratios in feature space with SP loss.
problem Filtering out unrealistic images from GANs trained with suboptimal discriminators.
method Develops DRE-F-SP method based on Softplus loss for density ratio estimation in feature space, and proposes three subsampling methods.
result Empirically shows substantial improvement over existing methods on synthetic and CIFAR-10 datasets.
Improved variational inference for logistic regression and classification.
problem Intractability of Evidence Lower Bound in variational logistic regression.
method Introducing a new bound for the expectation of softplus function, applied to variational logistic regression and Gaussian process classification.
result The new bound results in a tighter posterior and faster computation compared to Monte-Carlo methods.
A new method lifts training of input-convex neural networks to avoid dead weights and plateaued loss.
problem Training input-convex neural networks with non-negative weights.
method Introduces a hypernetwork that emits non-negative weights from a summary of the input batch, adding stochasticity to soften the loss landscape.
result The lift method achieves lower test loss than projected gradient descent and direct softplus reparametrization.
Deep networks can approximate various activation functions with modest adjustments.
problem Expressive power of deep neural networks with diverse activation functions.
method Approximation of any activation function in set A by ReLU networks with specific scaling factors.
result Approximation of any activation function in a specific subset of A by ReLU networks with (1,1) scaling factors.
New method calibrates adaptive learning rate to improve convergence of Adam.
problem Inconsistent adaptive learning rate across dimensions leads to convergence issues.
method Proposes a new AGM by calibrating the adaptive learning rate with a softplus function.
result Calibrated methods outperform existing AGMs and generalize better.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
New algorithm reduces sample complexity for learning CNNs.
problem Learning one-hidden-layer CNNs with various activation functions.
method Approximate gradient descent algorithm for training CNNs.
result Sample complexity matches information-theoretic lower bound for linear activation functions.
A neural network derived from first principles using MaxEnt.
problem Developing a neural network from first principles.
method Derived a neural network using the principle of Maximum Entropy, with linear dimension-reducing transformations and conditional mean estimators.
result Unified theoretical justification for activation functions like sigmoid, softplus, and relu.
AGGLIO optimizes non-convex functions with local convexity guarantees.
problem Optimizing non-convex functions with local convexity.
method Stage-wise, graduated optimization technique for locally convex functions.
result Global convergence to the global optimum for non-convex and locally convex objectives.
Global convergence of SGD proven for two-layer neural nets with regularization.
problem Proving global convergence of SGD for two-layer neural nets.
method Regularized empirical risk, SGD iterates, Villani functions.
result Global convergence of SGD for a special class of initializations.
LRA trains deep networks robustly with less sensitivity to initial weights.
problem Training deep networks is challenging due to issues like exploding and vanishing gradients.
method Local Representation Alignment (LRA) is a training procedure less sensitive to initial weights.
result LRA can train networks robustly, even with null initial weights, and outperforms other methods.
Gradient flows of neural networks converge to optimal values or diverge, with thresholds and asymptotic behaviors.
problem Understanding the convergence and divergence of gradient flows in neural networks.
method Analysis of gradient flows on loss landscapes of neural networks using o-minimal structures.
result Gradient flows either converge to optimal values or diverge to infinity, with thresholds and asymptotic behaviors.
Mish is a new activation function that improves neural network performance.
problem Improving the performance and training dynamics of neural networks.
method Mish is a self-regularized non-monotonic activation function defined as f(x)=xanh(softplus(x)). It outperforms other functions on benchmarks like ImageNet-1k and MS-COCO. result Mish outperforms Leaky ReLU and ReLU on benchmarks like MS-COCO and ImageNet-1k, respectively, with comparable network parameters.
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
Lower bound proves ridgeless regression performs poorly near interpolation threshold.
problem Proving performance of ridgeless regression near interpolation threshold.
method Distribution-independent lower bound for mean squared error in noisy ridgeless linear regression.
result Lower bound implies ridgeless regression performs poorly near interpolation threshold.
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
Minimum width for ReLU networks on compact domain is exactly max{d_x, d_y, 2}
problem Characterizing the minimum width for ReLU networks to approximate functions on compact domains
method Analyzing the minimum width for Lp approximation of Lp functions from [0,1]d to Rdy using ReLU-like activation functions result The minimum width for Lp approximation on a compact domain is exactly max{d_x, d_y, 2} for ReLU-like activation functions IGNIS uses neural networks to estimate copula parameters robustly.
problem Pathological properties of Archimedean copulas make traditional estimators brittle.
method Unified neural estimation framework with multi-input architecture and softplus output layer.
result Accurate and stable estimates for real-world datasets.
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
This work refines imitation learning to approximate optimal state-feedback policies for a quadcopter model.
problem Designing optimal control policies for complex systems like quadcopters.
method Supervised imitation learning using deep neural networks trained on optimal trajectories.
result Deep neural networks can approximate optimal state-feedback policies with high accuracy, even with two layers.
This work introduces an efficient method to sample high-quality images from conditional GANs.
problem Efficient subsampling of images from conditional GANs (cGANs) is challenging.
method Developed a novel conditional density ratio estimation method (cDRE-F-cSP) and rejection sampling scheme (cDR-RS).
result cDR-RS outperforms state-of-the-art methods in both effectiveness and efficiency.
This paper proposes a continuous timing strategy for growth vs. defensive style allocation.
problem Dynamic allocation of growth and defensive ETF baskets using macro-market timing signals.
method Continuous smooth score combining multiple factors, mapped to G/D weights, smoothed with EWMA.
result Continuous style timing strategy outperforms static benchmarks in risk-adjusted returns.