Paper proposes algorithms for active learning of reject option classifiers.
problem Active learning of reject option classifiers is unaddressed in machine learning.
method Developed novel algorithms using double ramp and double sigmoid loss functions.
result Proposed algorithms efficiently reduce the number of labeled examples required.
This paper characterizes and designs loss functions for robust classification with abstention.
problem Ensuring robustness against adversarial attacks and knowing when to abstain from prediction.
method Proposes adversarial robust reject option loss and characterizes surrogates for calibration.
result Shifted Double Ramp Loss and Shifted Double Sigmoid Loss satisfy the calibration conditions.
SIGTRON improves classification accuracy for imbalanced datasets.
problem Improving classification accuracy for imbalanced datasets.
method SIGTRON is a new sigmoid function with a convex loss function for imbalanced classification.
result SIGTRON models outperform existing methods in balanced and imbalanced datasets.
01 loss neural network improves adversarial robustness on CIFAR10.
problem Improving machine learning models' resistance to adversarial attacks.
method Proposes a single hidden layer 01 loss neural network trained with stochastic coordinate descent.
result 01 loss network has the largest adversarial distortion compared to other models.
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
We analyze double descent in finite-width neural networks using influence functions.
problem Understanding double descent in finite-width neural networks.
method Using influence functions to derive population loss bounds and investigate loss function effects.
result Derived bounds exhibit double descent behavior at the interpolation threshold.
We address the problem of aggregating an ensemble of predictors with known loss bounds in a semi-supervised binary classification setting, to minimize prediction loss incurred on the unlabeled data. We find the minimax optimal predictions for a very general class of loss functions including all convex and many non-conv…
We present surrogate regret bounds for arbitrary surrogate losses in the context of binary classification with label-dependent costs. Such bounds relate a classifier's risk, assessed with respect to a surrogate loss, to its cost-sensitive classification risk. Two approaches to surrogate regret bounds are developed. The…
Balanced Activation improves object detection performance on long-tailed datasets.
problem Mismatch between training and testing label distributions in object detection.
method Introduces Balanced Activation (Balanced Softmax and Balanced Sigmoid) to address label distribution shift.
result Balanced Activation provides ~3% gain in mAP on LVIS-1.0 compared to state-of-the-art methods.
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
Pixel-space diffusion models outperform latent models on high-resolution image synthesis.
problem Efficiency and quality trade-off in high-resolution image synthesis.
method Sigmoid loss-weighting, simplified architecture, and resolution scaling.
result Achieved 1.5 FID on ImageNet512, new SOTA results on other datasets.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
Analyzes double descent in binary classification models with different losses.
problem Understanding the double descent phenomenon in binary classification models.
method Analytic study of gradient descent with logistic and square losses on binary linear classification models.
result The double descent phenomenon persists but with differences compared to logistic loss.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
Paper investigates differentiable fuzzy implications and their suitability for learning.
problem Analyzing differentiable fuzzy implications and their suitability for learning.
method Investigates the properties of fuzzy implications in a differentiable setting and introduces a new family of fuzzy implications.
result Various fuzzy implications are unsuitable for differentiable learning, and a new family of fuzzy implications is introduced.
CNN predicts stock fluctuations using company news headlines.
problem Predicting next-day stock fluctuations based on company-specific news.
method Convolutional Neural Network (CNN) with reduced filter dimensions and multiple hidden layers. Fine-tuned word embeddings and various filter widths.
result 61.7% classification accuracy achieved using pre-learned embeddings.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
We describe loss surfaces using topological Betti numbers.
problem Understanding the complexity and structure of loss surfaces in neural networks.
method Topological analysis using Betti numbers for multilayer neural networks.
result Loss complexity is influenced by the number of hidden units and activation function.
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
Symmetric losses improve classifier robustness from corrupted labels.
problem Improving classifier performance from corrupted labels.
method Symmetric losses that satisfy a certain condition.
result Symmetric losses enhance robust classification from corrupted labels.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Study investigates overparametrization in survival models, revealing complex loss behavior.
problem Understanding overparametrization in survival models through interpolation.
method Defined interpolation and finite-norm interpolation, rigorously analyzed four survival models.
result Overparametrization can lead to improved performance in survival models, contrary to classical learning theory.
Generalizes double backpropagation for neural networks in Hilbert space.
problem Lack of a general description of derivatives in neural network loss functions.
method Develops a Hilbert space framework to cover various special cases of double backpropagation.
result Optimized backpropagation rules for real-world scenarios, reducing calculations by up to 33%.
Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.
problem Lack of smooth, tractable loss functions for multilabel classification.
method Introduces sigmoidF1, a smooth F1 score surrogate loss function.
result sigmoidF1 outperforms other loss functions on various datasets and metrics.
Study predicts doubling of U.S. maize insurance claims due to climate change.
problem Climate change increases U.S. maize loss probability, impacting insurance claims.
method Neural Network Monte Carlo simulations to predict crop loss metrics.
result Doubling of annual probability of maize Yield Protection insurance claims by mid-century.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
Early stopping improves neural networks' performance on binary classification tasks.
problem Improving shallow ReLU networks' performance on binary classification tasks.
method Gradient descent with early stopping on binary classification data.
result Gradient descent with early stopping achieves population risk arbitrarily close to optimal.
New method optimizes neural network learning by adjusting random parameters to target function features.
problem Difficulty in setting optimal random parameters for neural network learning.
method Adjusts sigmoid slopes and positions to target function features in a randomized learning method.
result Significantly better approximation of complex target functions compared to standard methods.
In this paper, we propose majority voting neural networks for sparse signal recovery in binary compressed sensing. The majority voting neural network is composed of several independently trained feedforward neural networks employing the sigmoid function as an activation function. Our empirical study shows that a choice…
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
We present a method for training multi-label, massively multi-class image classification models, that is faster and more accurate than supervision via a sigmoid cross-entropy loss (logistic regression). Our method consists in embedding high-dimensional sparse labels onto a lower-dimensional dense sphere of unit-normed …
Lower bound proves ridgeless regression performs poorly near interpolation threshold.
problem Proving performance of ridgeless regression near interpolation threshold.
method Distribution-independent lower bound for mean squared error in noisy ridgeless linear regression.
result Lower bound implies ridgeless regression performs poorly near interpolation threshold.
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
Deep networks can interpolate noisy data without losing generalization.
problem Characterizing the relationship between interpolation and generalization in overparameterized deep networks.
method Analyzing the loss landscape of neural network functions over volumes around training data points, varying model parameters and training epochs.
result Loss sharpness in the input space follows a double descent, with large models predicting noisy targets over larger volumes around training data points.
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherit…
We propose a differentiable sigmoid function for efficient p-value calculation in clustering.
problem Efficient and accurate p-value calculation for clustering algorithms.
method Designed a differentiable sigmoid function to approximate the Dip-p-value transformation.
result Accelerates computation and integrates well with gradient descent-based learning schemes.
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of H-consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings. result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
Study on H-consistency bounds for machine learning surrogates.
problem Estimating target loss error relative to surrogate loss error in machine learning.
method Developed H-consistency bounds for various surrogates and loss functions. result Stronger guarantees than existing methods, offering distribution-dependent and -independent bounds.
Paper develops a duality approach for robust loss functions in infinite-dimensional RKHSs.
problem Robustness issues in infinite-dimensional RKHSs with operator-valued kernels.
method Develops a duality approach to solve OVK machines for various loss functions.
result Empirical improvements and theoretical stability analysis for robust structured data applications.
Double descent phenomenon explained in simple terms.
problem Understanding the surprising drop in test error in overparameterized models.
method Informal explanation using linear algebra and probability, visual intuition with polynomial regression, mathematical analysis with ordinary linear regression.
result Three factors create double descent: data undersampling, model size, and parameter count. Ablating any one of these factors prevents double descent.
`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…
Survival trees exhibit end-cut preference, leading to biased splits.
problem End-cut preference in survival trees causes biased splits and poor interpretability.
method Proposed a smooth sigmoid surrogate (SSS) approach to replace hard-threshold indicator function.
result Smooth sigmoid surrogate (SSS) effectively mitigates end-cut preference in survival trees.
Model shows loss curve with two distinct exponents due to sparse activations.
problem Sparse activations impact neural network scaling laws.
method Introduced a model for neural scaling laws under sparse activations, derived asymptotic population loss, and analyzed gradient-descent dynamics.
result Loss curve exhibits double-descent peak near interpolation threshold with two distinct scaling exponents.