Survival trees exhibit end-cut preference, leading to biased splits.
problem End-cut preference in survival trees causes biased splits and poor interpretability.
method Proposed a smooth sigmoid surrogate (SSS) approach to replace hard-threshold indicator function.
result Smooth sigmoid surrogate (SSS) effectively mitigates end-cut preference in survival trees.
Study on H-consistency bounds for machine learning surrogates.
problem Estimating target loss error relative to surrogate loss error in machine learning.
method Developed H-consistency bounds for various surrogates and loss functions. result Stronger guarantees than existing methods, offering distribution-dependent and -independent bounds.
We present surrogate regret bounds for arbitrary surrogate losses in the context of binary classification with label-dependent costs. Such bounds relate a classifier's risk, assessed with respect to a surrogate loss, to its cost-sensitive classification risk. Two approaches to surrogate regret bounds are developed. The…
Proposes sigmoidF1 loss for multilabel classification, improving performance metrics.
problem Lack of smooth, tractable loss functions for multilabel classification.
method Introduces sigmoidF1, a smooth F1 score surrogate loss function.
result sigmoidF1 outperforms other loss functions on various datasets and metrics.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
Assessing heterogeneous treatment effects has become a growing interest in advancing precision medicine. Individualized treatment effects (ITE) play a critical role in such an endeavor. Concerning experimental data collected from randomized trials, we put forward a method, termed random forests of interaction trees (RF…
New method initializes sigmoidal MLPs for interpretable shapes.
problem Creating interpretable decision boundaries in neural networks.
method Introducing a geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs) using tropical geometry.
result Sigmoidal MLPs can have decision boundaries aligned with prescribed shapes at initialization.
We establish linear regret bounds for convex smooth losses using Fenchel-Young losses.
problem Establishing linear regret bounds for convex smooth losses.
method Constructing a convex smooth surrogate loss using Fenchel-Young losses generated by the convolutional negentropy.
result We derive a smooth loss with a linear surrogate regret bound.
This paper characterizes and designs loss functions for robust classification with abstention.
problem Ensuring robustness against adversarial attacks and knowing when to abstain from prediction.
method Proposes adversarial robust reject option loss and characterizes surrogates for calibration.
result Shifted Double Ramp Loss and Shifted Double Sigmoid Loss satisfy the calibration conditions.
Symmetric losses improve classifier robustness from corrupted labels.
problem Improving classifier performance from corrupted labels.
method Symmetric losses that satisfy a certain condition.
result Symmetric losses enhance robust classification from corrupted labels.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of H-consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings. result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
In this work we provide a theoretical framework for structured prediction that generalizes the existing theory of surrogate methods for binary and multiclass classification based on estimating conditional probabilities with smooth convex surrogates (e.g. logistic regression). The theory relies on a natural characteriza…
Surrogate-based analysis of interactions via local effect smooths
problem Detecting and characterizing feature interactions in machine learning models
method Surrogate-based analysis using generalized additive models
result Empirical validation of effectiveness for pairwise interactions
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
We consider network sparsification as an L0-norm regularized binary optimization problem, where each unit of a neural network (e.g., weight, neuron, or channel, etc.) is attached with a stochastic binary gate, whose parameters are jointly optimized with original network parameters. The Augment-Reinforce-Merge (ARM),…
In statistical learning theory, convex surrogates of the 0-1 loss are highly preferred because of the computational and theoretical virtues that convexity brings in. This is of more importance if we consider smooth surrogates as witnessed by the fact that the smoothness is further beneficial both computationally- by at…
Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.
problem The trade-off between smoothness and margin-based losses in classification and structured prediction.
method Linear-Core (LC) Surrogates, a family of convex loss functions that stitch a linear core to a smooth tail.
result LC Surrogates achieve fast linear consistency rates while maintaining differentiability and strict H-consistency bounds. Paper establishes a universal growth rate for smooth surrogate losses in classification.
problem Analyzing growth rates of consistency bounds for various surrogate losses.
method Proves square-root growth rate for smooth margin-based losses; extends to multi-class classification.
result Demonstrates a universal square-root growth rate for smooth comp-sum and constrained losses.
Proposes a method for inference in high-dimensional classification with non-differentiable surrogate losses.
problem Lack of inference procedures for identifying driving factors in high-dimensional classification with non-differentiable surrogate losses.
method Kernel-smoothed decorrelated score and cross-fitted version for hypothesis tests and interval estimators.
result Valid and superior inference methods for high-dimensional classification with non-differentiable surrogate losses.
Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some rece…
Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.
problem Vanishing gradients in sigmoid-type networks.
method Mathematical arguments and two remedies: regularization and rescaling.
result Demonstrates effectiveness of regularization and rescaling in practice.
The paper proposes a method to model non-smooth functions using clustering, classification, and Gaussian process modeling.
problem Modeling discontinuities and non-smoothness in expensive computational models.
method Three-stage approach combining clustering, classification, and Gaussian process modeling.
result The approach successfully models discontinuities and non-smoothness in various functions.
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
Decision trees and shallow neural networks have different geometric complexities, impacting their interpretability and accuracy.
problem The geometric simplicity of decision boundaries in decision trees conflicts with the approximation capabilities of shallow neural networks.
method Analysis of the Radon total variation (RTV) seminorm to compare geometric complexity of decision regions and neural network approximations.
result Smooth barrier scores can approximate decision regions with finite RTV, but their performance depends on the tube-mass condition near the decision boundary.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
Deep neural networks improve surrogate models for non-smooth quantities in uncertain geometries.
problem Building accurate surrogates for non-smooth quantities in uncertain geometries.
method Deep neural networks for point evaluation of solutions to interface problems with geometric uncertainties.
result Neural networks provide good surrogates without suffering from the curse of dimensionality.
A new method for multi-expert learning-to-defer avoids optimization issues.
problem Optimization issues in multi-expert learning-to-defer systems.
method A decoupled surrogate with a softmax classifier head and independent sigmoid heads per expert.
result First multi-expert L2D guarantee with a constant not growing with the expert pool.
We show that any smooth bi-Lipschitz h can be represented exactly as a composition hm∘...∘h1 of functions h1,...,hm that are close to the identity in the sense that each (hi−Id) is Lipschitz, and the Lipschitz constant decreases inversely with the number m of functions com…
New method improves robustness of smoothed classifiers against adversarial attacks.
problem Improving robustness of smoothed classifiers against adversarial attacks.
method Proposes worst-case adversarial loss over input distributions as a robustness certificate, and uses duality and smoothness properties to provide an easy-to-compute upper bound.
result Shows superior robustness performance over state-of-the-art certified or heuristic methods.
Optimizes hard-to-optimize metrics using adaptive surrogates.
problem Training models with black-box and hard-to-optimize metrics.
method Expresses metric as a function of surrogates, solves optimization problem over relaxed surrogate space.
result Approach performs on par with known methods and adds value when metric form is unknown.
Paper proposes a cost-sensitive conformal training method with provably controllable learning bounds.
problem Uncertainty quantification and learning bounds in conformal prediction.
method Cost-sensitive conformal training algorithm that minimizes the expected size of prediction sets using rank weighting.
result Theoretical analysis shows tightness between weighted objective and expected size of conformal prediction sets.
We propose Top-N-Rank, a novel family of list-wise Learning-to-Rank models for reliably recommending the N top-ranked items. The proposed models optimize a variant of the widely used discounted cumulative gain (DCG) objective function which differs from DCG in two important aspects: (i) It limits the evaluation of DCG …
Bayesian optimization tackles non-smooth tuning problems.
problem Optimizing black-box functions with non-smoothness and limited samples.
method Proposed a clustered Gaussian process (cGP) model for non-smooth optimization.
result Improvement of up to 90% in performance for repetitive experiments.
PPOS improves PPO by smoothing the surrogate objective function.
problem Performance instability and optimization inefficiency in PPO.
method Use of a functional clipping method instead of a flat clipping method.
result PPOS conducts more accurate updates and outperforms other PPO variants.
Generative Bayesian Computation improves surrogates for expensive simulations.
problem Limitations of Gaussian process surrogates in handling complex, non-stationary data.
method Generative Bayesian Computation via Implicit Quantile Networks (IQNs).
result Generative Bayesian Computation outperforms traditional Gaussian process methods across various benchmarks.
DeepSeekMoE improves language model efficiency with shared experts and normalized gating.
problem Improving sample efficiency in language model architectures.
method Theoretical and empirical analysis of shared experts and normalized sigmoid gating.
result Theoretical and empirical evidence of improved sample efficiency with shared experts and normalized gating.
We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…
The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc…
We develop a new approach to learn the parameters of regression models with hidden variables. In a nutshell, we estimate the gradient of the regression function at a set of random points, and cluster the estimated gradients. The centers of the clusters are used as estimates for the parameters of hidden units. We justif…
Annealed Entropic Allocation improves ranking and selection by mitigating hard switching and improving finite-budget discrimination.
problem Sequential budget allocation in ranking and selection
method Annealed weighted soft-min framework
result Surrogate converges uniformly to the hard minimum, soft-min weights concentrate on active challengers, and target allocation map is continuous.
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functi…
Deep conditional generative models are developed to simultaneously learn the temporal dependencies of multiple sequences. The model is designed by introducing a three-way weight tensor to capture the multiplicative interactions between side information and sequences. The proposed model builds on the Temporal Sigmoid Be…
New method tests tree models without causing computational pressure.
problem Incompatibility of adversarial robustness testing with tree ensembles.
method Smooths tree ensembles with sigmoid functions and uses gradient descent.
result Successfully reveals adversarial vulnerability of tree ensemble models.
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
Paper shows SVM can achieve super fast convergence rates.
problem Understanding fast convergence rates for SVM.
method Presented a simple mechanism to obtain fast convergence rates for SVM.
result SVM can exhibit exponential convergence rates without hard Tsybakov margin condition.