TimeMCL forecasts diverse time series futures using neural networks and WTA loss.
problem Forecasting multiple plausible time series futures.
method Multiple Choice Learning (MCL) framework with Winner-Takes-All (WTA) loss.
result TimeMCL efficiently predicts diverse time series futures at low computational cost.
New method uses geometric properties for better density estimation.
problem Uncertainty quantification in ambiguous tasks.
method Winner-takes-all training with centroidal Voronoi tessellations.
result Improved quantization and density estimation.
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
aMCL uses annealing to improve hypothesis diversity in ambiguous tasks.
problem Limitations of Winner-takes-all in predicting plausible hypotheses.
method Combines simulated annealing with Multiple Choice Learning (MCL).
result Enhanced exploration of hypothesis space during training.
Stochastic LWTA networks resist adversarial attacks while maintaining accuracy.
problem Adversarial robustness of neural networks.
method Replaced ReLU with stochastic LWTA activations, trained with Variational Bayesian and PGD.
result Stochastic LWTA networks achieve state-of-the-art robustness against adversarial attacks.
We propose a simple change to existing neural network structures for better defending against gradient-based adversarial attacks. Instead of using popular activation functions (such as ReLU), we advocate the use of k-Winners-Take-All (k-WTA) activation, a C0 discontinuous function that purposely invalidates the neural …
Inspired by the advances in biological science, the study of sparse binary projection models has attracted considerable recent research attention. The models project dense input samples into a higher-dimensional space and output sparse binary data representations after the Winner-Take-All competition, subject to the co…
Linear memory stores associations up to a logarithmic scale, but listwise retrieval can handle a quadratic scale.
problem How many key-value associations can a linear memory store?
method Analyzed linear memory models for top-1 and listwise retrieval, proving phase transitions and developing asymptotic theories.
result Linear memory has a logarithmic capacity for top-1 retrieval and a quadratic capacity for listwise retrieval.
rMCL improves on MCL by preserving diversity in predictions for regression problems.
problem Multimodal density estimation in regression settings with multiple targets.
method rMCL uses a learned scoring scheme based on Voronoi tessellations to maintain diversity among predictions.
result rMCL outperforms existing MCL variants in sound source localization tasks.
To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…
VOWEL trains WTA-SNNs for multi-valued events, overcoming resource limitations.
problem Training WTA-SNNs for multi-valued events is challenging due to non-differentiability and recurrent behavior.
method Develops a variational online local training rule (VOWEL) for WTA-SNNs using local pre- and post-synaptic information and a common reward signal.
result VOWEL outperforms conventional binary SNNs in real-world neuromorphic datasets with multi-valued events.
A new neural network method improves interpretability and detection of outliers.
problem Improving interpretability and detection of outliers in neural networks.
method Prototype-based learning (PbL) using a winner-take-all (WTA) network with two prototypes: positive and negative.
result The negative prototype is similar to the positive one, aligning with the BCM theory.
We present a modelling framework for the investigation of prototype-based classifiers in non-stationary environments. Specifically, we study Learning Vector Quantization (LVQ) systems trained from a stream of high-dimensional, clustered data.We consider standard winner-takes-all updates known as LVQ1. Statistical prope…
New framework for evaluating ad auctions using stochastic modeling.
problem Challenges in evaluating deterministic ad auctions.
method Repurposed bid landscape model to approximate propensity scores, enabling robust OPE estimators.
result Remarkable alignment with online A/B test results, achieving 92% MDA in CTR prediction.
We examine methods for clustering in high dimensions. In the first part of the paper, we perform an experimental comparison between three batch clustering algorithms: the Expectation-Maximization (EM) algorithm, a winner take all version of the EM algorithm reminiscent of the K-means algorithm, and model-based hierarch…
New study shows FTRL mechanism works with correlated events.
problem Forecasting competitions with correlated events.
method Introduces block correlation and uses FTRL mechanism.
result FTRL mechanism retains ε-optimal guarantee with O(b2log(n)/ε2) events for correlated events. Neuromorphic column performs online unsupervised clustering.
problem Real-time clustering of streaming data.
method Localized, spike timing-dependent plasticity (STDP) neural column.
result Prototype column performs similarly to k-means clustering.
Neural symbolic processing aims to combine the generalization of logical learning approaches and the performance of neural networks. The Neural Theorem Proving (NTP) model by Rocktaschel et al (2017) learns embeddings for concepts and performs logical unification. While NTP is promising and effective in predicting fact…
Power-law portfolios improve diversification by scaling weights sub-linearly.
problem Optimization methods struggle with unstable pair correlations and non-Gaussian risk measures.
method Construct portfolios with penalty proportional to arbitrary order moment of returns, leading to sub-linear weight scaling.
result Infinite order power-law portfolios are perfectly diversified, improving diversification over Kelly portfolios.
New deep learning model robust to adversarial attacks using stochastic LWTA units.
problem Adversarial robustness in deep learning networks.
method Introduces deep networks with stochastic LWTA activations, combining them with Bayesian non-parametric tools.
result Achieves high robustness to adversarial perturbations, outperforming state-of-the-art methods.
A new factor analysis method using ICA reduces portfolio concentration and diversifies excess kurtosis.
problem Standard factor analysis suffers from issues with pairwise correlations of asset returns.
method Identifies factors based on non-Gaussianity instead of variance, using ICA.
result Fat-tailed portfolios significantly reduce portfolio concentration and winner-takes-all problem.
New model improves deep learning robustness against adversarial attacks.
problem Improving adversarial robustness of deep learning models.
method Local competition principle, LWTA nonlinearities, Bayesian non-parametrics.
result The new model achieves high robustness to adversarial perturbations on MNIST and CIFAR10 datasets.
New analysis shows rational actors will deploy AGI despite negative social value due to catastrophic risk.
problem Rational actors will deploy AGI despite negative social value due to shared catastrophic risk.
method Continuous-time preemption game with shared catastrophic externalities, showing suicide region and welfare distortion.
result The suicide region widens as catastrophic risk grows, and two mechanisms can close it.
New research shows many recent defenses against adversarial examples are ineffective against black-box attacks.
problem The robustness of recent defenses against adversarial examples is insufficient, especially against black-box attacks.
method Evaluation of nine defenses on two black-box adversarial models and six attacks on CIFAR-10 and Fashion-MNIST datasets.
result Most recent defenses provide only marginal improvements in security (<25%) compared to undefended networks. This work tackles representation learning by introducing stochastic competition-based activations.
problem Learning diversified representations in deep learning models.
method Combining information-theoretic arguments with stochastic competition-based activations, using Stochastic Local Winner-Takes-All (LWTA) units.
result The proposed method yields significant discriminative representation learning abilities and allows for a principled investigation of intermediate network representations.
A new method for multi-expert learning-to-defer avoids optimization issues.
problem Optimization issues in multi-expert learning-to-defer systems.
method A decoupled surrogate with a softmax classifier head and independent sigmoid heads per expert.
result First multi-expert L2D guarantee with a constant not growing with the expert pool.
Historical tontines promised enormous rewards to the last survivors at the expense of those who died early. While this design appealed to the gambling instinct, it is a suboptimal way to manage longevity risk during retirement. This is why fair life annuities making constant payments -- where the insurance company is e…
Introduces Fitzpatrick losses, tighter than Fenchel-Young losses.
problem Improving loss functions for machine learning.
method Introduces Fitzpatrick losses based on the Fitzpatrick function.
result Fitzpatrick losses are tighter than Fenchel-Young losses.
We study losses for binary classification and class probability estimation and extend the understanding of them from margin losses to general composite losses which are the composition of a proper loss with a link function. We characterise when margin losses can be proper composite losses, explicitly show how to determ…
We present the Tamed Cross Entropy (TCE) loss function, a robust derivative of the standard Cross Entropy (CE) loss used in deep learning for classification tasks. However, unlike other robust losses, the TCE loss is designed to exhibit the same training properties than the CE loss in noiseless scenarios. Therefore, th…
Unified surrogate loss framework for multi-label learning with strong consistency guarantees.
problem Improving consistency and accounting for label correlations in multi-label learning.
method Introducing multi-label logistic loss and extending it to comprehensive multi-label comp-sum losses, proving strong consistency guarantees for any multi-label loss.
result Unified surrogate loss framework benefiting from strong consistency guarantees for any multi-label loss.
This paper introduces new loss functions for balanced multi-class classification.
problem Balancing class imbalance in multi-class classification.
method Introduces two new surrogate loss families: GLA and GCA.
result GCA losses offer stronger theoretical guarantees in imbalanced settings.
We present α-loss, α∈[1,∞], a tunable loss function for binary classification that bridges log-loss (α=1) and 0-1 loss (α=∞). We prove that α-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal y…
This work broadens calibeating to various proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses.
method Regret minimization and Bregman divergence approach.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
This work generalizes calibeating for a broader range of proper losses using Bregman divergence.
problem Calibration for a wide range of proper losses beyond Brier and log loss.
method Regret minimization based on Bregman divergence for a family of proper losses.
result U-calibration results for a family of Tsallis losses with logarithmic regret and dimension independence.
Proposes squentropy loss for improved classification accuracy and model calibration.
problem Theoretical and empirical evidence for cross-entropy loss is lacking.
method Introduces squentropy loss as the sum of cross-entropy and average square loss over incorrect classes.
result Squentropy loss outperforms cross-entropy and rescaled square losses in classification accuracy and model calibration.
New loss function calibrates WW-hinge loss for multiclass SVM.
problem WW-hinge loss not calibrated with 0-1 loss.
method Introduced ordered partition loss and proved WW-hinge loss is calibrated.
result WW-hinge loss is calibrated with ordered partition loss.
The study analyzes a model for aggregate losses with dependent and overdispersed inter-losses times.
problem Analyzing aggregate loss models with dependent and overdispersed inter-losses times.
method The study uses a two-state Markovian arrival process (MAP2) and a Markov renewal process to model the inter-losses times. Severities are modeled using a heavy-tailed, double-Pareto Lognormal distribution. The model is estimated via direct maximization of the likelihood function.
result The model with dependence and overdispersion in inter-losses times leads to higher capital charges compared to a Poisson process.
Symmetric losses improve classifier robustness from corrupted labels.
problem Improving classifier performance from corrupted labels.
method Symmetric losses that satisfy a certain condition.
result Symmetric losses enhance robust classification from corrupted labels.
Two new algorithms improve performance in adversarial bandits with unbounded losses.
problem Adversarial Multi-Armed Bandits with unbounded losses.
method Developed UMAB-NN and UMAB-G for non-negative and general unbounded losses respectively.
result UMAB-NN achieves the first adaptive and scale-free regret bound for non-negative unbounded losses.
Paper explores connections between loss functions and consistency in binary classification and regression.
problem Consistency in binary classification and regression applications.
method Characterization of conformable loss functions and derivation of a new Huber-type loss function.
result Margin-based loss functions are equivalent to loss functions of squared standardized logistic regression residuals.
Paper introduces a new topological loss for better convergence.
problem Optimizing topological losses for model's desired topological behavior.
method Introduces a new regularized topology-aware loss function.
result Guarantees efficient optimization of the new loss function.
Novel loss functions improve decision tree learning from noisy data.
problem Training decision trees with noisy labels.
method Introducing distribution losses and a new negative exponential loss.
result The negative exponential loss leads to efficient and robust decision tree learning.
Theoretical analysis of cross-entropy loss functions and their robustness.
problem Guarantees for using cross-entropy as a surrogate loss function.
method Theoretical analysis of a broad family of loss functions, including cross-entropy.
result First H-consistency bounds for comp-sum losses and smooth adversarial comp-sum losses. This paper improves operational risk modeling by selecting better loss severity distributions.
problem Inconsistent regulatory capital calculations due to changing loss severity distribution families.
method Presented truncation probability estimates and a consistent quantile scoring function for selection criteria. Also, recommended collecting loss frequencies below the minimum reporting threshold.
result More stable regulatory capital calculations through better selection of loss severity distributions.
This paper improves loss functions for deep learning with noisy labels.
problem Training deep neural networks with noisy labels.
method The paper introduces a normalization technique to make any loss function robust to noisy labels and proposes a framework called Active Passive Loss (APL) to combine robust loss functions.
result The proposed APL framework consistently outperforms state-of-the-art methods, especially under high noise rates.
Classification is the most important process in data analysis. However, due to the inherent non-convex and non-smooth structure of the zero-one loss function of the classification model, various convex surrogate loss functions such as hinge loss, squared hinge loss, logistic loss, and exponential loss are introduced. T…
We study cross-country GDP losses due to financial crises in terms of frequency (number of loss events per period) and severity (loss per occurrence). We perform the Loss Distribution Approach (LDA) to estimate a multi-country aggregate GDP loss probability density function and the percentiles associated to extreme eve…