arXiv research

Study assesses weakly-supervised methods for rare outcomes in medical records.

The paper studies causal effects of multiple treatments in healthcare databases with rare outcomes.

New method uses surrogate outcomes and single-record data to improve suicide risk modeling.

Rare diseases affecting 350 million individuals are commonly associated with delay in diagnosis or misdiagnosis. To improve those patients' outcome, rare disease detection is an important task for identifying patients with rare conditions based on longitudinal medical claims. In this paper, we present a deep learning m…

Generative model simulates rare events for better decision making.

AUC is unreliable in rare event settings but stable with moderate numbers of events.

Algorithmic trading systems on DEXs reject most candidate tokens, but the counterfactual outcome of rejected candidates is rarely measured.

Modern treatments for Type 1 diabetes (T1D) use devices known as artificial pancreata (APs), which combine an insulin pump with a continuous glucose monitor (CGM) operating in a closed-loop manner to control blood glucose levels. In practice, poor performance of APs (frequent hyper- or hypoglycemic events) is common en…

PRESTO improves rare event prediction by shrinking towards proportional odds model.

Caus-Modens uses deep ensembles to better predict causal outcomes in hidden confounding scenarios.

This paper shows that one cannot learn the probability of rare events without imposing further structural assumptions. The event of interest is that of obtaining an outcome outside the coverage of an i.i.d. sample from a discrete distribution. The probability of this event is referred to as the "missing mass". The impo…

Generative AI models improve clinical trial data by generating survival outcomes.

Study uses LLMs to create personalized treatment plans for rare gynecological tumors.

Two new estimators reduce costs and improve accuracy for EHR outcome prediction.

Two algorithms achieve optimal regret with limited adaptivity in multinomial logistic bandits.

This study presents a long-term alternative formula for stock price variation described by a geometric Brownian motion on the basis of median instead of mean or expected values. The proposed method is motivated by the observation made in remote fields, where optimality of bet-hedging or diversification strategies is ex…

New method uses geometric mean to avoid non-collapsibility in case-control studies.

Many real-world prediction tasks have outcome variables that have characteristic heavy-tail distributions. Examples include copies of books sold, auction prices of art pieces, demand for commodities in warehouses, etc. By learning heavy-tailed distributions, "big and rare" instances (e.g., the best-sellers) will have a…

The paper introduces diagnostic transport maps to improve the reliability of rare event predictions.

Classification tasks are common across many fields and applications where the decision maker's action is limited by resource constraints. In direct marketing only a subset of customers is contacted; scarce human resources limit the number of interviews to the most promising job candidates; limited donated organs are pr…

In epidemiology, identifying the effect of exposure variables in relation to a time-to-event outcome is a classical research area of practical importance. Incorporating propensity score in the Cox regression model, as a measure to control for confounding, has certain advantages when outcome is rare. However, in situati…

Proposes DR-ME test for interpretable distributional treatment effects.

The Artificial Prediction Market is a recent machine learning technique for multi-class classification, inspired from the financial markets. It involves a number of trained market participants that bet on the possible outcomes and are rewarded if they predict correctly. This paper generalizes the scope of the Artificia…

The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically…

We consider rules for discarding predictors in lasso regression and related problems, for computational efficiency. El Ghaoui et al (2010) propose "SAFE" rules that guarantee that a coefficient will be zero in the solution, based on the inner products of each predictor with the outcome. In this paper we propose strong …

A simple spin system is constructed to simulate dynamics of asset prices and studied numerically. The outcome for the distribution of prices is shown to depend both on the dimension of the system and the introduction of price into the link measure. For dimensions below 2, the associated risk is high and the price distr…

New method for valid prediction intervals with coarsened data.

New approach to algorithmic fairness for human-AI collaboration considers compliance with human decisions.

Bayesian hypergraph inference models disease pathways from EHR data.

The paper shows how to audit fairness in decisions with hidden risk factors.

The study improves compound selection in in silico screening by focusing on model's ability to predict desirable outcomes.

It is common in modern prediction problems for many predictor variables to be counts of rarely occurring events. This leads to design matrices in which many columns are highly sparse. The challenge posed by such "rare features" has received little attention despite its prevalence in diverse areas, ranging from natural …

We study the behavior of simple models for financial markets with widely spread frequency either in the trading activity of agents or in the occurrence of basic events. The generic picture of a phase transition between information efficient and inefficient markets still persists even when agents trade on widely spread …

AI boosts study of rare weather extremes with lower costs.

With the increased availability of large databases of electronic health records (EHRs) comes the chance of enhancing health risks screening. Most post-marketing detections of adverse drug reaction (ADR) rely on physicians' spontaneous reports, leading to under reporting. To take up this challenge, we develop a scalable…

Develops a method to estimate rare-event probabilities under distributional uncertainty.

Reduces false positives in classifying rare online platforms.

Method finds counterfactual explanations for random forest models.

This paper evaluates data enrichment techniques for rare event detection in manufacturing.

Enhanced diffusion sampling improves rare event sampling in biomolecular simulations.

Enhanced diffusion sampling tackles rare event sampling in biomolecular simulations.

Proposes a method to ensure accurate estimation of rare events in AI systems.

In many situations, classes of data points of primary interest also happen to be those that are least numerous. A well-known example is detection of fraudulent transactions among the collection of all financial transactions, the vast majority of which are legitimate. These types of problems fall under the label of `rar…

Gynaecologists and obstetricians visually interpret cardiotocography (CTG) traces using the International Federation of Gynaecology and Obstetrics (FIGO) guidelines to assess the wellbeing of the foetus during antenatal care. This approach has raised concerns among professionals with regards to inter- and intra-variabi…

WEST uses EHRs and expert cases to improve rare disease phenotyping.

Rare diseases affect a relatively small number of people, which limits investment in research for treatments and cures. Developing an efficient method for rare disease detection is a crucial first step towards subsequent clinical research. In this paper, we present a semi-supervised learning framework for rare disease …

Given a labeled dataset that contains a rare (or minority) class of of-interest instances, as well as a large class of instances that are not of interest, how can we learn to recognize future of-interest instances over a continuous stream? We introduce RaRecognize, which (i) estimates a general decision boundary betwee…

This research examines rare spurious correlations in neural networks and their impact on accuracy and privacy.