It is common in modern prediction problems for many predictor variables to be counts of rarely occurring events. This leads to design matrices in which many columns are highly sparse. The challenge posed by such "rare features" has received little attention despite its prevalence in diverse areas, ranging from natural …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “rare features”
Mixup improves feature learning by mixing common and rare features.
problem Improving generalization in deep learning models.
method Mixup, a data augmentation technique, is applied to feature learning. Theoretical and experimental studies are conducted to understand its benefits.
result Mixup effectively learns rare features from common ones, leading to better generalization.
Approach for modeling EHR data with rare features, improving prediction and interpretation.
problem Challenges in modeling rare binary features in EHR data.
method Tree-guided feature selection and logic aggregation for large-scale regression.
result Improved prediction and model interpretation of suicide risk in EHR data.
Gradient descent with error feedback performs better than vanilla when features are rare.
problem Improving communication complexity in distributed optimization with rare features.
method Gradient descent with greedy sparsification and error feedback for rare features.
result Communication complexity improves as features become more rare, potentially better than vanilla GD.
Dropout Regularization in Extended Generalized Linear Models based on Double Exponential Familiesstat.ML
Dropout improves regularization in flexible models for rare features.
problem Understanding theoretical properties of dropout in generalized linear models.
method Theoretical analysis and application to adaptive smoothing with B-splines.
result Dropout prefers rare features in mean and dispersion parameters.
In many situations, classes of data points of primary interest also happen to be those that are least numerous. A well-known example is detection of fraudulent transactions among the collection of all financial transactions, the vast majority of which are legitimate. These types of problems fall under the label of `rar…
In rare disease physician targeting, a major challenge is how to identify physicians who are treating diagnosed or underdiagnosed rare diseases patients. Rare diseases have extremely low incidence rate. For a specified rare disease, only a small number of patients are affected and a fractional of physicians are involve…
Bayesian neural networks explore rare fluctuations for better feature learning.
problem Understanding rare but dominant fluctuations in Bayesian neural networks.
method Large-deviation theory and joint optimization over predictors and internal kernels.
result Posterior rate function optimization reveals data-dependent kernel selection.
Memorizing rare examples helps neural networks generalize better.
problem Improving generalization in deep learning models.
method Theoretical analysis and experiments on neural networks with composition capability.
result Memorizing rare examples can help neural networks make correct predictions on rare test examples.
Fine-grained pretraining improves neural network's ability to learn rare features.
problem Improving generalization in deep learning models.
method Introducing a hierarchical multi-view structure to confine input data distribution.
result Fine-grained pretraining leads to better accuracy on hard downstream test samples.
Study assesses weakly-supervised methods for rare outcomes in medical records.
problem Identifying patients with specific medical conditions using electronic health records.
method Compared three methods (PheNorm, MAP, and sureLDA) in simulations with varying outcomes and silver labels.
result No single method consistently outperformed others, but sureLDA often did well.
WEST uses EHRs and expert cases to improve rare disease phenotyping.
problem Limited labeled data for rare diseases.
method Weakly supervised transformer model trained on probabilistic silver-standard labels.
result WEST outperforms existing methods in phenotype classification and subphenotyping.
Evaluating the Role of Data Enrichment Approaches Towards Rare Event Analysis in Manufacturingstat.ML
This paper evaluates data enrichment techniques for rare event detection in manufacturing.
problem Rare events in manufacturing lead to unplanned downtime and high energy consumption.
method Time series data augmentation, sampling, and imputation techniques combined with supervised machine learning.
result Data enrichment enhances rare failure event detection and prediction by up to 48%.
PRESTO improves rare event prediction by shrinking towards proportional odds model.
problem Difficult to predict rare events due to class imbalance.
method PRESTO relaxes proportional odds model by estimating separate weights for transitions between categories, imposing L1 penalty to shrink towards proportional odds.
result PRESTO consistently estimates decision boundary weights under sparsity assumption, improving rare probability estimation.
Generative model improves scene graph prediction for rare compositions.
problem Challenges in inferring unseen or rare scene graph compositions.
method Synthesizing rare scene graphs by perturbing real ones, using conditional GANs.
result Marginal but consistent improvements in zero- and few-shot metrics.
The committor function is a central object of study in understanding transitions between metastable states in complex systems. However, computing the committor function for realistic systems at low temperatures is a challenging task, due to the curse of dimensionality and the scarcity of transition data. In this paper,…
The rich set of sensors in smartphones and wearable devices provides the possibility to passively collect streams of data in the wild. The raw data streams, however, can rarely be directly used in the modeling pipeline. We provide a generic framework that can process raw data streams and extract useful features related…
Rare dense solutions clusters in asymmetric binary perceptrons -- local entropy via fully lifted RDTstat.ML
New method finds rare dense clusters in asymmetric binary perceptrons, resolving algorithmic hardness.
problem Resolving algorithmic hardness in asymmetric binary perceptrons.
method Fully lifted random duality theory (fl RDT) and large deviation upgrade (sfl LD RDT).
result Local entropy breaks down for constraint densities in (0.77, 0.78) interval, matching current solver limits.
Prior knowledge on properties of a target model often come as discrete or combinatorial descriptions. This work provides a unified computational framework for defining norms that promote such structures. More specifically, we develop associated tools for optimization involving such norms given only the orthogonal proje…
Efficient RL algorithms for linear function approximation with limited adaptivity constraints.
problem Limited adaptivity in reinforcement learning with linear function approximation.
method Proposed two efficient online RL algorithms for episodic linear Markov decision processes under batch learning and rare policy switch models.
result Achieved efficient regret bounds for both batch learning and rare policy switch models, with substantial reduction in adaptivity.
Generative model simulates rare events for better decision making.
problem Rare events impact decision making but are hard to sample.
method Normalizing Flow coupled with Importance Sampling.
result Accurate estimation of rare events improves decision outcomes.
New method for mixed data FI controls type I error and achieves high power.
problem Statistical inadequacy of feature importance measures for mixed data.
method Combining CPI framework with sequential knockoffs for mixed data.
result Our method controls type I error and achieves high power for mixed data.
AI boosts study of rare weather extremes with lower costs.
problem Difficulty in studying rare weather events due to limited data and models.
method Coupling AI forecasts with physics models using rare-event algorithms.
result Efficiently characterizes very rare events like once-per-millennium heatwaves.
New RL algorithm achieves nearly optimal performance for linear MDPs.
problem Optimal reinforcement learning for episodic linear MDPs.
method Weighted linear regression with variance estimator and rare-switching policy.
result Achieves nearly minimax optimal regret .
Rare diseases affecting 350 million individuals are commonly associated with delay in diagnosis or misdiagnosis. To improve those patients' outcome, rare disease detection is an important task for identifying patients with rare conditions based on longitudinal medical claims. In this paper, we present a deep learning m…
Develops a method to estimate rare-event probabilities under distributional uncertainty.
problem Distributional uncertainty limits the effectiveness of rare-event simulation techniques.
method Wasserstein distributionally robust rare-event simulation (DRIS) framework.
result DRIS achieves vanishing relative error in estimating rare-event probabilities.
Extreme classification seeks to assign each data point, the most relevant labels from a universe of a million or more labels. This task is faced with the dual challenge of high precision and scalability, with millisecond level prediction times being a benchmark. We propose DEFRAG, an adaptive feature agglomeration tech…
Reduces false positives in classifying rare online platforms.
problem Challenges in accurately identifying rare online platforms with ML.
method Calibrated probabilities and ensembles to reduce bias.
result Significantly reduces false positives in rare event detection.
Consider a two-class classification problem where the number of features is much larger than the sample size. The features are masked by Gaussian noise with mean zero and covariance matrix , where the precision matrix is unknown but is presumably sparse. The useful features, also unknown, are sparse and e…
Enhanced diffusion sampling improves rare event sampling in biomolecular simulations.
problem Efficiently sampling rare transition events in biomolecular systems.
method Quantitative steering protocols to generate biased ensembles and exact reweighting.
result Fast, accurate, and scalable estimation of equilibrium properties.
Enhanced diffusion sampling tackles rare event sampling in biomolecular simulations.
problem Efficiently sampling rare transition events in biomolecular simulations.
method Quantitative steering protocols to generate biased ensembles, followed by exact reweighting.
result Fast, accurate, and scalable estimation of equilibrium properties for folding free energies.
GMM-HMMs improve malware classification compared to discrete HMMs.
problem Improving malware classification accuracy using GMM-HMM models.
method Used GMM-HMM models for opcode and entropy-based features comparison with discrete HMMs.
result GMM-HMMs generally improve significantly on entropy-based features classification results.
Proposes a method to ensure accurate estimation of rare events in AI systems.
problem Lack of efficiency guarantees in black-box systems for rare-event simulation.
method Integrates deep learning with importance sampling to create a statistically guaranteed estimator.
result Demonstrates effective estimation of rare-event probabilities in AI systems.
Rare diseases affect a relatively small number of people, which limits investment in research for treatments and cures. Developing an efficient method for rare disease detection is a crucial first step towards subsequent clinical research. In this paper, we present a semi-supervised learning framework for rare disease …
A method to improve clustering explainability using bagging and feature dropout.
problem Lack of explainability in clustering methods.
method Bagging and feature dropout to generate feature importance scores.
result Improved stability and robustness of cluster definition, especially in small-sample or noisy settings.
Given a labeled dataset that contains a rare (or minority) class of of-interest instances, as well as a large class of instances that are not of interest, how can we learn to recognize future of-interest instances over a continuous stream? We introduce RaRecognize, which (i) estimates a general decision boundary betwee…
This research examines rare spurious correlations in neural networks and their impact on accuracy and privacy.
problem Rare spurious correlations in neural networks and their privacy risks.
method Introducing spurious patterns correlated with a fixed class to a few training examples, analyzing regularization and Gaussian noise.
result Rare spurious correlations can significantly impact neural network accuracy and privacy, and specific mitigation methods can be effective.
Simulates patient pathways to detect delayed rare disease diagnoses.
problem Delayed rare disease diagnoses in France, causing health system and patient harm.
method Probabilistic modelling of patient pathways to create an alert system.
result Alert system detects and refers wandering patients to CRMRs.
Markov chain Monte Carlo (MCMC) algorithms for hidden Markov models often rely on the forward-backward sampler. This makes them computationally slow as the length of the time series increases, motivating the development of sub-sampling-based approaches. These approximate the full posterior by using small random subsequ…
Model predicts travel time under rare conditions using a vector-space model.
problem Predicting travel time under rare temporal conditions (e.g., holidays, school vacations) is challenging due to limited historical data and other temporal changes.
method Presented a vector-space model for encoding rare temporal conditions, allowing coherent representation learning across different conditions.
result Increased performance for travel time prediction over different baselines when using the vector-space encoding for representing the temporal setting.
FeAT improves OOD generalization by learning richer features.
problem Improving feature learning for out-of-distribution (OOD) generalization.
method Feature Augmented Training (FeAT) iteratively augments and retains features from different subsets of training data.
result FeAT effectively learns richer features, boosting OOD performance.
Study compares resampling methods for rare event prediction in longitudinal studies.
problem Predicting rare events in longitudinal follow-up studies.
method Comparison of resampling methods to improve standard regression models.
result Effect of sampling rate on model predictive performance.
Determining risk contributions of unit exposures to portfolio-wide economic capital is an important task in financial risk management. Computing risk contributions involves difficulties caused by rare-event simulations. In this study, we address the problem of estimating risk contributions when the total risk is measur…
AFR simplifies reducing reliance on spurious features, improving model performance.
problem Reducing reliance on spurious features for out-of-distribution generalization.
method Automatic Feature Reweighting (AFR) updates the model with a weighted loss.
result AFR improves model performance on benchmarks with minimal compute.
The paper analyzes logistic regression for rare events data, deriving new insights on estimator efficiency and sampling strategies.
problem Binary logistic regression for rare events data with significantly fewer events than controls.
method Derives asymptotic distribution of MLE, proves under-sampling advantage, and compares over-sampling efficiency.
result Under-sampling a small proportion of nonevents can improve efficiency in rare events data analysis.
Towards Dynamic Causal Discovery with Rare Events: A Nonparametric Conditional Independence Teststat.ML
New test uncovers causal links in rare event dynamics.
problem Causal discovery for rare event phenomena in dynamic systems.
method Nonparametric conditional independence test on time-invariant data.
result Validated across simulated and real-world datasets.
Feature selection is an important and challenging task in high dimensional clustering. For example, in genomics, there may only be a small number of genes that are differentially expressed, which are informative to the overall clustering structure. Existing feature selection methods, such as Sparse K-means, rarely tack…
Proposes Ada-Sit method for mortality prediction of rare diseases.
problem Data insufficiency and clinical diversity of rare diseases make mortality prediction hard.
method Initialization-sharing multi-task learning method (Ada-Sit) for fast adaptation to similar tasks.
result Experimental results show the proposed model is effective for mortality prediction of diverse rare diseases.