This study compares machine learning methods for high-cardinality categorical variables.
problem Machine learning struggles with high-cardinality categorical variables.
method Empirical comparison of tree-boosting, deep neural networks, and linear mixed effects models.
result Tree-boosting with random effects outperforms deep neural networks with random effects.
CardiCat generates synthetic data for high-cardinality tabular datasets.
problem Learning complexities of high-cardinality categorical features in tabular data.
method Substitutes one-hot encoding with regularized dual encoder-decoder embedding layers.
result Generates high-quality synthetic data with a smaller parameter space.
Structured high-cardinality data arises in many domains, and poses a major challenge for both modeling and inference. Graphical models are a popular approach to modeling structured data but they are unsuitable for high-cardinality variables. The count-min (CM) sketch is a popular approach to estimating probabilities in…
Novel GLMMNet model tackles high-cardinality categorical features in actuarial applications.
problem Inadequate encoding methods for high-cardinality categorical features in actuarial data.
method Generalised Linear Mixed Model Neural Network (GLMMNet) integrating a generalised linear mixed model in a deep learning framework.
result GLMMNet often outperforms or performs comparably with entity embedded neural networks, providing transparency.
Proposes a new DNN framework for count data with high-cardinality features.
problem Real-world data often have correlations and high-cardinality categorical features that traditional DNNs overlook.
method Introduces a hierarchical likelihood learning framework with gamma random effects for Poisson DNNs.
result Improves prediction performance by capturing nonlinear effects and subject-specific cluster effects.
Regularized target encoding beats traditional methods for high cardinality features in ML.
problem Efficiently encoding high cardinality categorical variables for ML algorithms.
method Regularized target encoding compared to traditional encodings like integer and one-hot encoding.
result Regularized target encoding consistently provided the best results in a large-scale benchmark experiment.
Statistical models usually require vector representations of categorical variables, using for instance one-hot encoding. This strategy breaks down when the number of categories grows, as it creates high-dimensional feature vectors. Additionally, for string entries, one-hot encoding does not capture information in their…
Combines boosting with Gaussian process and mixed effects models.
problem Model misspecifications and independence assumptions in boosting.
method Relaxes zero or linearity assumption in Gaussian process and mixed effects models, and independence assumption in boosting.
result Increased prediction accuracy compared to existing approaches.
Improved classification model for high-cardinality categorical predictors.
problem Handling high-cardinality categorical predictors and non-linear data.
method Data-driven binning of spline functions and shrinkage estimators.
result Improved classification precision with interpretable predictors.
For statistical learning, categorical variables in a table are usually considered as discrete entities and encoded separately to feature vectors, e.g., with one-hot encoding. "Dirty" non-curated data gives rise to categorical variables with a very high cardinality but redundancy: several categories reflect the same ent…
Unified comparison of gradient boosting algorithms for insurance claims.
problem Improving predictive accuracy and computational efficiency in insurance claim prediction.
method Unified notation and comprehensive numerical study comparing 12 gradient boosting algorithms on 5 datasets.
result No trade-off between model adequacy and predictive accuracy.
Boosting trees predict Twitch subscriptions from user activity.
problem Predicting Twitch user subscriptions from activity data.
method Used boosting trees and target-encodings for high cardinality categoricals.
result User activity can be better predicted than content alone.
HNHN learns from hypergraphs with hyperedge neurons for better classification.
problem Learning from hypergraphs with complex relationships.
method Hypergraph convolution network with hyperedge neurons and adaptive normalization.
result Improved classification accuracy and speed compared to state-of-the-art methods.
Method reduces categorical data to lower dimensions using density matrices.
problem Dimensionality reduction for categorical data.
method Density-matrix construction from class-conditional frequencies; spectral embedding.
result Low-dimensional spectral embeddings with controlled rank.
MMbeddings reduces categorical embeddings by treating them as latent effects, significantly decreasing parameters and mitigating overfitting.
problem Large cardinalities in categorical embeddings lead to high parameter counts and overfitting.
method MMbeddings treats embeddings as latent random effects in a variational autoencoder framework, reducing parameter count and mitigating overfitting.
result MMbeddings consistently outperforms traditional embeddings across various tasks, demonstrating its potential in machine learning applications.
New Krylov subspace methods speed up mixed-effects models with crossed random effects.
problem Slow computations for high-dimensional crossed random effects in mixed-effects models.
method Krylov subspace-based methods for generalized mixed-effects models with cross effects.
result Speedups by factors of up to 10,000 in computations for mixed-effects models.
This paper simplifies OPE in large state spaces using state abstractions.
problem Accurately evaluating policies offline in large state spaces.
method Developed a backward-model-irrelevance condition and an iterative state abstraction procedure.
result Deeply-abstracted states substantially simplify OPE sample complexity.
CoSMIC extends flow-based SVI to transdimensional problems.
problem Bayesian structure learning and model selection with multi-model parameter spaces.
method Normalizing flows with a combined stochastic variational transdimensional inference approach.
result Improved performance on high-cardinality model spaces.
Combines boosting and latent Gaussian models for better predictions.
problem Boosting's assumptions and latent Gaussian models' limitations.
method Integrates tree-boosting and latent Gaussian models.
result Increased prediction accuracy in simulations and real-world data.
This paper proposes a method to reduce complexity in GLMs with categorical predictors.
problem Wasteful, hard-to-interpret, and prone to overfitting of traditional one-hot encoding for high-cardinality categorical predictors.
method Clustering categories of categorical predictors through a numerical method that preserves or improves accuracy while reducing the number of coefficients.
result Clustering categories of categorical predictors reduces complexity substantially without harming accuracy.
MC-GMENN improves neural networks for clustered data using Monte Carlo methods.
problem Improving neural network performance on clustered data with correlations.
method MC-GMENN employs Monte Carlo methods to train generalized mixed effects neural networks.
result MC-GMENN outperforms existing models in generalization and quantifying inter-cluster variance.
Recommendation models mainly deal with categorical variables, such as user/item ID and attributes. Besides the high-cardinality issue, the interactions among such categorical variables are usually long-tailed, with the head made up of highly frequent values and a long tail of rare ones. This phenomenon results in the d…
This research compares two encoding methods for categorical attributes in machine learning, affecting model fairness.
problem The impact of encoding protected categorical attributes on fairness in machine learning models.
method Comparison of one-hot encoding and target encoding methods.
result Target encoding can lead to more unfair models compared to one-hot encoding due to induced bias.
Study investigates how preprocessing, feature selection, and model selection affect performance on imbalanced genetic data.
problem Challenges in using machine learning on imbalanced genetic datasets.
method Comparative analysis of data preprocessing, feature selection techniques, and machine learning models on imbalanced genetic data.
result Class-imbalanced target variables and skewed predictors have little to no impact on classification performance.
SYNTHONY selects tabular synthesizers based on stress profiling and user intent.
problem Non-uniform performance of tabular generative models across datasets.
method Stress profiling and intent-conditioned tabular synthesis selection.
result Meta-features predict synthesizer performance, improving selection accuracy.
Applied Data Scientists throughout various industries are commonly faced with the challenging task of encoding high-cardinality categorical features into digestible inputs for machine learning algorithms. This paper describes a Bayesian encoding technique developed for WeWork's lead scoring engine which outputs the pro…
Bayesian optimization tackles expensive discrete and mixed parameter spaces.
problem Optimizing expensive functions with discrete and mixed parameters.
method Probabilistic reparameterization to maximize expectation of AF over continuous parameters.
result Our approach provably converges to a maximizer of the AF and enjoys the same regret bounds as standard BO.