We present a general framework, the coupled compound Poisson factorization (CCPF), to capture the missing-data mechanism in extremely sparse data sets by coupling a hierarchical Poisson factorization with an arbitrary data-generating model. We derive a stochastic variational inference algorithm for the resulting model …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “non-random missing data”
The paper analyzes sparse PCA for incomplete data and proves support recovery conditions.
problem Support recovery in sparse PCA with non-random missing data.
method Semidefinite relaxation of the -regularized PCA problem.
result Support of the sparse leading eigenvector can be recovered with high probability.
Proposes SSL method for non-randomly sampled data.
problem Evaluation of prediction rules under non-random sampling.
method Two-step procedure with imputation and augmentation.
result Proposed method outperforms supervised methods in efficiency.
Proposes a method to combine datasets with missing values using Gaussian process latent variables.
problem Combining datasets with missing values under non-Missing at Random (NMAR) missingness.
method Gaussian process latent variable model for non-MAR missing data.
result Valid estimates are obtained using the proposed method, while existing methods provide severely biased estimates.
The paper proposes a method to assess surrogate heterogeneity in non-randomized data.
problem Lack of methods to evaluate surrogate heterogeneity in non-randomized data.
method Proposes a framework using meta-learners to assess surrogate heterogeneity in real-world data.
result Identifies individuals for whom the surrogate is a valid replacement of the primary outcome.
Many real datasets contain values missing not at random (MNAR). In this scenario, investigators often perform list-wise deletion, or delete samples with any missing values, before applying causal discovery algorithms. List-wise deletion is a sound and general strategy when paired with algorithms such as FCI and RFCI, b…
Proposes MGPLL for PL learning with non-random noise.
problem Partial label learning with non-random label noise.
method Bi-directional mapping framework, conditional noise label generation, multi-class predictor, adversarial learning.
result Demonstrates state-of-the-art performance in partial label learning.
It is hypothesized that price charts can be empirically decomposed into two components as random and non random. The non random component, which can be treated as approximately regular behavior of the prices (trend) in an epoch, is a geometric line. Thus, the random component fluctuates around the non random component …
This paper studies node embeddings of networks, revealing their geometric properties.
problem Understanding the geometric properties of node embeddings in random networks.
method Characterization of ergodic limits, generalization, and convex relaxations of random walk node embedding objectives.
result The optimal node embedding Grammians have rank 1 for a nuclear norm relaxation of the non-randomized objective.
Enhanced ELM reduces randomness in neural network training.
problem Challenges in ELM architecture design and sensitivity to random weight initialization.
method Introduces Effective Non-Random ELM (ENR-ELM) incorporating signal processing concepts.
result ENR-ELM simplifies architecture design and eliminates random weight selection.
Paper develops new patterns for unique matrix completions.
problem Developing unique completions for non-random matrix patterns.
method Formulated low-rank matrix completion using Plucker coordinates.
result Provides two families of patterns for any rank.
For the pedestrian observer, financial markets look completely random with erratic and uncontrollable behavior. To a large extend, this is correct. At first approximation the difference between real price changes and the random walk model is too small to be detected using traditional time series analysis. However, we s…
Study relaxes identification assumptions for natural direct effects in non-randomized settings.
problem Identifying causal direct effects under unmeasured confounding.
method Developed relaxed conditions for identifying natural direct effects in non-randomized settings.
result Identified natural direct effect under unmeasured confounding conditions.
Causal Responder Detectionstat.ME
CARD detects treatment responders with machine learning and adjustment.
problem Identifying responders in non-random treatment settings.
method Conformal prediction, machine learning, propensity score adjustment.
result High power responder detection in various scenarios.
To have a superior generalization, a deep learning neural network often involves a large size of training sample. With increase of hidden layers in order to increase learning ability, neural network has potential degradation in accuracy. Both could seriously limit applicability of deep learning in some domains particul…
A new model for the stock market price analysis is proposed. It is suggested to look at price as an everywhere discontinuous function of time of bounded variation.
For the pedestrian observer, financial markets look completely random with erratic and uncontrollable behavior. To a large extend, this is correct. At first approximation the difference between real price changes and the random walk model is too small to be detected using traditional time series analysis. However, we s…
A new model for the stock market price analysis is proposed. It is suggested to look at price as an everywhere discontinuous function of time of bounded variation.
The influence of missing data mechanisms and simple missing data handling techniques on fairnessstat.ML
Study on missing data mechanisms and simple imputation methods in fairness of machine learning algorithms.
problem Impact of missing data mechanisms and simple imputation methods on fairness of machine learning algorithms.
method Three popular datasets for classification fairness were used. Missing values were generated using three missing data mechanisms. Various missing data handling techniques (listwise deletion, mean imputation, mode imputation, multiple imputation) were applied to the datasets. Fairness was assessed using classification algorithms (random forests).
result Missing data mechanism does not significantly impact fairness; listwise deletion gives highest fairness on average.
Missing data imputation can help improve the performance of prediction models in situations where missing data hide useful information. This paper compares methods for imputing missing categorical data for supervised classification tasks. We experiment on two machine learning benchmark datasets with missing categorical…
Develops a VAE model for datasets with missing data.
problem Applying VAEs to datasets with missing data.
method A novel latent variable model of a corruption process generating missing data, with a tractable ELBO.
result Improved marginal log-likelihood and better missing data imputation compared to existing approaches.
Autoencoder neural network is implemented to estimate the missing data. Genetic algorithm is implemented for network optimization and estimating the missing data. Missing data is treated as Missing At Random mechanism by implementing maximum likelihood algorithm. The network performance is determined by calculating the…
New algorithms handle missing data to improve fairness in machine learning.
problem Missing values in data can lead to unfair outcomes in machine learning models.
method Developed scalable and adaptive algorithms to handle missing values while preserving predictive information.
result Our adaptive algorithms consistently achieve higher fairness and accuracy than standard impute-then-classify methods.
Trinary decision tree improves handling of missing data in machine learning.
problem Improving accuracy in decision tree algorithms when dealing with missing data.
method Introduces Trinary decision tree, which does not assume missing values contain information about the response.
result Trinary decision tree outperforms other algorithms in Missing Completely at Random settings, especially when data is only missing out-of-sample.
DeepIFSAC uses attention mechanisms and contrastive learning to impute missing values in tabular data.
problem Missing values in tabular data, especially when high and not random.
method Row and column attention in a contrastive learning framework with CutMix data augmentation.
result Proposed method outperforms state-of-the-art methods for missing rates between 10% and 90% and various missing value types.
This paper compares imputation and direct parameter estimation methods for missing data in correlation matrix visualization.
problem Missing data challenges in estimating correlation coefficients for accurate visualization.
method Comparison of imputation and direct parameter estimation methods for handling missing data.
result Direct parameter estimation (DPER) outperforms imputation for accurate correlation matrix visualization.
Missing Data in Signal Processing and Machine Learning: Models, Methods and Modern Approacheseess.SP
This tutorial covers methods for handling missing data in SP and ML.
problem Dealing with missing data in signal processing and machine learning.
method Grouping strategies into three tasks: imputation, estimation, and prediction.
result Promising and future research directions are discussed.
IFGAN uses feature-specific GANs for missing value imputation.
problem Missing value imputation in data mining.
method Feature-specific Generative Adversarial Networks (GAN).
result IFGAN outperforms state-of-the-art algorithms in various missing conditions.
Missing data enhances privacy in differential privacy.
problem Privacy preservation in datasets with missing values.
method Formalized missing data as a privacy amplification mechanism within differential privacy.
result Incomplete data can yield privacy amplification for differentially private algorithms.
The paper explores how missing data problems are related to causal inference.
problem Missing data in experiments makes causal inference difficult.
method The paper reinterprets missing data as a form of causal inference by considering counterfactual variables.
result Identification assumptions in missing data can be encoded using graphical models of counterfactual and observed variables.
We present an automatic classification method for astronomical catalogs with missing data. We use Bayesian networks, a probabilistic graphical model, that allows us to perform inference to pre- dict missing values given observed data and dependency relationships between variables. To learn a Bayesian network from incom…
A novel sequence-to-sequence model predicts missing sensor data.
problem Missing sensor data in sequences.
method Formulated a novel sequence-to-sequence model using forward and backward RNNs.
result The model produces the lowest errors in 12% more cases than the current state-of-the-art.
On random surface areamath.PR
Consider a random smooth Gaussian field , where is a compact in . We derive a formula for average area of a surface generated by the equation and give some applications. As an auxiliary result we obtain an integral expression for area of a surface induced by zeros of a \e…
MIM adds indicator variables to improve model performance on incomplete data.
problem Missing data in incomplete data sets.
method Missing Indicator Method (MIM) and Selective MIM (SMIM).
result MIM improves model performance for informative missing values and high-dimensional data.
ELMV uses ensemble learning to handle missing values in EHR data.
problem Significant missing values in EHR data cause bias and unreliable conclusions.
method ELMV constructs multiple subsets with lower missing rates and uses a support set for ensemble learning.
result ELMV outperforms conventional methods in critical feature identification and outcome prediction.
Robust Variable Selection for High-dimensional Regression with Missing Data and Measurement Errorsstat.ME
Robust variable selection for high-dimensional data with missing and measurement errors.
problem Missing data and measurement errors confound data distribution.
method Exponential loss function with inverse probability weighting and additive error models.
result The Atan punishment method improves robust variable selection.
Two methods use BART to model missing data in leaf photosynthetic trait data.
problem Handling missing data in multivariate outcomes with non-ignorable mechanisms.
method Bayesian Additive Regression Trees (BART) for joint modeling of data and missingness indicators.
result Both methods effectively recover various missingness mechanisms and outperform existing approaches.
Stock markets are complex systems exhibiting collective phenomena and particular features such as synchronization, fluctuations distributed as power-laws, non-random structures and similarity to neural networks. Such specific properties suggest that markets operate at a very special point. Financial markets are believe…
Gaussian Processes improve missing value imputation in datasets.
problem Handling missing values in large datasets.
method Sparse Gaussian Processes combined with stochastic variational inference.
result MGP significantly outperforms other imputation methods.
Sparse regression such as the Lasso has achieved great success in handling high-dimensional data. However, one of the biggest practical problems is that high-dimensional data often contain large amounts of missing values. Convex Conditioned Lasso (CoCoLasso) has been proposed for dealing with high-dimensional data with…
A new method for imputing missing data using graphical models.
problem Missing data in graphs and its impact on analysis.
method MMG framework based on conditional independence and PAI principle.
result Valid and efficient method for imputing missing data.
Paper explores Bayes rule for Gaussian mixtures with missing data, outperforming supervised classifiers.
problem Improving classification accuracy in partially classified samples with missing data.
method Generative model framework with missing-data mechanism, Bayes rule allocation.
result Bayes rule classifier with missing-data mechanism outperforms fully supervised classifiers in various conditions.
Proposes methods to handle missing data in clustering models.
problem Missing data, especially MNAR, hinders model-based clustering.
method Developed a mixture model for different types of data, including MNAR, using Expectation Maximization algorithm.
result The proposed MNARz model simplifies inference and enables clustering with MNAR data.
DIVE learns video representations even with missing data.
problem Missing data in video sequences.
method Disentangled Imputed Video autoEncoder (DIVE) with missingness latent variable.
result DIVE outperforms state-of-the-art baselines in imputing and predicting missing video frames.
Deep learning for missing data with flexible modelling.
problem Missing data with missingness dependent on the missing values.
method Deep latent variable models with a neural network for missingness pattern modeling, importance-weighted variational inference, and stochastic gradients.
result Explicitly modelling missingness improves inference on various data sets.
Two new methods improve clustering with missing data.
problem Handling missing data in Gaussian Mixture Models.
method Proposes two methods using Monte Carlo Expectation-Maximization (MCEM) for data augmentation.
result Proposed methods outperform multiple imputation in clustering and density estimation.
BlockEcho method improves imputation of block-wise missing data.
problem Block-wise missing data reduces interpolation capability and predictive power.
method Integrates Matrix Factorization (MF) within Generative Adversarial Networks (GAN) to retain long-distance inter-element relationships.
result Superior performance on public datasets across three domains, especially at higher missing rates.
Framework improves PV forecasting by accounting for missing data uncertainty.
problem Uncertainty from missing data in PV power data.
method Combines stochastic multiple imputation with Rubin's rule.
result Improves prediction interval calibration without sacrificing point prediction accuracy.