arXiv research

A new screening method for high-dimensional data reduces computational cost.

Although much progress has been made in classification with high-dimensional features \citep{Fan_Fan:2008, JGuo:2010, CaiSun:2014, PRXu:2014}, classification with ultrahigh-dimensional features, wherein the features much outnumber the sample size, defies most existing work. This paper introduces a novel and computation…

Learning expressive low-dimensional representations of ultrahigh-dimensional data, e.g., data with thousands/millions of features, has been a major way to enable learning methods to address the curse of dimensionality. However, existing unsupervised representation learning methods mainly focus on preserving the data re…

Statistical inference can be computationally prohibitive in ultrahigh-dimensional linear models. Correlation-based variable screening, in which one leverages marginal correlations for removal of irrelevant variables from the model prior to statistical inference, can be used to overcome this challenge. Prior works on co…

We propose a flexible nonparametric regression method for ultrahigh-dimensional data. As a first step, we propose a fast screening method based on the favored smoothing bandwidth of the marginal local constant regression. Then, an iterative procedure is developed to recover both the important covariates and the regress…

Solar improves variable selection in high-dimensional data with complicated dependence structures.

Modern bio-technologies have produced a vast amount of high-throughput data with the number of predictors far greater than the sample size. In order to identify more novel biomarkers and understand biological mechanisms, it is vital to detect signals weakly associated with outcomes among ultrahigh-dimensional predictor…

Proposes a method to select features for deep learning in noisy, high-dimensional data.

Estimates complex dependency structures in multi-omics data.

Develops high-dimensional measurement error models for non-linear loss functions.

The lasso model has been widely used for model selection in data mining, machine learning, and high-dimensional statistical analysis. However, with the ultrahigh-dimensional, large-scale data sets now collected in many real-world applications, it is important to develop algorithms to solve the lasso that efficiently sc…

Penalized regression models such as the lasso have been extensively applied to analyzing high-dimensional data sets. However, due to memory limitations, existing R packages like glmnet and ncvreg are not capable of fitting lasso-type models for ultrahigh-dimensional, multi-gigabyte data sets that are increasingly seen …

Variable selection in high-dimensional space characterizes many contemporary problems in scientific discovery and decision making. Many frequently-used techniques are based on independence screening; examples include correlation ranking (Fan and Lv, 2008) or feature selection using a two-sample t-test in high-dimension…

A new graphical model for discrete data without parametric restrictions.

The paper uses graph learning to detect valid instruments in high-dimensional data for house pricing.

Proposes a two-stage method for selecting correlated predictors in high-dimensional data.

Sparse symmetric tensor regression reduces brain connectivity complexity.

MDS selects assets by combining daily returns and intraday risk curves, improving portfolio performance.

Ultrahigh-dimensional variable selection plays an increasingly important role in contemporary scientific discoveries and statistical research. Among others, Fan and Lv [J. R. Stat. Soc. Ser. B Stat. Methodol. 70 (2008) 849-911] propose an independent screening framework by ranking the marginal correlations. They showed…

A new method reduces feature screening cost from $O(np)$ to $O(\sqrt{n}p)$.

Intertrade duration of equities is an important financial measure characterizing the trading activities, which is defined as the waiting time between successive trades of an equity. Using the ultrahigh-frequency data of a liquid Chinese stock and its associated warrant, we perform a comparative investigation of the sta…

Price gap, defined as the logarithmic price difference between the first two occupied price levels on the same side of a limit order book (LOB), is a key determinant of market depth, which is one of the dimensions of liquidity. However, the properties of price gaps have not been thoroughly studied due to the less avail…

Computed tomography (CT) is widely used in screening, diagnosis, and image-guided therapy for both clinical and research purposes. Since CT involves ionizing radiation, an overarching thrust of related technical research is development of novel methods enabling ultrahigh quality imaging with fine structural details whi…

This work refines Cover's theory for binary classification on low-dimensional data.

New DR method uses Gromov-Wasserstein distance for high-dimensional data.

PCENet reduces uncertainty in high-dimensional data efficiently.

New method visualizes noisy data better than existing techniques.

GTSNE improves data visualization for high-dimensional data.

High-dimensional big data appears in many research fields such as image recognition, biology and collaborative filtering. Often, the exploration of such data by classic algorithms is encountered with difficulties due to `curse of dimensionality' phenomenon. Therefore, dimensionality reduction methods are applied to the…

Geometric framework detects outliers in high-dimensional data.

Anomaly detection for high-dimensional data using large deviations principle.

Generative models learn complex data from low-dimensional manifolds.

Method determines latent dimensionality in international trade flows.

GDMaps reduces high-dimensional data to lower dimensions for better classification.

The paper finds non-Gaussian directions in high-dimensional data using Wasserstein distance.

New method interpolates high-dimensional scattered data using kernel theory.

A new method reduces high-dimensional data's impact on CWMs using TSNE.

Modern techniques simplify complex high-dimensional data.

Subspace clustering refers to the problem of clustering unlabeled high-dimensional data points into a union of low-dimensional linear subspaces, assumed unknown. In practice one may have access to dimensionality-reduced observations of the data only, resulting, e.g., from "undersampling" due to complexity and speed con…

New method estimates intrinsic dimensionality using angles, not distances.

This paper improves diffusion models for low-dimensional data.

Proposes generating virtual data points to overcome the curse of dimensionality.

Dimensionality reduction methods are very common in the field of high dimensional data analysis. Typically, algorithms for dimensionality reduction are computationally expensive. Therefore, their applications for the analysis of massive amounts of data are impractical. For example, repeated computations due to accumula…

HD-BWDM improves clustering validation in high-dimensional data.

Real world data often exhibit low-dimensional geometric structures, and can be viewed as samples near a low-dimensional manifold. This paper studies nonparametric regression of Hölder functions on low-dimensional manifolds using deep ReLU networks. Suppose $n$ training data are sampled from a Hölder function in $\mathc…

Review of privacy-preserving linear models for high-dimensional data.

Proposes a deep neural network for multi-dimensional functional data classification.

This study evaluates clustering algorithms on high-dimensional data.