DeepFS uses deep neural networks to select significant features in ultra high-dimensional data.
problem Challenges in traditional feature selection methods for high-dimensional, low-sample-size data.
method Two-step nonparametric approach combining deep neural networks and feature screening.
result DeepFS effectively identifies significant features with high precision for ultra high-dimensional data.
To model high dimensional data, Gaussian methods are widely used since they remain tractable and yield parsimonious models by imposing strong assumptions on the data. Vine copulas are more flexible by combining arbitrary marginal distributions and (conditional) bivariate copulas. Yet, this adaptability is accompanied b…
In data sets with many more features than observations, independent screening based on all univariate regression models leads to a computationally convenient variable selection method. Recent efforts have shown that in the case of generalized linear models, independent screening may suffice to capture all relevant feat…
A new method scales sparse machine learning to ultra-high dimensional problems.
problem Sparse and interpretable machine learning in ultra-high dimensional data.
method Two-phase approach: backbone set determination followed by reduced problem solving.
result The backbone set contains truly relevant features with high probability.
Machine learning methods are used to discover complex nonlinear relationships in biological and medical data. However, sophisticated learning models are computationally unfeasible for data with millions of features. Here we introduce the first feature selection method for nonlinear learning problems that can scale up t…
Study predicts price predictability in ultra-high frequency financial data using entropy tests.
problem Tackles predictability of ultra-high frequency financial data.
method Develops statistical tests based on Shannon entropy and Kullback-Leibler divergence to analyze predictability.
result Degree of randomness increases with aggregation level in transaction time.
A new feature selection method using random forest and Kolmogorov filter.
problem Ultra-high dimensional data feature selection.
method Fused Kolmogorov filter with random forest based recursive feature elimination.
result Selection and L2 consistency under weak conditions. A variable screening procedure via correlation learning was proposed Fan and Lv (2008) to reduce dimensionality in sparse ultra-high dimensional models. Even when the true model is linear, the marginal regression can be highly nonlinear. To address this issue, we further extend the correlation learning to marginal nonp…
FL-Sailer enables federated learning for scATAC-seq data, reducing dimensionality and noise.
problem Privacy-preserving federated learning for ultra-high dimensional, sparse, and heterogeneous scATAC-seq data.
method FL-Sailer integrates adaptive leverage score sampling and an invariant VAE architecture.
result FL-Sailer converges to an approximate solution with bounded error, surpassing centralized methods.
A streaming algorithm estimates quadratic covariation from financial data efficiently.
problem Estimating quadratic covariation from ultra-high-frequency financial data with limited memory.
method Formulated multi-scale, realized kernel, pre-averaging, and modulated realized covariance estimators with fixed bandwidth.
result Fixed bandwidth estimators require higher bandwidth for positive semidefiniteness.
This study examine the theoretical and empirical perspectives of the symmetric Hawkes model of the price tick structure. Combined with the maximum likelihood estimation, the model provides a proper method of volatility estimation specialized in ultra-high-frequency analysis. Empirical studies based on the model using t…
We propose a novel application of the Simultaneous Orthogonal Matching Pursuit (S-OMP) procedure for sparsistant variable selection in ultra-high dimensional multi-task regression problems. Screening of variables, as introduced in \cite{fan08sis}, is an efficient and highly scalable way to remove many irrelevant variab…
We propose an empirical Bayes estimator based on Dirichlet process mixture model for estimating the sparse normalized mean difference, which could be directly applied to the high dimensional linear classification. In theory, we build a bridge to connect the estimation error of the mean difference and the misclassificat…
Algorithms with fast convergence, small number of data access, and low per-iteration complexity are particularly favorable in the big data era, due to the demand for obtaining \emph{highly accurate solutions} to problems with \emph{a large number of samples} in \emph{ultra-high} dimensional space. Existing algorithms l…
We propose an algorithm, semismooth Newton coordinate descent (SNCD), for the elastic-net penalized Huber loss regression and quantile regression in high dimensional settings. Unlike existing coordinate descent type algorithms, the SNCD updates each regression coefficient and its corresponding subgradient simultaneousl…
The Dantzig selector has received popularity for many applications such as compressed sensing and sparse modeling, thanks to its computational efficiency as a linear programming problem and its nice sampling properties. Existing results show that it can recover sparse signals mimicking the accuracy of the ideal procedu…
Proposes a method to select features for deep learning in noisy, high-dimensional data.
problem Feature selection for deep learning in ultra-high dimensional and highly correlated data.
method Data-adaptive multi-resolutional screening and cleaning with deep learning.
result Achieves high power while keeping false discovery rate low.
Paper uses TCN with attention to predict UHF stock price changes.
problem Predicting discrete dynamic distribution of UHF stock price changes.
method Classified price changes, used TCN with attention mechanism.
result TCN and TCN (attention) models outperform GARCH and LSTM models.
Proposes FarmHazard model for hazard regression with correlated covariates.
problem Model selection challenges in high-dimensional data with correlated covariates.
method Factor-Augmented Regularized Model for Hazard Regression (FarmHazard) that learns latent factors and idiosyncratic components.
result Proves model selection and estimation consistency under mild conditions.
Variable selection is a challenging issue in statistical applications when the number of predictors p far exceeds the number of observations n. In this ultra-high dimensional setting, the sure independence screening (SIS) procedure was introduced to significantly reduce the dimensionality by preserving the true mod…
Model selection is crucial to high-dimensional learning and inference for contemporary big data applications in pinpointing the best set of covariates among a sequence of candidate interpretable models. Most existing work assumes implicitly that the models are correctly specified or have fixed dimensionality. Yet both …
Sparse alpha-norm regularization has many data-rich applications in Marketing and Economics. Alpha-norm, in contrast to lasso and ridge regularization, jumps to a sparse solution. This feature is attractive for ultra high-dimensional problems that occur in demand estimation and forecasting. The alpha-norm objective is …
In order to identify important variables that are involved in making optimal treatment decision, Lu et al. (2013) proposed a penalized least squared regression framework for a fixed number of predictors, which is robust against the misspecification of the conditional mean model. Two problems arise: (i) in a world of ex…
This paper proposes a model-free and data-adaptive feature screening method for ultra-high dimensional datasets. The proposed method is based on the projection correlation which measures the dependence between two random vectors. This projection correlation based method does not require specifying a regression model an…
Diagnosing basal cell carcinomas (BCC), one of the most common cutaneous malignancies in humans, is a task regularly performed by pathologists and dermato-pathologists. Improving histological diagnosis by providing diagnosis suggestions, i.e. computer-assisted diagnoses is actively researched to improve safety, quality…
This study examines how financial tick data becomes more random with time aggregation.
problem Investigating the randomness of financial tick data over time.
method Applied statistical randomness tests from NIST and TestU01 batteries to ultra-high frequency financial data.
result Financial tick data becomes increasingly random as the aggregation level of transaction time increases.
A method to identify important features without solving the full problem.
problem Identifying important features in high-dimensional data.
method Persistent reduction using extreme ray identification on a polyhedral cone.
result A subset of features can be guaranteed to have zero coefficients in all optimal solutions.
Efficiently solves Elastic Net in high dimensions with Newton method.
problem Feature selection in high-dimensional data with non-negligible collinearity.
method Semi-smooth Newton Augmented Lagrangian Method.
result Significantly reduces computational cost compared to competitors.
GIDS reduces high-dimensional response and predictor spaces, improving interpretability and computational efficiency.
problem Challenges in modeling interactions among high-dimensional multimodal data.
method Graph Independence Dual Screening (GIDS) framework that reduces both response and predictor dimensions.
result GIDS reduces feature space to 9,000 CpGs and 2,000 transcripts, revealing coordinated regulatory mechanisms.
Social and economic systems are complex adaptive systems, in which heterogenous agents interact and evolve in a self-organized manner, and macroscopic laws emerge from microscopic properties. To understand the behaviors of complex systems, computational experiments based on physical and mathematical models provide a us…
Two important goals of high-dimensional modeling are prediction and variable selection. In this article, we consider regularization with combined L1 and concave penalties, and study the sampling properties of the global optimum of the suggested method in ultra-high dimensional settings. The L1-penalty provides th…
We study the distributions of event-time returns and clock-time returns at different microscopic timescales using ultra-high-frequency data extracted from the limit-order books of 23 stocks traded in the Chinese stock market in 2003. We find that the returns at the one-trade timescale obey the inverse cubic law. For la…
A detailed analysis of correlation between stock returns at high frequency is compared with simple models of random walks. We focus in particular on the dependence of correlations on time scales - the so-called Epps effect. This provides a characterization of stochastic models of stock price returns which is appropriat…
This paper develops robust confidence intervals in high-dimensional and left-censored regression. Type-I censored regression models are extremely common in practice, where a competing event makes the variable of interest unobservable. However, techniques developed for entirely observed data do not directly apply to the…
AJL framework detects dynamic patterns in high-dimensional time-varying models.
problem Complex time-varying associations and abrupt regime shifts in longitudinal processes.
method Hierarchical regularization framework integrating functional variable selection with structural changepoint detection.
result The refined estimator achieves the oracle property in ultra-high-dimensional settings.
We present a large-scale study of commonality in liquidity and resilience across assets in an ultra high-frequency (millisecond-timestamped) Limit Order Book (LOB) dataset from a pan-European electronic equity trading facility. We first show that extant work in quantifying liquidity commonality through the degree of ex…
RaSE screens variables via random subspaces, identifying joint effects.
problem Missing joint effects of predictors in ultra-high dimensional data.
method Random Subspace Ensemble (RaSE) framework combining subspace evaluation criteria.
result RaSE identifies signals with no marginal effect or high-order interactions.
Hardware accelerations of deep learning systems have been extensively investigated in industry and academia. The aim of this paper is to achieve ultra-high energy efficiency and performance for hardware implementations of deep neural networks (DNNs). An algorithm-hardware co-optimization framework is developed, which i…
Structured weight pruning is a representative model compression technique of DNNs to reduce the storage and computation requirements and accelerate inference. An automatic hyperparameter determination process is necessary due to the large number of flexible hyperparameters. This work proposes AutoCompress, an automatic…
At the ultra high frequency level, the notion of price of an asset is very ambiguous. Indeed, many different prices can be defined (last traded price, best bid price, mid price,...). Thus, in practice, market participants face the problem of choosing a price when implementing their strategies. In this work, we propose …
Using ultra-high-frequency data extracted from the order flows of 23 stocks traded on the Shenzhen Stock Exchange, we study the empirical regularities of order placement in the opening call auction, cool period and continuous auction. The distributions of relative logarithmic prices against reference prices in the thre…
Paper forecasts financial trading durations using a new point process model.
problem Forecasting limit order book durations in high-frequency financial data.
method Self-exciting flexible residual point process incorporating empirical distributional features.
result The model achieves strong predictive performance compared to alternative approaches.
VOLARE provides standardized realized volatility measures from financial data.
problem Lack of standardized realized volatility measures from ultra-high-frequency data.
method Asset-specific pipeline for cleaning and sampling data, providing a wide range of realized estimators.
result Comprehensive set of realized estimators for equities, exchange rates, and futures.
Microstructures of a material form the bridge linking processing conditions - which can be controlled, to the material property - which is the primary interest in engineering applications. Thus a critical task in material design is establishing the processing-structure relationship, which requires domain expertise and …
In real world industrial applications of topic modeling, the ability to capture gigantic conceptual space by learning an ultra-high dimensional topical representation, i.e., the so-called "big model", is becoming the next desideratum after enthusiasms on "big data", especially for fine-grained downstream tasks such as …
We consider the problem of learning a high-dimensional multi-task regression model, under sparsity constraints induced by presence of grouping structures on the input covariates and on the output predictors. This problem is primarily motivated by expression quantitative trait locus (eQTL) mapping, of which the goal is …
Sparse support vector machine (SVM) is a popular classification technique that can simultaneously learn a small set of the most interpretable features and identify the support vectors. It has achieved great successes in many real-world applications. However, for large-scale problems involving a huge number of samples a…
When stock prices are observed at high frequencies, more information can be utilized in estimation of parameters of the price process. However, high-frequency data are contaminated by the market microstructure noise which causes significant bias in parameter estimation when not taken into account. We propose an estimat…