Estimates itemset significance using Maximum Entropy and Kullback-Leibler divergence.
problem Defining the significance of itemsets in data mining.
method Maximum Entropy estimation and Kullback-Leibler divergence for deviation measurement.
result Significance measure goes to zero for derivable itemsets and ranks as a statistical test.
The problem of finding itemsets that are statistically significantly enriched in a class of transactions is complicated by the need to correct for multiple hypothesis testing. Pruning untestable hypotheses was recently proposed as a strategy for this task of significant itemset mining. It was shown to lead to greater s…
Algorithm extracts non-monotonic rules from statistical models using HUIM.
problem Extracting non-monotonic rules from statistical learning models.
method Reduces problem to HUIM, uses TreeExplainer for feature importance.
result Significant improvement in classification metrics and training time.
A new method for selecting high quality itemsets from a large collection.
problem Selecting a small, high-quality subset of patterns from a larger collection of itemsets.
method Using decomposable families of itemsets and a junction tree structure.
result The method provides high quality results and is computationally efficient.
The problem of multiple hypothesis testing arises when there are more than one hypothesis to be tested simultaneously for statistical significance. This is a very common situation in many data mining applications. For instance, assessing simultaneously the significance of all frequent itemsets of a single dataset entai…
We present a novel algorithm, Westfall-Young light, for detecting patterns, such as itemsets and subgraphs, which are statistically significantly enriched in one of two classes. Our method corrects rigorously for multiple hypothesis testing and correlations between patterns through the Westfall-Young permutation proced…
CICLAD efficiently mines frequent closed itemsets from data streams with minimal memory usage.
problem Mining frequent closed itemsets from data streams is resource-intensive.
method CICLAD is an intersection-based sliding-window FCI miner that optimizes memory usage while maintaining performance.
result CICLAD achieves significantly lower memory footprint compared to existing methods.
Mining itemsets that are the most interesting under a statistical model of the underlying data is a commonly used and well-studied technique for exploratory data analysis, with the most recent interestingness models exhibiting state of the art performance. Continuing this highly promising line of work, we propose the f…
Significant pattern mining, the problem of finding itemsets that are significantly enriched in one class of objects, is statistically challenging, as the large space of candidate patterns leads to an enormous multiple testing problem. Recently, the concept of testability was proposed as one approach to correct for mult…
Frequent Itemsets (FIs) mining is a fundamental primitive in data mining. It requires to identify all itemsets appearing in at least a fraction θ of a transactional dataset D. Often though, the ultimate goal of mining D is not an analysis of the dataset \emph{per se}, but the understanding of …
The paper improves itemset quality assessment by incorporating background knowledge.
problem Assessing the quality of discovered itemsets is challenging due to many patterns being explainable by background knowledge.
method The authors introduce a maximum entropy approach to efficiently infuse additional background knowledge such as row margins, lazarus counts, and bounds of ones.
result More sophisticated models that incorporate background knowledge fit the data better and improve frequency prediction of itemsets.
Enhances distributed Apriori-like frequent itemsets mining performance.
problem Improving performance of distributed Apriori-like frequent itemsets mining.
method Presented a new distributed approach considering Apriori algorithm's characteristics and distribution aspects.
result The proposed approach significantly enhances performance and achieves good scalability compared to a typical distributed Apriori algorithm.
Proposes SCR-Apriori for efficient mining of SCR-patterns.
problem Mining high-quality `Set of Contrasting Rules'-pattern (SCR-pattern) efficiently.
method Integrates SCR-pattern structure into Apriori algorithm to prune search space.
result Significantly reduces computational cost compared to state-of-the-art.
This paper introduces the combinatorial Boolean model (CBM), which is defined as the class of linear combinations of conjunctions of Boolean attributes. This paper addresses the issue of learning CBM from labeled data. CBM is of high knowledge interpretability but naïve learning of it requires exponentially large compu…
In binary-transaction data-mining, traditional frequent itemset mining often produces results which are not straightforward to interpret. To overcome this problem, probability models are often used to produce more compact and conclusive results, albeit with some loss of accuracy. Bayesian statistics have been widely us…
FSR efficiently discovers significant patterns with few resampled datasets.
problem Mining significant patterns in transactional data, especially subgroups.
method FSR uses resampling to bound the supremum deviation of quality statistics, providing rigorous guarantees on false discoveries.
result FSR effectively discovers significant subgroups with a small number of resampled datasets.
Method detects interactions for better CTR prediction.
problem Predicting click-through rate with high-dimensional categorical features and time-varying interactions.
method Online Random Intersection Chains (ORIC) for detecting informative interactions.
result ORIC detects high-interpretability interactions that improve CTR prediction.
Pattern sampling has been proposed as a potential solution to the infamous pattern explosion. Instead of enumerating all patterns that satisfy the constraints, individual patterns are sampled proportional to a given quality measure. Several sampling algorithms have been proposed, but each of them has its limitations wh…
This paper creates a tagging system for paintings using historical descriptions.
problem Tagging accuracy for paintings over time with varying expert descriptions.
method A neural network with frequent itemsets as tags, followed by clustering.
result Improved tagging accuracy for paintings over time.
OMBA learns product and user representations for better online market basket analysis.
problem Limited ability to uncover rarely occurring and temporal associations in MBA.
method Jointly learns product and user representations, captures temporal dynamics, scalable online method.
result OMBA outperforms state-of-the-art methods by 21% on real-world datasets.
Proposes a new method for finding frequent closed patterns in transaction bases.
problem Frequent closed patterns in transaction bases.
method Partitioning the search space into subcontexts and updating frequent closed patterns with their minimal generators.
result Proposed approach called UFCIGs-DAC for efficient search of frequent closed itemsets.
IGML learns discriminative metrics for graph classification.
problem Defining an appropriate distance metric for graph data.
method Supervised distance metric learning in a subgraph-based feature space with sparsity-inducing penalty.
result IGML identifies important subgraphs for graph classification.
Scales gradual pattern discovery from imprecise data.
problem Managing large collections of gradual patterns from imprecise data.
method Introduces gradualness thresholds and considers attribute value distribution.
result Eliminates noise patterns, showing a smaller set of meaningful patterns.
nFBST tests neural networks using Bayesian methods.
problem Traditional significance testing struggles with complex nonlinear relationships.
method nFBST uses Bayesian neural networks to test neural networks.
result nFBST can test global, local, and instance-wise significance.
Paper presents a machine learning method to improve significance tests for misspecified linear models.
problem Misspecification of linear assumptions in social science models leads to inaccurate significance levels.
method Apply machine learning to fit ground truth function, calculate linear approximation, and adjust the estimator.
result The method significantly outperforms linear regression for non-linear ground truth functions.
Develops a fast test to identify significant features in machine learning models.
problem Identifying significant features and interactions in machine learning models efficiently.
method Forward-selection approach for any model, learning task, and variable type, non-asymptotic, straightforward implementation.
result Identifies statistically significant features and feature interactions of any order.
Investigates the number of experiments needed for statistical significance in medication testing.
problem Determining the number of experiments needed for a statistically significant result.
method Examines binomial and general probability distributions, considering placebo efficacy and varying distributions.
result The number of experiments needed can be significantly higher when placebo efficacy is considered.
The paper proposes a method to find subgroups with significant treatment effects in noisy data.
problem Estimating the causal effects of interventions on noisy outcomes.
method A machine-learning method specifically optimized for finding subgroups with significant effects, designed to maximize the probability of obtaining a statistically significant positive treatment effect.
result The proposed method yields higher power in detecting subgroups affected by the treatment compared to standard tree-based tools.
New method tracks significant shifts in nonparametric bandits.
problem Tracking significant changes in nonparametric contextual bandits.
method Proposed a notion of 'experienced significant shifts' to adapt to minimax rate without knowledge of change parameters.
result Experienced significant shifts count fewer changes than traditional metrics, leading to an adaptive algorithm.
The paper compares two spectrum definitions and finds stability in one modification.
problem Generalizing eigenvalues to arbitrary functionals with stability.
method Comparison of Gromov's homotopy significant spectrum and Krasnoskii spectrum, with a modified definition of the homotopy significant spectrum.
result The modified homotopy significant spectrum is stable, and Cheeger constant corresponds to Krasnoskii eigenvalue.
We introduce a minorization-maximization approach to optimizing common measures of discovery significance in high energy physics. The approach alternates between solving a weighted binary classification problem and updating class weights in a simple, closed-form manner. Moreover, an argument based on convex duality sho…
New method quantifies deep kNN anomaly detection significance.
problem Lack of uncertainty quantification in deep kNN AD.
method Selective Inference for anomaly scoring.
result Validates AD reliability with controlled false positives.
si4onnx enables selective inference on deep learning models.
problem Establishing the reliability of AI systems through statistical significance of identified regions.
method Selective inference techniques implemented through a Python package.
result Controlled type I error rates for hypothesis testing on deep learning models.
Develops a test to assess feature significance in neural networks.
problem Assessing the statistical significance of feature variables in neural networks.
method Gradient-based test statistic, asymptotic analysis using nonparametric techniques.
result Tests enable ranking variables by their influence on neural network predictions.
Random forest identifies key features for diagnosing machine faults.
problem Identifying specific factors causing machine faults.
method Gaussian mixture model clustering, spectrum analysis, random forest classification.
result Identified significant features for different machine states.
The paper integrates statistical significance and discriminative power in pattern discovery.
problem Discovering actionable patterns that meet rigorous statistical significance and discriminative power criteria.
method Integrates statistical significance and discriminative power criteria into state-of-the-art algorithms.
result Improves discriminative power and statistical significance of discovered patterns without quality deterioration.
We assume data sampled from a mixture of d-dimensional linear subspaces with spherically symmetric distributions within each subspace and an additional outlier component with spherically symmetric distribution within the ambient space (for simplicity we may assume that all distributions are uniform on their correspondi…
This paper predicts significant stock price changes using neural networks.
problem Predicting significant stock price changes.
method Three neural network models (MLP, CNN, LSTM) and two benchmark models (Random Forest, Relative Strength Index) were tested on 10-year daily stock price data of four major US companies.
result Neural network models significantly outperform traditional methods in predicting significant stock price changes.
We discuss a common suspicion about reported financial data, in 10 industrial sectors of the 6 so called "main developing countries" over the time interval [2000-2014]. These data are examined through Benford's law first significant digit and through distribution distances tests. It is shown that several visually anoma…
Analyzing large-scale, multi-experiment studies requires scientists to test each experimental outcome for statistical significance and then assess the results as a whole. We present Black Box FDR (BB-FDR), an empirical-Bayes method for analyzing multi-experiment studies when many covariates are gathered per experiment.…
Conditional independence tests (CI tests) have received special attention lately in Machine Learning and Computational Intelligence related literature as an important indicator of the relationship among the variables used by their models. In the field of Probabilistic Graphical Models (PGM)--which includes Bayesian Net…
Paper uses machine learning to forecast significant currency exchange rate fluctuations.
problem Forecasting significant daily returns in foreign exchange markets.
method Applying nine modern machine learning algorithms to data on four major currency pairs over 10 years, focusing on outlier detection methods.
result Outlier detection methods significantly outperform traditional techniques, with PKDE method producing the best results.
Paper proposes a machine learning method to predict sale efficacy.
problem Determining the efficacy of online sales from discounts alone.
method Machine learning-based heuristic using Support Vector Machine.
result Predicts sale efficacy with 91.11% accuracy.
Using methods introduced by Scargle in 1978 we derive a cumulative version of the Lomb periodogram that exhibits frequency independent statistics when applied to cumulative noise. We show how this cumulative Lomb periodogram allows us to estimate the significance of log-periodic signatures in the S&P 500 anti-bubble th…
Deep learning model predicts wind-wave relationship.
problem Characterize ocean wave climate for engineering applications.
method Two-stage deep learning model: CNN for spatial features, LSTM for temporal dependencies.
result Predicts spatio-temporal relationship between wind and significant wave height.
We investigated financial market data to determine which factors affect information flow between stocks. Two factors, the time dependency and the degree of efficiency, were considered in the analysis of Korean, the Japanese, the Taiwanese, the Canadian, and US market data. We found that the frequency of the significant…
Develops a new method for neural network significance testing without strict constraints.
problem Testing neural networks without bounded weights or specific architectural constraints.
method Uses Rademacher complexity bounds, weakened Sobolev space membership conditions, and a modified sieve space construction.
result Achieves optimal convergence rates and valid asymptotic distributions for test statistics.
Study finds significant premium for low-beta stocks in firm-level idiosyncratic return distributions.
problem Understanding the role of common idiosyncratic quantile factors in asset pricing.
method Quantile factor analysis to extract common idiosyncratic quantile factors with asymmetric pricing effects.
result Significant premium for innovations to the lower-tail factor: high-beta stocks outperform low-beta stocks by around 7-8% per year.