Causal discovery algorithms infer causal relations from data based on several assumptions, including notably the absence of measurement error. However, this assumption is most likely violated in practical applications, which may result in erroneous, irreproducible results. In this work we show how to obtain an upper bo…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “data-based discovery”
Bivariate Causal Discovery for Categorical Data via Classification with Optimal Label Permutationstat.ML
Proposes a new classifier for causal discovery in categorical data.
problem Causal discovery for categorical data.
method Classification with optimal label permutation (COLP) and simple learning algorithm.
result Favorable performance compared to state-of-the-art methods.
Paper discovers governing equations from data using differential invariants.
problem Discovering partial differential equations from data is challenging.
method The paper proposes a pipeline based on differential invariants to reduce the search space and adhere to symmetry.
result DI-SINDy method outperforms other symmetry-informed methods in PDE discovery.
Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systemsphysics.comp-ph
The paper tackles physical constraints in probabilistic machine learning for CG models of high-dimensional systems.
problem Introducing physical constraints in probabilistic machine learning objectives for coarse-graining dynamical systems.
method Formulating coarse-graining process using probabilistic state-space model and accounting for constraints as virtual observables.
result Probabilistic inference tools can identify coarse-grained variables without needing a fine-to-coarse projection or time-derivatives.
Missing data are ubiquitous in many domains including healthcare. When these data entries are not missing completely at random, the (conditional) independence relations in the observed data may be different from those in the complete data generated by the underlying causal process. Consequently, simply applying existin…
C-PP-COAD detects anomalies with limited real data, reducing dependency on real calibration data.
problem Limited real calibration data for online anomaly detection.
method Context-aware prediction-powered conformal online anomaly detection (C-PP-COAD).
result Significantly reduces dependency on real calibration data without compromising FDR control.
The lack of interpretability often makes black-box models difficult to be applied to many practical domains. For this reason, the current work, from the black-box model input port, proposes to incorporate data-based prior information into the black-box soft-margin SVM model to enhance its interpretability. The concept …
This paper addresses the model-free nonlinear optimal problem with generalized cost functional, and a data-based reinforcement learning technique is developed. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton-Jacobi-Bellman (HJB) equation, which is a nonlinear partial differ…
Efficiently samples multimodal distributions using data-based initialization.
problem Sampling multimodal distributions with limited samples.
method Data-based initialization for Markov chains with spectral gap.
result Efficiently generates samples close to stationary distribution.
Precision medicine aims for personalized prognosis and therapeutics by utilizing recent genome-scale high-throughput profiling techniques, including next-generation sequencing (NGS). However, translating NGS data faces several challenges. First, NGS count data are often overdispersed, requiring appropriate modeling. Se…
Online structure learning approaches, such as those stemming from Statistical Relational Learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel appro…
We propose a novel probabilistic dimensionality reduction framework that can naturally integrate the generative model and the locality information of data. Based on this framework, we present a new model, which is able to learn a smooth skeleton of embedding points in a low-dimensional space from high-dimensional noisy…
Conditional independence testing is a fundamental problem underlying causal discovery and a particularly challenging task in the presence of nonlinear and high-dimensional dependencies. Here a fully non-parametric test for continuous data based on conditional mutual information combined with a local permutation scheme …
Hierarchical-CPI improves variable importance measurement for medical data.
problem Limited interpretability of complex medical models.
method Hierarchical-CPI measures conditional variable importance with statistical control, handling correlated data.
result Hierarchical-CPI outperforms existing methods in medical datasets.
Recent advances in statistical theory, together with advances in the computational power of computers, provide alternative methods to do mass-univariate hypothesis testing in which a large number of univariate tests, can be properly used to compare MEEG data at a large number of time-frequency points and scalp location…
A new DDR framework learns low-dimensional data representations using dynamical systems.
problem Learning efficient low-dimensional data representations.
method DDR framework based on nonlinear dynamical systems, using linear combinations of functions and regularization.
result DDR method outperforms other methods on synthetic and real datasets.
Causal Graph Aided Causal Discovery in an Observational Aneurysmal Subarachnoid Hemorrhage Studystat.ME
Causal graph aids observational study insights in aSAH patients.
problem Lack of clear objectives and tools for identifying necessary adjustments in observational studies.
method Uses causal directed acyclic graphs (DAGs) to provide insights mid-study and identify necessary data enhancements.
result Midway insights and necessary data enhancements identified for meaningful causal questions.
Study develops a data-based model for in-cylinder pressure and cyclic variations in RCCI engines.
problem Lack of models capturing cyclic variations in combustion concepts like RCCI.
method Combines Principle Component Decomposition and Gaussian Process Regression.
result Model predicts combustion measures with high accuracy, especially peak-pressure rise-rate.
In an earlier work we identified the types and numbers of static equilibrium points of solids arising from fine, equidistant -discretrizations of smooth, convex surfaces. We showed that such discretizations carry equilibrium points on two scales: the local scale corresponds to the discretization, the global scale to…
Differentiable causal discovery methods perform robustly under model violations.
problem Causal discovery algorithms struggle with real-world data due to unverifiable causal assumptions.
method Benchmarked differentiable causal discovery methods under eight model assumption violations.
result Differentiable causal discovery methods exhibit robust performance under Structural Hamming Distance and Structural Intervention Distance metrics.
New framework uses background knowledge to speed up causal discovery.
problem Scalable causal discovery for large datasets.
method Utilizes background knowledge during causal discovery process.
result Background knowledge reduces computational requirements and improves structure quality.
New method prevents invalid inference after causal discovery.
problem Invalid inference after causal discovery.
method Developed tools for valid post-causal-discovery inference.
result Our method provides reliable coverage while achieving more accurate causal discovery.
Local Discovery by Partitioning: Polynomial-Time Causal Discovery Around Exposure-Outcome Pairsstat.ML
LDP speeds up causal discovery by partitioning, improving VAS recall and runtime.
problem Hard causal discovery in nonparametric settings with exponential complexity.
method Local Discovery by Partitioning (LDP) for causal inference around exposure-outcome pairs.
result LDP yields less biased and more precise estimates than baseline methods.
Physics-aware, deep probabilistic modeling of multiscale dynamics in the Small Data regimephysics.comp-ph
The paper develops a physics-aware method for modeling multiscale dynamics with reduced data.
problem Discovering effective, lower-dimensional models for high-dimensional dynamical systems.
method Probabilistic deep neural networks incorporating physical constraints.
result The method reduces the need for extensive multiscale simulations (Small Data regime).
The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
Proposes LLM-DCD for improved causal discovery from data.
problem Challenges in discovering causal relationships from observational data.
method Uses LLM to initialize DCD optimization, incorporating priors.
result Higher accuracy on benchmark datasets compared to state-of-the-art.
Probabilistic grammars improve equation discovery from data.
problem Discovering scientific laws from data using equations.
method Proposed probabilistic context-free grammars to encode soft constraints and a Monte-Carlo algorithm.
result Probabilistic grammars lead to more efficient equation discovery.
Causal discovery improves fMRI analysis, but faces challenges.
problem Challenges in applying causal discovery to fMRI data.
method Identifying and addressing nine challenges in fMRI causal discovery.
result Current methods for fMRI causal discovery need improvement.
New methods for Markov Blanket discovery using MML outperform existing approaches.
problem Causal discovery from large datasets.
method Developed three new methods of Markov Blanket discovery using Minimum Message Length.
result Our best MML method is consistently competitive and has advantageous features.
L2D-CD learns to defer expert recommendations in causal discovery.
problem Combining expert knowledge with data-driven results in causal discovery when expert recommendations may contradict data.
method Adapting learning-to-defer algorithms for pairwise causal discovery, L2D-CD learns a deferral function to select between expert recommendations and data-driven methods.
result L2D-CD outperforms both causal discovery methods and the expert used in isolation, identifying domains where the expert's performance is strong or weak.
Review of automation's role in chemical discovery, emphasizing future challenges.
problem Improving automation's contribution to chemical discovery.
method Analysis of exemplary studies and open research directions.
result Future autonomous systems need improvement in data handling, model building, and experiment automation.
New method controls false discoveries in financial asset pricing.
problem Controlling false discoveries in time series with unknown correlations.
method Double bootstrapping method to control false discovery rate.
result Superior statistical power and controlled false discovery rate.
Interpretable ML helps discover insights from big data.
problem Validating data-driven discoveries from complex datasets.
method Statistical and machine learning techniques for interpretable models.
result Challenges in validating data-driven discoveries remain.
We introduce interactive structure discovery, a generic framework that encompasses many interactive learning settings, including active learning, top-k item identification, interactive drug discovery, and others. We adapt a recently developed active learning algorithm of Tosh and Dasgupta (2017) for interactive structu…
Review of automation's role in chemical discoveries.
problem Improving autonomous discovery in chemistry.
method Classification of discovery types, assessment of autonomy, case studies.
result Rapid advancements in automation and machine learning are transforming experimentation and modeling.
KEEL improves causal discovery with fuzzy knowledge and complex data.
problem Challenges in causal discovery due to prior knowledge, domain inconsistencies, and small sample sizes.
method Weakly-supervised fuzzy knowledge and data co-driven causal discovery method (KEEL).
result KEEL outperforms state-of-the-art methods in accuracy, robustness, and computational efficiency.
Detects project management anti-patterns using code and issue data.
problem Detecting project management anti-patterns requires expert judgment and is expensive.
method Convert descriptions to detectable metrics, quantify deviations, and optimize patterns.
result Automatic calibration enhances pattern detection and severity assessment.
Cluster-DAGs improve causal discovery with prior knowledge.
problem Finding cause-effect relationships from high-dimensional data.
method Cluster-DAGs as prior knowledge framework, modified constraint-based algorithms Cluster-PC and Cluster-FCI.
result Cluster-PC and Cluster-FCI outperform baselines without prior knowledge.
Despite its popularity, it is widely recognized that the investigation of some theoretical aspects of clustering has been relatively sparse. One of the main reasons for this lack of theoretical results is surely the fact that, whereas for other statistical problems the theoretical population goal is clearly defined (as…
The paper examines how timing of observations affects causal discovery methods.
problem The sensitivity of causal discovery methods to mismatched observation timing.
method Empirical and theoretical analysis of classical and recent causal discovery methods.
result Causal discovery methods are sensitive to sampling rate and window length.
New algorithm improves materials discovery using max K-Armed Bandit.
problem Maximizing material breakthroughs in materials discovery.
method Proposed a search algorithm based on max K-Armed Bandit (MKB) for materials discovery.
result Stable performance in late search stages, outperforming other bandit algorithms.
This work integrates domain knowledge into A*-based causal discovery methods.
problem Efficiently incorporating domain knowledge into A*-based causal discovery methods.
method Integrates various types of domain knowledge into A*-based causal discovery methods, reducing the graph search space and improving computational gains.
result Small amounts of domain knowledge can dramatically speed up A*-based causal discovery and improve its performance and practicality.
New methods control false discoveries near the boundary in conformal novelty detection.
problem Over-optimistic assessments near the rejection threshold in conformal novelty detection.
method Support line (SL) correction and alternative procedures to control boundary false discovery rate (bFDR).
result New procedures control the boundary false discovery rate (bFDR) in the conformal setting.
Paper presents a new dataset for testing causal discovery methods in industrial systems.
problem Lack of real-world datasets for evaluating causal discovery methods on time series data.
method Develops a dataset from an industrial system and its known causal graph.
result Provides a benchmark for evaluating causal discovery methods in complex systems.
Python library for causal discovery from observational data.
problem Revealing causal relations from observational data.
method Comprehensive collection of causal discovery methods in Python.
result Ease of use for non-specialists and modular building blocks for developers.
The paper establishes bounds for score-matching in causal discovery and generative modeling.
problem Estimating causal relationships from data.
method Training a deep neural network to estimate the score function and applying it to causal discovery.
result Bounds on the error rate of causal discovery methods using score-matching.
The aim of process discovery, originating from the area of process mining, is to discover a process model based on business process execution data. A majority of process discovery techniques relies on an event log as an input. An event log is a static source of historical data capturing the execution of a business proc…
Visualizes deep generative models for drug design.
problem Limited visualization tools for deep generative models in drug discovery.
method Proposes a visualization framework for deep graph generative models.
result Interactive visualization and molecular optimization tools.