Data-driven models analyze power grids under incomplete physical information, and their accuracy has been mostly validated empirically using certain training and testing datasets. This paper explores error bounds for data-driven models under all possible training and testing scenarios, and proposes an evaluation implem…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “data-driven training”
Paper proposes MA-BERT for efficient data-driven ATM models.
problem Long training time and need for large datasets in data-driven ATM models.
method Multi-Agent Bidirectional Encoder Representations from Transformers (MA-BERT) and transfer learning framework.
result MA-BERT saves training time and achieves high performance with little data.
Framework improves data-driven ROMs for complex systems using Bayesian operator inference.
problem Improving the quality of data-driven reduced-order models for complex dynamical systems.
method Develops an active learning framework using Bayesian operator inference to identify and select training parameters.
result The proposed adaptive sampling strategy consistently yields more stable and accurate ROMs than random sampling.
Study compares data-driven vs model-based MRS quantification strategies, focusing on resilience to out-of-distribution effects.
problem Resilience to out-of-distribution effects in data-driven MRS quantification.
method Compared three data-driven strategies (supervised regression, self-supervised learning, test-time adaptation) against model-based fitting tools.
result Test-time adaptation proved most resilient to out-of-distribution effects, while self-supervised learning achieved intermediate performance.
The paper discusses scalable learning for wireless data-driven systems.
problem Expanding data volume and model complexity limit centralized learning solutions.
method Discusses scalable architecture and local learning strategies.
result Promising research directions in scalable data-driven wireless communications.
Bayesian Imaging With Data-Driven Priors Encoded by Neural Networks: Theory, Methods, and Algorithmsstat.ME
Bayesian imaging uses neural networks to learn prior knowledge from data.
problem Performing Bayesian inference in imaging problems with limited prior knowledge.
method Constructs a data-driven prior on a sub-manifold of the image space using neural networks, and performs Bayesian computation on this manifold.
result Established the existence and well-posedness of the posterior distribution and moments, and demonstrated superior performance compared to existing methods.
Proposes data-driven methods for estimating conditional expectations.
problem Estimating conditional expectations when underlying density is unknown.
method Data-driven techniques to directly estimate conditional expectations from training data.
result Extends data-driven method to solve nonlinear equations in stochastic optimization.
Quantifying model form uncertainty in Reynolds-averaged turbulence models with Bayesian deep neural networksphysics.flu-dyn
Data-driven methods for improving turbulence modeling in Reynolds-Averaged Navier-Stokes (RANS) simulations have gained significant interest in the computational fluid dynamics community. Modern machine learning algorithms have opened up a new area of black-box turbulence models allowing for the tuning of RANS simulati…
With the advent of modern data collection and storage technologies, data-driven approaches have been developed for discovering the governing partial differential equations (PDE) of physical problems. However, in the extant works the model parameters in the equations are either assumed to be known or have a linear depen…
Physics-consistent method improves seismic inversion accuracy.
problem Challenges in seismic full-waveform inversion (FWI) due to ill-posedness and high cost.
method Hybrid approach combining physics-based models with data-driven methodologies, incorporating physics into data augmentation.
result Physics-consistent data-driven inversion yields higher accuracy and better generalization.
Most structure inference methods either rely on exhaustive search or are purely data-driven. Exhaustive search robustly infers the structure of arbitrarily complex data, but it is slow. Data-driven methods allow efficient inference, but do not generalize when test data have more complex structures than training data. I…
Most of Markov Chain Monte Carlo (MCMC) and sequential Monte Carlo (SMC) algorithms in existing probabilistic programming systems suboptimally use only model priors as proposal distributions. In this work, we describe an approach for training a discriminative model, namely a neural network, in order to approximate the …
New measures detect HFT activity, revealing its impact on stock prices.
problem Lack of public data on HFT activity.
method Developed machine learning models to predict HFT activity using proprietary and public data.
result Measures outperform conventional proxies and reveal HFT's impact on price discovery.
Develops a data-driven fault diagnosis framework for time-series data.
problem Fault diagnosis of dynamic systems using imbalanced and unknown fault classes.
method Kullback-Leibler divergence, data-driven fault classification, open-set classification.
result Framework handles imbalanced datasets, class overlapping, and unknown faults.
Data-driven factor graphs improve BCJR detection robustness.
problem Implementing BCJR detection with accurate channel model knowledge.
method Learn factor graph using machine learning from labeled data.
result BCJRNet learns to implement BCJR detection from small training sets.
New methods quantify uncertainties in AI weather forecasts.
problem Uncertainty in AI weather predictions.
method Comparing ensemble and post-hoc uncertainty quantification methods.
result Probabilistic forecasts improve over ensemble physics-based models.
InVAErt networks use data-driven methods for system synthesis and identifiability analysis.
problem Model synthesis and identifiability analysis for complex systems.
method Deterministic encoder and decoder, normalizing flow, variational encoder, loss function penalty coefficients, latent space sampling.
result Validation through various system types, demonstrating effectiveness of the framework.
Enhances Bayesian model selection for high-dimensional problems.
problem Bayesian model selection for high-dimensional problems.
method Proximal nested sampling with data-driven priors.
result Improves model selection for log-convex likelihood models.
In this work, we provide an efficient and realistic data-driven approach to simulate astronomical images using deep generative models from machine learning. Our solution is based on a variant of the generative adversarial network (GAN) with progressive training methodology and Wasserstein cost function. The proposed so…
The paper proposes machine learning models for option pricing without using historical or implied volatility.
problem Capturing option pricing without traditional volatility inputs.
method Three supervised machine learning approaches using data from multiple assets.
result Trained models outperform or match Black-Scholes formula for option pricing.
HAL accelerates the generation of training sets for accurate interatomic potentials.
problem Generating accurate and transferable interatomic potentials is time-consuming and requires expert input.
method HAL framework using a physically motivated sampler with a biasing term to drive high uncertainty configurations.
result HAL-generated training databases for alloys and polymers predict macroscopic properties with high accuracy.
WeldNet reduces complex dynamics to simpler, manageable segments.
problem Complex, high-dimensional time-dependent datasets from physical processes are costly to simulate.
method Windowed Encoders for Learning Dynamics, splitting time domain into windows for nonlinear dimension reduction and propagator training.
result WeldNet captures nonlinear latent structures and dynamics, outperforming existing methods.
Proposes a physics-informed VAE for disentangling physics from confounding influences.
problem Challenges in inferring and predicting physical systems under partial knowledge.
method Physics-informed variational autoencoder with adversarial training.
result Model successfully disentangles known physics from confounding influences.
We present a regression technique for data-driven problems based on polynomial chaos expansion (PCE). PCE is a popular technique in the field of uncertainty quantification (UQ), where it is typically used to replace a runnable but expensive computational model subject to random inputs with an inexpensive-to-evaluate po…
Pipeline learns topological features for protein stability prediction.
problem Predicting protein stability using topological features.
method Data-driven method to learn topological features, comparing with expert features.
result Topological features achieve 92%-99% of SME-based models' performance.
Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methodsphysics.ao-ph
Kernel methods improve geophysical forecasting accuracy and efficiency.
problem Improving geophysical forecasting models for accuracy and efficiency.
method Data-driven modeling of geophysical processes using kernel flows for faster, more accurate predictions.
result Kernel methods outperform neural networks and classical models in geophysical forecasting.
Proposes DMOC for more nuanced neural network robustness.
problem Lipschitz continuity is too coarse for nuanced data-dependent behavior.
method Data-driven, architecture-agnostic framework based on DMOC.
result DMOC provides a finer notion of robustness relative to data distribution.
MAD framework learns operators from physics-embedded data efficiently.
problem Data-driven methods require costly labeled datasets and model-driven techniques face efficiency-accuracy trade-offs.
method Integrates physical laws with data-driven learning to generate physics-embedded analytical solutions and synthetic data.
result Eliminates dependence on experimental or simulated training data, enabling efficient operator learning across multi-parameter systems.
Probabilistic atlases provide essential spatial contextual information for image interpretation, Bayesian modeling, and algorithmic processing. Such atlases are typically constructed by grouping subjects with similar demographic information. Importantly, use of the same scanner minimizes inter-group variability. Howeve…
Data-driven topology design using a deep generative modelphysics.comp-ph
Paper proposes a new method for designing materials using deep learning.
problem Designing high-performance material distributions from given distributions.
method Iterative process of selecting, generating, and merging material distributions using a deep generative model.
result The method improves material performance through iterative refinement.
Improved MCMC sampling for expensive, irregular likelihoods.
problem Bayesian inference challenges with irregular, expensive likelihoods.
method Adapt subset samplers, introduce data-driven proxies, adaptive controller.
result Improved HINTS algorithm achieves best sampling error in fixed budget.
This paper addresses the data-driven identification of latent dynamical representations of partially-observed systems, i.e., dynamical systems for which some components are never observed, with an emphasis on forecasting applications, including long-term asymptotic patterns. Whereas state-of-the-art data-driven approac…
In recent years, data-driven methods have been developed to learn dynamical systems and partial differential equations (PDE). The goal of such work is discovering unknown physics and the corresponding equations. However, prior to achieving this goal, major challenges remain to be resolved, including learning PDE under …
We introduce physics informed neural networks -- neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear partial differential equations. In this two part treatise, we present our developments in the context of solving two main classes …
While modern day web applications aim to create impact at the civilization level, they have become vulnerable to adversarial activity, where the next cyber-attack can take any shape and can originate from anywhere. The increasing scale and sophistication of attacks, has prompted the need for a data driven solution, wit…
OceanForecastBench offers a comprehensive benchmark for data-driven ocean forecasting models.
problem Lack of open-source, standardized benchmarks for data-driven ocean forecasting models.
method Proposes OceanForecastBench, a benchmark with high-quality data and evaluation pipeline.
result Offers the most comprehensive benchmarking framework for data-driven ocean forecasting.
This paper emphasizes the need for uncertainty quantification in data-driven ML models for nuclear engineering.
problem Uncertainty in ML predictions due to data noise, model architecture, and stochastic training.
method Explains and compares uncertainties in physics-based and data-driven models, and presents techniques to quantify ML prediction uncertainties.
result The importance of uncertainty quantification in ML models for nuclear engineering applications.
The study provides theoretical foundations for using smaller instances to predict algorithm performance on larger ones.
problem Scalability challenge in evaluating algorithms on large instances.
method Formalized size generalization, providing theoretical guarantees for predicting algorithm performance on large instances using smaller, representative instances.
result Characterized the subsample size sufficient to ensure performance on the subsample reflects performance on the full instance.
DD-SP uses ML to improve SP for Lorenz 96 systems, outperforming LR and DD-P.
problem Improving computational efficiency in weather/climate modeling.
method Data-driven super-parameterization using recurrent neural networks.
result DD-SP is more accurate and cheaper than SP, especially with scale separation.
A new method improves the interpretability of data-driven models in ironmaking processes.
problem Lack of transparency in machine learning models used in industrial processes.
method Combines Variational Autoencoder (VAE) with Local Interpretable Model-agnostic Explanations (LIME) for better model interpretability.
result Improved local fidelity of local interpretable linear models compared to LIME.
Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningstat.ML
Many decision problems in science, engineering and economics are affected by uncertain parameters whose distribution is only indirectly observable through samples. The goal of data-driven decision-making is to learn a decision from finitely many training samples that will perform well on unseen test samples. This learn…
Bayesian nonparametrics improves data-driven risk optimization under distributional uncertainty.
problem Improving out-of-sample performance in machine learning models due to distributional uncertainty.
method Combining Bayesian nonparametric theory and decision-theoretic preferences to propose a robust optimization criterion.
result The proposed robust optimization procedure provides favorable statistical guarantees and tractable approximations.
VAE leverages MMSE channel estimation with data-driven modeling.
problem Data-driven channel estimation for wireless communications.
method Variational autoencoder (VAE) modeling of channel distribution and LMMSE approximation.
result VAE-based channel estimators approximate MMSE performance with practical training methods.
Designs a robust data-driven decision-making model to handle multiple overfitting sources.
problem Overfitting in data-driven models due to statistical error, data noise, and data misspecification.
method Holistic distributionally robust optimization formulation combining Kullback-Leibler and Lévy-Prokhorov approaches.
result Guaranteed holistic protection against statistical error, data noise, and data misspecification.
Paper proposes a robust hypothesis testing method using Sinkhorn distance.
problem Hypothesis testing for small samples.
method Data-driven approach using Sinkhorn uncertainty sets.
result The method provides a more flexible detector compared to Wasserstein robust test.
METASET selects diverse unit cells for efficient data-driven metamaterial design.
problem Imbalanced datasets in unit cells can bias data-driven metamaterial design.
method METASET uses similarity metrics and DPPs to select diverse subsets of unit cells.
result Smaller, diverse subsets improve search process and structural performance.
Localization of unknown faults in industrial systems is a difficult task for data-driven diagnosis methods. The classification performance of many machine learning methods relies on the quality of training data. Unknown faults, for example faults not represented in training data, can be detected using, for example, ano…
Unified framework for DRO and DTA using Bayesian nonparametrics.
problem Combining DRO and DTA under ambiguity.
method Unified framework using DP and HDPs, with outlier robustness.
result Favorable performance in prediction accuracy and stability.