Simulator imperfection, often known as model error, is ubiquitous in practical data assimilation problems. Despite the enormous efforts dedicated to addressing this problem, properly handling simulator imperfection in data assimilation remains to be a challenging task. In this work, we propose an approach to dealing wi…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “ensemble data assimilation”
AD-EnKFs use machine learning to improve data assimilation in high-dimensional systems.
problem Data assimilation in high-dimensional, unknown dynamics systems.
method Auto-differentiable ensemble Kalman filters blending machine learning and ensemble Kalman filters.
result AD-EnKFs outperform existing methods in the Lorenz-96 model.
EnFF uses flows to speed up DA in high dimensions.
problem Efficiently assimilating noisy data in high-dimensional systems.
method Flow Matching (FM) for training-free, scalable data assimilation.
result EnFF accelerates DA with improved cost-accuracy tradeoffs and scalability.
EnKF's update is shown to be similar to Matheron's method in Gaussian process regression.
problem Data assimilation in high-dimensional systems.
method Empirical Matheron update applied to EnKF.
result Ensemble Kalman Filter's update is equivalent to an empirical Matheron update.
Supervised learning from noisy observations: Combining machine-learning techniques with data assimilationphysics.data-an
Combines machine learning and data assimilation for improved forecasting.
problem Improving forecast accuracy with noisy observations.
method Sequentially learns a machine-learning model using an ensemble Kalman filter.
result The combined model achieves good forecast skill and computational efficiency.
This work preserves linear invariants in ensemble filters for non-Gaussian data assimilation.
problem Maintaining critical invariants like mass, stoichiometric balance, and charge in non-Gaussian data assimilation.
method Introducing a novel class of nonlinear ensemble filters using measure transport theory.
result Recovery of a constrained Kalman filter for Gaussian settings and combination with regularization techniques.
EnCF improves data assimilation for implicit, non-smooth observations.
problem Data assimilation challenges with implicit, many-to-one observations.
method EnCF uses a stochastic controlled flow to update forecast distributions.
result EnCF outperforms Kalman filters for non-Gaussian, implicit observations.
Proposes LAE-EnKF for improved nonlinear data assimilation.
problem Performance of EnKF deteriorates for strongly nonlinear dynamics.
method Reformulates assimilation in a learned latent space with linear dynamics.
result LAE-EnKF yields more accurate and stable assimilation.
Latent-EnSF improves data assimilation for high-dimensional systems with sparse observations.
problem Challenges in high-dimensional, nonlinear Bayesian filtering with sparse observations.
method A novel data assimilation method using latent representations and a coupled VAE for efficient state encoding and reconstruction.
result Latent-EnSF outperforms traditional methods in accuracy, convergence, and efficiency for complex systems.
Data assimilation for subsurface flow using latent diffusion models shows that ensemble Kalman methods may overestimate posterior uncertainty, while Monte Carlo sampling is more reliable.
problem Data assimilation for subsurface flow
method Ensemble Kalman smoother and Markov chain Monte Carlo sampling
result Monte Carlo sampling is more reliable than ensemble Kalman methods
Paper analyzes ensemble Kalman updates for effective dimension and localization.
problem Why small ensemble sizes work well in inverse problems and data assimilation.
method Non-asymptotic analysis of ensemble Kalman updates, focusing on effective dimension and localization.
result Rigorously explains why a small ensemble size is sufficient when prior covariance has moderate effective dimension.
EnSF uses image inpainting to handle partial observations in data assimilation.
problem Data assimilation challenges with partial observations.
method EnSF integrates image inpainting with diffusion models to predict unobserved states.
result EnSF successfully tracks SQG dynamics with partial observations.
A new ensemble filter uses transport maps and MMD optimization for high-dimensional data assimilation.
problem High-dimensional data assimilation challenges in ensemble filtering.
method Optimized Maximum Mean Discrepancy (MMD) for transport map construction.
result Significant improvement in robustness and posterior approximation.
New distributed EnKF method for non-sequential assimilation of large datasets.
problem Computational intensity and order dependencies in traditional EnKF.
method Distributed computing for full model error covariance matrix.
result Non-sequential assimilation outperforms sequential in performance.
Proposes a new method for data assimilation using closed-form conditional diffusion models.
problem Data assimilation for systems with complex, non-Gaussian probability distributions.
method Uses kernel density estimation to model joint distributions and leverages the score function for efficient evaluation.
result Outperforms ensemble Kalman and particle filters in nonlinear data assimilation problems.
Long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systemsmath.DS
This paper proves long-time accuracy of ensemble Kalman filters for chaotic and machine-learned systems.
problem Ensuring long-term accuracy of ensemble Kalman filters for complex dynamical systems.
method Established conditions for long-time accuracy of ensemble Kalman filters for chaotic and machine-learned dynamical systems.
result Ensemble Kalman filters maintain small estimation error over long time horizons for chaotic and machine-learned systems.
CG-EnKF and NS-EnKF outperform deep learning-based SF in data assimilation.
problem Data assimilation with non-linear perturbations.
method Two non-linear extensions of EnKF: CG-EnKF and NS-EnKF.
result CG-EnKF and NS-EnKF outperform SF in high-dimensional multiscale data assimilation.
LD-EnSF speeds up data assimilation with sparse observations.
problem Efficiently assimilate sparse and noisy data into complex dynamical systems.
method LD-EnSF uses latent dynamics networks and history-aware LSTM encoders to process sparse observations without full-space simulations.
result Achieves significant speedups over existing methods while maintaining high accuracy.
Analog ensemble data assimilation and a method for constructing analogs with variational autoencodersphysics.comp-ph
Analog methods improve forecast accuracy in complex models.
problem Improving forecast accuracy in complex models like Lorenz-96.
method Constructing analogs using variational autoencoders for ensemble data assimilation.
result Constructed analogs perform as well as a full ensemble square root filter.
The literature about history matching is vast and despite the impressive number of methods proposed and the significant progresses reported in the last decade, conditioning reservoir models to dynamic data is still a challenging task. Ensemble-based methods are among the most successful and efficient techniques current…
This paper explores online learning of dynamics and state using ensemble Kalman filters.
problem Reconstructing dynamics from partial and noisy observations in real-time.
method Ensemble Kalman filter (EnKF) family of algorithms for online learning of dynamics and state.
result Demonstrates the efficiency and accuracy of online learning methods using Lorenz models.
BI-EqNO: Generalized Approximate Bayesian Inference with an Equivariant Neural Operator Frameworkstat.ML
BI-EqNO improves Bayesian inference with flexible neural operators.
problem Inaccurate estimation of marginal likelihoods in approximate Bayesian methods.
method Equivariant neural operator framework for generalized approximate Bayesian inference.
result BI-EqNO enhances both deterministic and stochastic approaches to Bayesian inference.
DAISI improves data assimilation for complex systems with noisy observations.
problem Limited accuracy of classical DA methods in complex, nonlinear systems.
method Generative models with inverse sampling for flexible probabilistic inference.
result DAISI achieves accurate filtering results in challenging nonlinear systems.
A new ML-based filter improves data assimilation for nonlinear systems.
problem Improving data assimilation for nonlinear systems using ensemble methods.
method Developed a machine learning-based conditional mean filter (ML-EnCMF) integrating ANN and linear functions.
result ML-EnCMF outperforms EnKF and likelihood-based EnCMF in nonlinear systems.
GEnBP combines EnKF and GaBP for efficient high-dimensional inference.
problem Efficient inference in high-dimensional models.
method Gaussian Ensemble Belief Propagation algorithm combining EnKF and GaBP.
result GEnBP outperforms existing methods in accuracy and efficiency.
Adaptive algorithm improves nonlinear data assimilation for non-Gaussian systems.
problem Challenges of non-Gaussian statistics in data assimilation.
method Triangular measure transport with P-spline basis functions and an information criterion.
result Automatic selection of parsimonious parametrization for efficient adaptation.
Deep learning speeds up pressure prediction in carbon storage reservoirs.
problem Accurately forecasting reservoir pressure in geologic carbon storage projects with sparse well data.
method Combining InSAR surface displacement data with deep learning and data assimilation techniques.
result Workflow can predict reservoir pressure with high efficiency and uncertainty quantification.
Deep learning improves chaotic dynamics filtering without ensemble.
problem Discovering efficient DA schemes for chaotic dynamics.
method Residual Convolutional Neural Network for the analysis step.
result Deep learning achieves ensemble filtering accuracy without an ensemble.
Improving Bayesian filtering with strictly proper scoring rules
problem Bayesian filtering of partially and noisily observed dynamical systems
method Proper scoring ensemble filter (PSEF)
result Accurate approximation of challenging filtering distributions
The Ensemble Kalman Inversion Racephysics.data-an
Ensemble Kalman methods improve climate model calibration from noisy observations.
problem Calibrating parameters in complex climate models from noisy data.
method Comparing ensemble Kalman methods for efficiency in climate model calibration.
result Ensemble Kalman methods are more efficient and robust for parameter learning in climate models.
Surrogate models improve tidal model calibration efficiency.
problem Efficiently calibrate complex tidal models for climate change scenarios.
method Proposes two surrogate-based methods to replace complex models: PODEn3DVAR and POD-PCE-3DVAR.
result Both methods show superior convergence and robustness to noise compared to classical 3DVAR.
A novel method, based on the combination of data assimilation and machine learning is introduced. The new hybrid approach is designed for a two-fold scope: (i) emulating hidden, possibly chaotic, dynamics and (ii) predicting their future states. The method consists in applying iteratively a data assimilation step, here…
A new method estimates time-varying parameters in earth system models using offline and online data assimilation.
problem Estimating time-varying parameters in complex earth system models.
method Hybrid Offline Online Parameter Estimation with Particle Filtering (HOOPE-PF)
result HOOPE-PF outperforms existing methods, especially with small ensemble sizes.
Combining data assimilation and machine learning to infer unresolved scale parametrisationphysics.comp-ph
Combines ML and DA to infer unresolved scale parametrisation from noisy data.
problem Training ML-based parametrisations from realistic, noisy and sparse observations.
method Two-step process: DA for state estimation, ML for model error prediction.
result Hybrid model produces better forecasts and attractor representation.
EnKBS smoothes complex systems with future observations for causal inference.
problem Improving state estimation in complex systems with rapid dynamics.
method Continuous-time ensemble Kalman-Bucy smoother for nonlinear dynamical systems.
result EnKBS provides derivative-free framework with high skill in various scientific problems.
Developing parsimonious ensembles using ensemble diversity within a reinforcement learning frameworkcs.LG
Heterogeneous ensembles built from the predictions of a wide variety and large number of diverse base predictors represent a potent approach to building predictive models for problems where the ideal base/individual predictor may not be obvious. Ensemble selection is an especially promising approach here, not only for …
MEDIDA discovers model errors in chaotic systems using sparse regression and data assimilation.
problem Model errors in chaotic systems lead to significant discrepancies between model predictions and real-world states.
method MEDIDA combines Bayesian sparse regression and data assimilation to estimate and interpret model errors from noisy observations.
result MEDIDA successfully identifies different types of model errors in the chaotic Kuramoto-Sivashinsky system.
Combines deep generative models with ensemble methods for subsurface property estimation.
problem Estimating spatially distributed subsurface properties from sparse measurements.
method Wasserstein Generative Adversarial Network (WGAN-GP) and Ensemble Smoother with Multiple Data Assimilation (ES-MDA).
result The proposed method outperforms variational inversion methods, especially for channelized and fractured fields.
Combining data assimilation and machine learning to estimate parameters of a convective-scale modelphysics.ao-ph
Artificial neural networks estimate model parameters from observations, reducing model errors.
problem Estimating parameters of convection-permitting models from observations.
method Training Bayesian neural networks and point estimate neural networks on atmospheric state observations.
result Artificial neural networks can estimate model parameters and their statistics.
SDA method reduces memory and time for assimilating noisy geophysical data.
problem Challenges in identifying state trajectories of high-dimensional geophysical systems.
method Score-based data assimilation with modified score network architecture.
result Promising results for a two-layer quasi-geostrophic model.
A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problemscs.LG
A deep-learning-based surrogate model is developed and applied for predicting dynamic subsurface flow in channelized geological models. The surrogate model is based on deep convolutional and recurrent neural network architectures, specifically a residual U-Net and a convolutional long short term memory recurrent networ…
SOAD model improves data assimilation for nonlinear systems.
problem Challenges in classical data assimilation with high nonlinearity.
method State-Observation Augmented Diffusion (SOAD) model for data-driven assimilation.
result SOAD model matches true posterior distribution under mild assumptions.
Machine learning improves solving inverse problems and integrating data.
problem Solving complex inverse problems and integrating data effectively.
method Integrates machine learning techniques with inverse problems and data assimilation.
result Demonstrates machine learning's potential to enhance these fields.
New method calibrates predictions in chaotic systems using variational inference.
problem Uncertainty in data assimilation for chaotic systems.
method Variational inference applied to multivariate Gaussian distribution.
result Nearly perfectly calibrated predictions in chaotic Lorenz-96 dynamics.
MASF improves score-based filters for high-dimensional nonlinear systems with spatially sparse measurements.
problem Challenges in data assimilation for nonlinear, high-dimensional systems with spatially sparse measurements.
method Developed a forward process tailored for filtering that transforms the system state toward the measurement space, enabling a theoretically sound formulation of the likelihood score.
result MASF shows improved performance over existing score-based filters and ensemble-type Kalman filters, achieving up to a 28.2× wall-clock speedup.
ACI uses Bayesian data assimilation to trace causes from effects in complex systems.
problem Capturing instantaneous, time-evolving causal relationships in complex, high-dimensional systems.
method Assimilative causal inference (ACI) leverages Bayesian data assimilation to trace causes backward from observed effects.
result ACI provides online tracking of causal roles that may reverse intermittently and reveals how far effects propagate.
New method combines neural networks and data assimilation for indoor air quality prediction.
problem Accurate and fast prediction of indoor air quality using real-time data.
method Combines Data Assimilation and Machine Learning using a Convolutional neural network and Long-Short-Term-Memory.
result Improved accuracy of dynamic system representation by integrating real data.
Stochastic methods improve data assimilation with high-frequency sensor data.
problem Computational challenges in data assimilation with high-frequency sensor data.
method Adapted stochastic approximation methods to handle high-frequency observations.
result Produces high-quality estimates using all observations without compromising statistical accuracy.