This paper reviews recent advances in the field of optimization under uncertainty via a modern data lens, highlights key research challenges and promise of data-driven optimization that organically integrates machine learning and mathematical programming for decision-making under uncertainty, and identifies potential r…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “data-driven optimization”
Proposes data-driven methods for estimating conditional expectations.
problem Estimating conditional expectations when underlying density is unknown.
method Data-driven techniques to directly estimate conditional expectations from training data.
result Extends data-driven method to solve nonlinear equations in stochastic optimization.
Optimal data-driven formulations are found for learning and decision-making with historical data.
problem Designing optimal learning and decision-making formulations from historical data.
method Define a yardstick for measuring formulation quality, then construct an optimal formulation that is uniformly closer to the true cost.
result Existence of three distinct out-of-sample performance regimes with corresponding optimal formulations.
New method uses FY loss for better inverse optimization.
problem Estimating unknown parameters from noisy and suboptimal solutions.
method Fenchel-Young loss approach for efficient gradient-based optimization.
result Significant improvement in parameter estimation accuracy and computational speed.
Stochastic cutting planes improve data-driven optimization speed.
problem Data-driven Mixed-Integer Nonlinear Optimization problems.
method Stochastic version of cutting-plane method.
result Stochastic algorithm converges to ε-optimal solution with high probability.
Study integrates machine learning with SAA for optimizing decisions based on uncertain parameters and covariates.
problem Optimizing decisions under uncertain parameters and covariates.
method Data-driven frameworks integrating machine learning prediction models within SAA for scenario generation.
result Consistent and asymptotically optimal solutions under certain conditions, with finite sample guarantees.
Data-driven Distributionally Robust Optimization (DD-DRO) via optimal transport has been shown to encompass a wide range of popular machine learning algorithms. The distributional uncertainty size is often shown to correspond to the regularization parameter. The type of regularization (e.g. the norm used to regularize)…
Data-driven Option Pricingq-fin.PR
Data-driven method for option pricing using historical asset prices.
problem Tackling the gap between historical asset prices and risk-neutral option pricing.
method Identifying a pricing kernel process, solving utility maximization and functional optimization problems using deep learning.
result Demonstrated the efficiency of the data-driven option pricing methodology.
End-to-end framework optimizes constrained trajectories using data-driven methods.
problem Optimizing trajectories under constraints with limited dynamics knowledge.
method Data-driven approach decomposes trajectories into function basis, uses maximum a posteriori for optimization, and incorporates linear constraints.
result Commanding results in aeronautics and sailing route optimization.
Study optimal stopping for diffusion processes using data-driven methods.
problem Optimal stopping for diffusion processes under unknown conditions.
method Data-driven approach, deriving upper and lower bounds on simple and cumulative regret.
result Verified minimax optimality and improved convergence rates.
The paper tackles data-driven optimal control of unknown nonlinear systems using RKHS.
problem Unknown nonlinear dynamics and stage cost functions.
method Embed state densities into RKHS, learn Markov operators, solve Hamilton-Jacobi-Bellman recursions.
result Solves a wide range of nonlinear control problems, including depth regulation.
Data-driven optimization improves mean-variance portfolios by penalizing norms.
problem Estimation error in mean-variance optimization.
method Augment MVO with norm penalties, use neural networks for optimization, and compute derivatives implicitly.
result Data-driven optimization reduces portfolio risk compared to standard MVO.
A new framework for optimizing interventions with limited data.
problem Small data, default intervention data, unmodeled objectives, unforeseen consequences.
method Bandit data-driven optimization combining online bandit learning and offline predictive analytics.
result PROOF algorithm achieves no-regret and superior performance in simulations and real-world application.
Bayesian nonparametrics improves data-driven risk optimization under distributional uncertainty.
problem Improving out-of-sample performance in machine learning models due to distributional uncertainty.
method Combining Bayesian nonparametric theory and decision-theoretic preferences to propose a robust optimization criterion.
result The proposed robust optimization procedure provides favorable statistical guarantees and tractable approximations.
A new framework optimizes manufacturing decisions with less data and time.
problem Optimizing complex systems with multiple conflicting objectives.
method Data-driven Bayesian optimization using sequential learning.
result The proposed algorithm achieves the actual Pareto front with less data.
The optimization of composition and processing to obtain materials that exhibit desirable characteristics has historically relied on a combination of scientist intuition, trial and error, and luck. We propose a methodology that can accelerate this process by fitting data-driven models to experimental data as it is coll…
New insights into bias-variance tradeoff for data-driven optimization under local misspecification.
problem Understanding the relative performance of SAA, IEO, and ETO under local misspecification.
method Developed a local misspecification perspective using contiguity theory in statistics.
result Explicit expressions for decision bias and geometric understanding of variance.
Proposes a sliding window method for better portfolio trading.
problem Log-optimal portfolio problem with time-varying weights.
method Data-driven sliding window approach to solve log-optimal portfolio problem.
result Trading strategy outperforms classical log-optimal portfolio in cumulative returns.
Proposes robust assortment optimization from observational data.
problem Real-world scenarios often violate assumptions of stable customer preferences and correct choice models.
method Develops a robust framework that accounts for potential distributional shifts in customer choice behavior.
result Uncovered the notion of ``robust item-wise coverage'' as the minimal data requirement for sample-efficient robust assortment learning.
Optimizes decisions without knowing the true distribution using historical data.
problem Optimizing decisions without knowing the true distribution.
method Combines sampling and bisection search algorithms to solve an optimization problem.
result Proves sufficient conditions for local out-of-sample optimality.
Designs a robust data-driven decision-making model to handle multiple overfitting sources.
problem Overfitting in data-driven models due to statistical error, data noise, and data misspecification.
method Holistic distributionally robust optimization formulation combining Kullback-Leibler and Lévy-Prokhorov approaches.
result Guaranteed holistic protection against statistical error, data noise, and data misspecification.
Minimizing the Weighted Number of Tardy Jobs: Data-Driven Heuristic for Single-Machine Schedulingcs.LG
A heuristic minimizes tardy jobs' total weight on single-machine scheduling.
problem Minimizing tardy jobs' total weight on single-machine scheduling.
method Data-driven heuristic combining machine learning and problem-specific characteristics.
result Significantly outperforms state-of-the-art in optimality gap and adaptability.
Adaptive BO improves solder joint reliability by 3% with half the computational cost.
problem Improving solder joint reliability under thermomechanical loading.
method Adaptive Bayesian optimization with Gaussian process regression.
result Adaptive BO outperforms regular BO by 3% on average at any given computational budget.
The paper tackles robust control with uncertain dependence using data-driven methods.
problem Nonparametric robust control under dependence uncertainty in multi-period stochastic systems.
method Nonparametric adaptive robust control framework using stochastic gradient descent ascent algorithm.
result The controller benefits from knowing more about the uncertain model.
Recently, (Blanchet, Kang, and Murhy 2016, and Blanchet, and Kang 2017) showed that several machine learning algorithms, such as square-root Lasso, Support Vector Machines, and regularized logistic regression, among many others, can be represented exactly as distributionally robust optimization (DRO) problems. The dist…
New method speeds up k-means clustering using sketch-and-solve.
problem Efficiently solving k-means clustering for large datasets.
method Sketch-and-solve approach with Peng-Wei semidefinite relaxation.
result Provides high-confidence lower bounds on k-means optimal value.
Unified approach for data-driven control of stochastic processes.
problem Developing practical strategies for stochastic control problems with unknown dynamics.
method Reduction to rate-optimal estimators of invariant distribution risk.
result Data-driven strategies can achieve better performance than known methods.
Most of Markov Chain Monte Carlo (MCMC) and sequential Monte Carlo (SMC) algorithms in existing probabilistic programming systems suboptimally use only model priors as proposal distributions. In this work, we describe an approach for training a discriminative model, namely a neural network, in order to approximate the …
This paper tackles regularization parameter learning in inverse problems using data-driven bilevel optimization.
problem Finding optimal regularization parameters in inverse problems.
method Data-driven bilevel optimization approach, analyzing performance in large data samples.
result The approach can reduce computational cost through online numerical schemes based on stochastic gradient descent.
New data-driven Cartan connection tracks complex vascular structures.
problem Tracking complex vascular structures in multi-orientation images.
method Formulated a data-driven Cartan connection on for geodesic tracking.
result Improved geodesic tracking of vascular trees with globally optimal curves.
New method for fair resource allocation in AI-aware networks with unknown utility functions.
problem Fair resource allocation in AI-aware communication networks with unknown utility functions.
method Distributed, data-driven bilevel optimization approach to learn surrogate utility functions.
result The proposed algorithm learns from data to autotune surrogate utility functions for unknown utility functions.
Data-driven method for error estimation without needing class complexity.
problem Constructing confidence intervals for a class of estimates.
method Data-driven approach to derive high-probability upper bounds on maximum error.
result Method naturally adapts to unknown correlation structures and works for finite and infinite classes.
Optimizes data-driven design problems on implicit manifolds using score functions.
problem Optimizing over implicit low-dimensional manifolds in high-dimensional data.
method Introduces a link function connecting data distribution to manifold operations, enabling efficient optimization.
result Establishes theoretical guarantees for feasibility and optimality of proposed algorithms.
Optimizes natural frequencies of cellular composites with various microstructures.
problem Designing cellular composites with diverse microstructures for maximizing natural frequencies.
method Data-driven topology optimization with a latent-variable Gaussian process model.
result Cellular designs with multiclass microstructures achieve higher natural frequencies.
Machine Learning and Hamilton-Jacobi-Bellman Equation for Optimal Decumulation: a Comparison Studymath.OC
A neural network approach solves optimal decumulation problems for pension plans.
problem Optimal asset allocation and withdrawal strategies for DC pension holders.
method Data-driven neural network optimization with customized activation functions.
result The neural network approach learns near-optimal solutions comparable to HJB PDE methods.
Study optimizes sampling to avoid extreme tail risks in unknown heavy-tailed distributions.
problem Identify optimal alternative with minimal extreme tail risk from unknown heavy-tailed distributions.
method Data-driven sequential sampling policies to maximize likelihood of selecting the optimal alternative.
result Proposed methods outperform existing approaches in identifying the optimal alternative.
Paper proposes a fast data-driven AC-OPF method using sparse hybrid Gaussian processes.
problem Optimizing electricity generation and delivery under generation uncertainty in modern power grids.
method Data-driven approach using sparse hybrid Gaussian processes to model power flow equations.
result Shows up to two times faster and more accurate solutions compared to state-of-the-art methods.
Paper develops a novel approach for optimal control using kernel methods.
problem Optimal control of nonlinear stochastic systems.
method Infinitesimal generator approach in reproducing kernel Hilbert spaces.
result Data-driven solution to optimal control problems.
New method improves combinatorial optimization by overcoming inefficient sampling.
problem Wandering in contours: sampling similar solutions for long periods.
method Reheated Gradient-based Discrete Sampling with a novel mechanism.
result Superior performance across various combinatorial optimization problems.
Cleaning the covariance matrix of strongly nonstationary systems with time-independent eigenvaluesstat.AP
We propose a method to clean covariance matrices of nonstationary systems by using time-independent eigenvalues.
problem Noise in covariance matrices of nonstationary systems with time-independent eigenvalues.
method Data-driven approach to use independent eigenvalues encoding long-term influence of future on present.
result Our method outperforms optimal stationary methods for filtering covariance matrix and its inverse.
Unified framework for DRO and DTA using Bayesian nonparametrics.
problem Combining DRO and DTA under ambiguity.
method Unified framework using DP and HDPs, with outlier robustness.
result Favorable performance in prediction accuracy and stability.
New optimization method corrects data-driven optimizer's curse.
problem Over-optimistic evaluation in data-driven optimization.
method Smoothed -Divergence Distributionally Robust Optimization (DRO).
result Statistical bound on out-of-sample performance nearly tightest.
Enhances topology optimization with multiclass microstructures using latent variable Gaussian process.
problem Lack of an inherent ordering or distance measure between different classes of microstructures.
method Extended latent-variable Gaussian process (LVGP) models to multi-response LVGP (MR-LVGP) models for metamaterials.
result Improved performance through consistent load-transfer paths for micro- and macro-structures.
Develops a machine learning approach for solving AC-OPF problems.
problem Nonlinear and computationally demanding AC chance-constrained OPF problem.
method Uses Gaussian process regression to approximate AC power flow equations.
result Demonstrates competitive and promising results compared to state-of-the-art approaches.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
Proposes CPO framework for robust decision-making with explainable uncertainty regions.
problem Overly conservative uncertainty regions in data-driven optimization lead to suboptimal decisions.
method Conformal-Predict-Then-Optimize (CPO) framework using conditional generative models and visual summaries.
result Demonstrates improved robustness and explainability in decision-making.
Optimizes basis functions for learning dynamical systems from data.
problem Learning suitable basis functions for dynamical systems from data.
method Gradient-based optimization framework for learning basis functions.
result Efficacy demonstrated on various benchmark problems.
Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimizationstat.ME
OptCS optimizes model selection after conformal inference, controlling FDR and power loss.
problem Challenges in model selection for conformal inference, especially when limited labeled data and many model choices are available.
method OptCS framework that allows valid statistical testing after flexible data-driven model optimization, using novel multiple testing procedures.
result Valid conformal p-values constructed despite substantial data reuse, maintaining FDR control.