Study shows proper initialisation of binary weights is crucial for deep neural networks.
problem Training stochastic binary neural networks with continuous surrogates is challenging.
method Developed new surrogates based on Markov chain theory and mean field analysis.
result Critical initialisations are necessary for training deep networks with binary weights.
MVRSM optimizes expensive functions with mixed variables, outperforming state-of-the-art methods.
problem Minimizing expensive functions with mixed continuous and integer variables.
method Mixed-Variable ReLU-based Surrogate Modelling (MVRSM) using rectified linear units.
result MVRSM outperforms state-of-the-art methods on synthetic and real-life benchmarks.
This paper develops nudging algorithms using learned surrogates for state estimation in dynamical systems.
problem Estimating the state of a dynamical system from partial observations when dynamics are unknown or expensive to simulate.
method Unified finite-dimensional analysis of nudging algorithms employing learned surrogate models of the dynamics.
result Nudging algorithms with surrogate models retain exponential convergence up to an explicit error floor.
SPARC improves continual learning with minimal memory and computational overhead.
problem Efficient continual learning for deep neural networks.
method Combines task-specific working memories and task-agnostic semantic memory.
result Significantly reduces parameter usage (6% of full-model surrogates) while maintaining performance.
The paper proposes a method to estimate treatment effects using surrogates when primary outcomes are missing.
problem Missing primary outcomes in causal inference applications can lead to biased estimates.
method Doubly robust method that uses both labeled and unlabeled data, incorporating surrogates.
result The proposed estimator is asymptotically normal and has improved variance compared to methods using only labeled data.
We propose a robust adversarial prediction framework for general multiclass classification. Our method seeks predictive distributions that robustly optimize non-convex and non-continuous multiclass loss metrics against the worst-case conditional label distributions (the adversarial distributions) that (approximately) m…
New model guarantees integer optimal solutions for combinatorial problems.
problem Finding optimal solutions for combinatorial problems with costly or noisy evaluations.
method Developed a surrogate model with integer-valued minima for combinatorial optimization.
result Outperforms other optimization algorithms on specific combinatorial problems.
Develops a numerical algorithm for stochastic impulse control using regression surrogates.
problem Optimal impulse control in stochastic processes.
method Generates statistical surrogates for continuation and intervention functions, recursively trained over simulated state trajectories.
result Demonstrates flexibility and extensibility of the numerical scheme through case studies.
The paper tackles fast rates in structured prediction problems.
problem Structured prediction problems with discrete outputs.
method Introducing continuous surrogate problems and leveraging their convergence rates for discrete problems.
result Super fast rates, including exponential rates, for excess risk in structured prediction problems.
Machine learning models estimate nutrient concentrations from water quality surrogates.
problem Estimating high frequency nutrient concentrations from limited in-situ measurements.
method Used machine learning (Random Forests) to estimate nutrient concentrations using surrogate measures.
result Reduced RMSE by up to 60.1% compared to linear models, with additional sensors not providing significant benefits.
New GP kernel handles mixed-categorical data, improving model accuracy.
problem Improving Gaussian process models for mixed-categorical data.
method Extends continuous exponential kernels to handle mixed-categorical variables.
result The proposed GP model gives higher likelihood and smaller residual error.
Deep Jump Gaussian Processes model high-dimensional piecewise functions.
problem Modeling high-dimensional piecewise continuous functions with limited accuracy.
method Integrates region-specific locally linear projections with Jump Gaussian Processes (JGP) to capture local low-dimensional subspace structures.
result DJGP achieves superior predictive accuracy and more reliable uncertainty quantification compared to existing methods.
A new emulator connects observables directly from data.
problem Constructing fast and accurate surrogate models for robust predictions.
method Introduces Multiparameter Eigenvalue Problem (MEP) emulator trained with Eigenvector Continuation (EC) and Parametric Matrix Model (PMM) data.
result The MEP emulator can make predictions directly from observables to observables.
SMT-EX enhances SMT for explaining surrogate models of mixed-variable design problems.
problem Making decisions and understanding complex systems using surrogate models of mixed-variable design problems.
method Integrates explainability techniques into SMT, including Shapley Additive Explanations, Partial Dependence Plot, and Individual Conditional Expectations.
result Demonstrates versatility in addressing diverse problem characteristics.
Study on calibration and consistency of adversarial surrogate losses.
problem Designing robust classifiers with theoretical guarantees.
method Extensive analysis of H-calibration and H-consistency of adversarial surrogate losses.
result Some convex loss functions and supremum-based convex losses are not H-calibrated for important hypothesis sets.
A hybrid model for Bayesian optimization handles mixed variables using MCTS for categorical and GP for continuous.
problem Optimizing functions with mixed variable types (continuous, integer, categorical).
method Merges MCTS for categorical and GP for continuous variables, integrates UCTS search strategy, and dynamically selects kernels.
result Hybrid models outperform traditional methods in Bayesian optimization.
This text explores strategies for learning discrete latent structures in neural networks.
problem Learning discrete latent structures in neural networks is challenging.
method Continuous relaxation, surrogate gradients, and probabilistic estimation.
result Many latent structure learning strategies use the same fundamental building blocks but apply them differently.
Proposes a framework to incorporate global sensitivity into local surrogate models.
problem Narrowing focus to local scale in surrogate modeling leads to re-learning global trends.
method Integrates global sensitivity analysis into local surrogate models through input warping.
result Local models become equally sensitive to all input directions, focusing on local dynamics.
SBBO optimizes complex spaces using sampling-based models.
problem Optimizing complex spaces with discrete variables.
method Simulation Based Bayesian Optimization (SBBO) using sampling-based surrogate models.
result Empirical effectiveness of SBBO in combinatorial optimization.
New PG losses improve decision optimization in misspecified models.
problem Improving decision optimization in models that are not perfectly specified.
method Introducing Perturbation Gradient (PG) losses to connect decision loss with directional derivatives and optimizing using gradient techniques.
result PG losses yield best-in-class policies asymptotically, even in misspecified settings.
Deep adaptive sampling improves surrogate modeling for complex systems.
problem Statistical errors in random sampling for high-dimensional problems.
method DAS^2 method, using deep generative models to refine training sets.
result Reduces statistical errors in approximating solutions for low-regularity problems.
A method to improve surrogate model accuracy using multiple fidelity models.
problem Efficiently combining models of varying accuracy and computational cost.
method Multifidelity Gaussian process models and leave-one-out cross-validation.
result Reduced LOO-CV error at the highest fidelity through adaptive learning.
Annealed Entropic Allocation improves ranking and selection by mitigating hard switching and improving finite-budget discrimination.
problem Sequential budget allocation in ranking and selection
method Annealed weighted soft-min framework
result Surrogate converges uniformly to the hard minimum, soft-min weights concentrate on active challengers, and target allocation map is continuous.
Bayesian optimization uses BNNs as efficient surrogate models for expensive function evaluations.
problem Optimizing expensive objective functions using Gaussian process surrogates.
method Study of Bayesian neural networks (BNNs) as alternatives to standard Gaussian process (GP) surrogates for optimization.
result Infinite-width BNNs are particularly promising, especially in high dimensions.
New method estimates active subspaces for jump-discontinuous functions.
problem Estimating active subspaces for discontinuous functions like ABMs.
method Extending active subspaces to discontinuous functions, using Gaussian process.
result Identifies important parameters in ABM simulations of refugee movement.
Unified framework for modeling hierarchical spaces in design problems.
problem Challenges in modeling hierarchical, conditional, heterogeneous, or tree-structured domains.
method Unified framework combining feature modeling and graph theory, introducing meta and partially-decreed variables.
result Demonstrated effectiveness on complex system design problems, including neural networks and green-aircraft.
Develops active learning for Jump Gaussian Process models.
problem Optimizing experimental designs and steering data acquisition in complex systems.
method Active learning of piecewise Jump Gaussian Process (Jump GP) models, accounting for model bias.
result Demonstrates the importance of accounting for model bias in Jump GP models.
The paper analyzes top-k classification and proposes consistent loss functions.
problem Understanding consistency of top-k classification in challenging tasks.
method Theoretical analysis, defining top-k calibration, proposing new loss functions.
result Proposes a new consistent hinge loss and a top-k calibrated convex loss.
PPOS improves PPO by smoothing the surrogate objective function.
problem Performance instability and optimization inefficiency in PPO.
method Use of a functional clipping method instead of a flat clipping method.
result PPOS conducts more accurate updates and outperforms other PPO variants.
In this paper, we consider an ℓ0-norm penalized formulation of the generalized eigenvalue problem (GEP), aimed at extracting the leading sparse generalized eigenvector of a matrix pair. The formulation involves maximization of a discontinuous nonconcave objective function over a nonconvex constraint set, and is…
The paper interprets policy-gradient algorithms using continuation theory.
problem Optimizing nonconvex functions in reinforcement learning.
method Formulates policy optimization as optimization by continuation, interprets policy-gradient algorithms as implicitly optimizing deterministic policies.
result Exploration in policy-gradient algorithms is seen as computing a continuation of the return of the policy.
Reward tweaking optimizes behavior for long-term goals by adjusting the reward function.
problem Optimizing behavior for long-term goals in reinforcement learning with unstable long planning horizons.
method Reward tweaking learns a surrogate reward function that induces optimal behavior for the original task.
result Reward tweaking guides agents towards better long-term returns while planning for short horizons.
A new tree-based model improves uncertainty estimation in sequential optimization.
problem Improving uncertainty estimation in sequential model-based optimization.
method Proposed a new ensemble of randomized trees (BwO forest) with bagging and oversampling.
result BwO forest outperforms existing tree-based models in various optimization scenarios.
Study uses neural networks to detect nonlinear dynamics in short time series.
problem Challenges in testing dynamical nonlinearities in short time series.
method Recurrent neural network classification framework using raw time series data.
result Classifier accuracy is higher than 50% for chaotic processes, around 50% for nonlinearly correlated noise.
O3 optimizes samples from generative models without extra training.
problem Optimizing specific criteria within rich data distributions.
method Surrogate latent spaces for efficient black-box optimisation.
result Surrogate-space optimisation finds higher-scoring samples.
New bounds show polyhedral surrogates are optimal for generalization.
problem Proving generalization rates for polyhedral loss functions.
method Developed two general results for polyhedral surrogates.
result Polyhedral surrogates provide linear surrogate regret bounds, translating directly to target rates.
A new method for creating simpler models from complex ones.
problem Creating accurate approximations of complex models at reduced costs.
method Sequential adaptive surrogate modeling based on locally spectral expansions.
result Stochastic spectral embedding (SSE) shows good approximation capabilities and scalability.
Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifi…
Estimates log-likelihood of interacting particle systems using virtual particles.
problem Inconsistent estimation of finite-particle log-likelihood in large particle systems.
method Stochastic gradient estimate using continuous trajectory and virtual particle systems.
result Convergence to stationary points of limiting mean-field system's log-likelihood.
ZICO learns DAGs from zero-inflated count data efficiently.
problem Learning network structures from zero-inflated count data.
method ZICO uses node-wise likelihoods with canonical links and a differentiable surrogate constraint for acyclicity.
result ZICO achieves superior performance and faster runtimes on simulated data.
SRMC framework reduces Monte Carlo variance by history-based sampling in high-dimensional spaces.
problem Efficient sampling in high-dimensional discrete or continuous state spaces.
method Score-Repellent Monte Carlo (SRMC) framework that summarizes history through running average of score evaluations.
result Improves estimator variance and mode coverage with constant memory usage.
This thesis tackles learning with multi-class abstention and multi-expert deferral, improving model reliability and efficiency.
problem Improving model reliability and efficiency in large language models (LLMs) by leveraging multiple experts.
method Developed new surrogate losses and consistency guarantees for multi-class classification and regression with deferral.
result Strong consistency guarantees for surrogate losses in multi-class classification and regression with deferral.
Develops a new method for learning discrete distributions without embedding them in a continuous space.
problem Challenges in learning discrete distributions using current methodologies.
method Introduces a MAD invertible map and a mixed variational flow (MAD Mix) for discrete distributions.
result MAD Mix produces more reliable approximations than continuous-embedding flows.
Improving in-context learning for latent space Bayesian optimization by adapting pretraining on molecular latent space.
problem Improving in-context learning for latent space Bayesian optimization.
method Adapting pretraining on molecular latent space.
result Achieving strong performance on held-out molecular optimization benchmarks.
Paper tackles regression with cost-based rejection, balancing prediction and rejection costs.
problem Regression with cost-based rejection, balancing prediction and rejection costs in a continuous target space.
method Formulated expected risk, derived Bayes optimal solution, proposed surrogate loss function.
result Bayes optimal solution can be recovered by the proposed surrogate loss function.
This work tackles robust optimization with multiple objectives for structural design.
problem Optimizing structural design under uncertainty for multiple conflicting objectives.
method Adaptive surrogate modeling using Kriging to approximate computationally expensive models.
result The importance of replacing the heating system in building renovations is highlighted.
We develop a framework for consistent polyhedral surrogates in classification and prediction.
problem Designing consistent polyhedral surrogates for classification and prediction problems.
method Formalizing and studying embeddings of predictions as points in R^d, assigning original loss values, and convexifying to create surrogates.
result Established a strong connection between embeddings and polyhedral surrogates, providing constructions and proofs of consistency or inconsistency.
Revisits PPO design choices, exposing failure modes and proposing alternatives.
problem Failure modes of standard PPO in new environments.
method Revisits standard PPO design choices, exposes failure modes, and proposes alternative approaches.
result Alternative design choices prevent failure modes in new environments.