Gaussian surrogates improve Poisson imaging performance at low doses.
problem Improving Poisson imaging performance at low doses.
method Analysis of Poisson and Gaussian surrogate reconstruction objectives under Poisson noise.
result Gaussian surrogates can achieve MSE comparable to Poisson MAP at low doses.
In this work we provide a theoretical framework for structured prediction that generalizes the existing theory of surrogate methods for binary and multiclass classification based on estimating conditional probabilities with smooth convex surrogates (e.g. logistic regression). The theory relies on a natural characteriza…
We study consistency properties of machine learning methods based on minimizing convex surrogates. We extend the recent framework of Osokin et al. (2017) for the quantitative analysis of consistency properties to the case of inconsistent surrogates. Our key technical contribution consists in a new lower bound on the ca…
In this paper, we consider an ℓ0-norm penalized formulation of the generalized eigenvalue problem (GEP), aimed at extracting the leading sparse generalized eigenvector of a matrix pair. The formulation involves maximization of a discontinuous nonconcave objective function over a nonconvex constraint set, and is…
Proposes SPFB method for optimizing partition functions in stochastic learning.
problem Optimizing partition functions in stochastic learning settings.
method Stochastic Gradient Bound (SPFB) method based on upper-bounding the partition function with a quadratic surrogate.
result Sub-linear convergence rate of SPFB method and efficient training of deep learning models.
SRRM improves recursive transport surrogates in the small-discrepancy regime.
problem Insufficient understanding of recursive partitioning methods' statistical behavior and resolution in the small-discrepancy regime.
method Introduced Selective Recursive Rank Matching (SRRM) to improve the resolution of Recursive Rank Matching (RRM).
result SRRM yields a higher-fidelity practical surrogate for the Wasserstein distance at moderate additional computational cost.
HAMD optimizes cubic portfolios without quadratization, achieving better results.
problem Optimizing higher-order portfolio models with reduced distortion.
method Hybrid pipeline combining continuous Hamiltonian search, cardinality-preserving projection, and iterated local search.
result HAMD achieves significantly lower native cubic objective values than classical heuristics.
We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions. For such a problem, we construct a sequence of surrogate convex constrained optimiza…
Signal estimation problems with smoothness and sparsity priors can be naturally modeled as quadratic optimization with ℓ0-"norm" constraints. Since such problems are non-convex and hard-to-solve, the standard approach is, instead, to tackle their convex surrogates based on ℓ1-norm relaxations. In this paper…
Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.
problem The performance of local update methods in federated learning and meta-learning is sensitive to learning rates.
method Proved that local update methods perform SGD on a surrogate loss function, characterized the surrogate loss, and derived convergence rates.
result Proper learning rate tuning is crucial for near-optimal behavior in communication-limited settings.
Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.
problem The trade-off between smoothness and margin-based losses in classification and structured prediction.
method Linear-Core (LC) Surrogates, a family of convex loss functions that stitch a linear core to a smooth tail.
result LC Surrogates achieve fast linear consistency rates while maintaining differentiability and strict H-consistency bounds. For semi-supervised techniques to be applied safely in practice we at least want methods to outperform their supervised counterparts. We study this question for classification using the well-known quadratic surrogate loss function. Using a projection of the supervised estimate onto a set of constraints imposed by the u…
Enhanced kernel framework for advanced data forecasting.
problem Complex mechanical behaviors and timefrequency dynamics in aircraft systems.
method Frequency-aware surrogate modeling with SMT kernels, extending kernel types and incorporating derivatives.
result Improved accuracy in forecasting CO2 concentrations and airline passenger traffic.
New insights into convergence and accuracy trade-offs in federated and meta-learning.
problem Understanding the trade-offs between convergence and accuracy in federated and meta-learning.
method Generalized local update methods, proving equivalence to first-order optimization on a surrogate loss.
result Novel convergence rates and insights into the importance of algorithmic choices in communication-limited settings.
In this paper, a new sequential surrogate-based optimization (SSBO) algorithm is developed, which aims to improve the global search ability and local search efficiency for the global optimization of expensive black-box models. The proposed method involves three basic sub-criteria to infill new samples asynchronously to…
Develops a gradient-enhanced approach for online estimation in high-dimensional generalized linear models with streaming data.
problem Online estimation for high-dimensional generalized linear models with streaming data.
method Proposes a gradient-enhanced surrogate loss for non-distributed setting and extends to distributed streaming data.
result Derives non-asymptotic error bounds under high-dimensional scaling without batch-number constraint.
Gaussian equivalence fails for simple polynomial embeddings in quadratic scaling RF models.
problem Failure of Gaussian equivalence in polynomial feature embeddings under quadratic scaling.
method Introduced Conditional Gaussian Equivalent (CGE) model to capture non-Gaussian behavior.
result Correct asymptotics derived for training and test errors in CGE model.
A standard introduction to online learning might place Online Gradient Descent at its center and then proceed to develop generalizations and extensions like Online Mirror Descent and second-order methods. Here we explore the alternative approach of putting Exponential Weights (EW) first. We show that many standard meth…
This study proposes a graph partitioning method to improve spatial prediction models.
problem Improving interpretability of spatial prediction models in industries.
method Graph partitioning problem to minimize within-segment variances, formulated as mixed-integer quadratic programming.
result Approximation scheme efficiently identifies spatial segments, improving computational efficiency.
New attacks and defenses for GNNs on large graphs.
problem Vulnerability of GNNs to adversarial attacks on large graphs.
method Proposed two sparsity-aware first-order optimization attacks and a robust aggregation function.
result Attacks can double in strength, and defenses are effective at all scales.
New technique debiases distributed optimization, improving convergence rate.
problem Bias in local estimates limits effectiveness of distributed second order optimization.
method Surrogate sketching and scaled regularization to eliminate bias.
result The debiased local estimates lead to faster convergence in distributed optimization.
Scalable method completes ill-conditioned matrices from few samples.
problem Matrix completion from few samples for ill-conditioned matrices.
method Iterative algorithm combining IRLS, smoothing Newton, and proximal gradient methods.
result Local quadratic convergence rate and well-conditioned linear systems.
New bounds show polyhedral surrogates are optimal for generalization.
problem Proving generalization rates for polyhedral loss functions.
method Developed two general results for polyhedral surrogates.
result Polyhedral surrogates provide linear surrogate regret bounds, translating directly to target rates.
Structural reliability methods aim at computing the probability of failure of systems with respect to some prescribed performance functions. In modern engineering such functions usually resort to running an expensive-to-evaluate computational model (e.g. a finite element model). In this respect simulation methods, whic…
Study of eigenvalues in nonlinear kernels for classification of separable data.
problem Understanding the applicability of linear equivalents in nonlinearly separable data classification.
method Analysis of conjugate kernels and their quadratic equivalents for a canonical nonlinearly separable dataset (XOR problem).
result Identification of regimes where nonlinear kernels deviate from linear equivalents, leading to label-aligned eigenspaces.
We present a framework for automatically structuring and training fast, approximate, deep neural surrogates of stochastic simulators. Unlike traditional approaches to surrogate modeling, our surrogates retain the interpretable structure and control flow of the reference simulator. Our surrogates target stochastic simul…
Randomizing the Fourier-transform (FT) phases of temporal-spatial data generates surrogates that approximate examples from the data-generating distribution. We propose such FT surrogates as a novel tool to augment and analyze training of neural networks and explore the approach in the example of sleep-stage classificat…
Neural network discovers exact solutions to QP with linear constraints.
problem Discovering exact solutions to Quadratic Programs (QP) with linear constraints using neural networks.
method Proposes a neural network modeling approach that analytically derives model parameters from problem coefficients, ensuring closed-form solutions without training.
result The closed-form NN model produces exact solutions for every critical region of the QP solution function, outperforming DNNs and commercial solvers in terms of optimality and feasibility.
We formalize and study the natural approach of designing convex surrogate loss functions via embeddings, for problems such as classification, ranking, or structured prediction. In this approach, one embeds each of the finitely many predictions (e.g.\ rankings) as a point in Rd, assigns the original loss val…
The paper proposes a scalable framework for uncertainty quantification and propagation in surrogate-based Bayesian inference.
problem Uncertainty in surrogate models and its impact on inference and decision-making.
method Bayesian inference methods for surrogate models with measurement data.
result Scalable framework for uncertainty quantification and propagation in surrogate models.
Study assesses neural nets for optimization problems, highlighting SiLU's effectiveness.
problem Using neural nets for optimization problems, especially for accurate approximations.
method Determined best activation function (SiLU) for nonlinear optimization problems. Analyzed function approximations using neural networks and interpolation/regression models.
result Neural nets can deliver competitive zero- and first-order approximations but underperform on second-order approximations.
Paper proposes hybrid modeling to improve surrogate accuracy using multiple data sources.
problem Improving surrogate model accuracy by integrating simulation and real-world data.
method Two novel probabilistic approaches: separate and combined surrogates with weighting strategy.
result Hybrid models improve predictive accuracy and coverage compared to single-source surrogates.
Improving predictive understanding of Earth system variability and change requires data-model integration. Efficient data-model integration for complex models requires surrogate modeling to reduce model evaluation time. However, building a surrogate of a large-scale Earth system model (ESM) with many output variables i…
The paper proposes a method to assess surrogate heterogeneity in non-randomized data.
problem Lack of methods to evaluate surrogate heterogeneity in non-randomized data.
method Proposes a framework using meta-learners to assess surrogate heterogeneity in real-world data.
result Identifies individuals for whom the surrogate is a valid replacement of the primary outcome.
Thanks to their versatility, ease of deployment and high-performance, surrogate models have become staple tools in the arsenal of uncertainty quantification (UQ). From local interpolants to global spectral decompositions, surrogates are characterised by their ability to efficiently emulate complex computational models …
This paper develops efficient surrogate models for optimization of complex dynamical systems.
problem Computational expense in solving complex dynamical systems through numerical simulation.
method Combination of proper orthogonal decomposition and radial basis functions for constructing low-dimensional surrogate models.
result Surrogate models reduce computational time for optimization problems while maintaining accuracy.
We present surrogate regret bounds for arbitrary surrogate losses in the context of binary classification with label-dependent costs. Such bounds relate a classifier's risk, assessed with respect to a surrogate loss, to its cost-sensitive classification risk. Two approaches to surrogate regret bounds are developed. The…
It is widely conjectured that the reason that training algorithms for neural networks are successful because all local minima lead to similar performance, for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al., 2014). Performance is typically measured in terms of two metrics: training performanc…
The augmented Lagrangian (AL) method that solves convex optimization problems with linear constraints has drawn more attention recently in imaging applications due to its decomposable structure for composite cost functions and empirical fast convergence rate under weak conditions. However, for problems such as X-ray co…
This work improves surrogate models for balancing accuracy and cost in multi-fidelity methods.
problem Balancing accuracy and computational cost in multi-fidelity methods.
method Develops context-aware surrogate models for multi-fidelity importance sampling and Bayesian inverse problems.
result Context-aware surrogate models can lead to runtime speedups of up to one order of magnitude.
Adversarial consistency depends on the uniqueness of adversarial Bayes classifiers.
problem Consistency of adversarial surrogate losses is not guaranteed.
method Connected consistency of adversarial surrogate losses to the uniqueness of adversarial Bayes classifiers.
result A convex surrogate loss is statistically consistent for adversarial learning if and only if the adversarial Bayes classifier is unique.
New method simplifies checking consistency of differentiable loss functions.
problem Verifying consistency of differentiable loss functions is difficult.
method Developed a new approach called strong indirect elicitation (strong IE) to simplify checking consistency.
result Strong IE is equivalent to calibration for strongly convex, differentiable surrogates.
Paper bounds convergence rate of adversarial surrogate risk.
problem Vulnerability of binary classification models to adversarial attacks.
method Characterizes conditions for adversarial consistency and provides surrogate risk bounds.
result Surrogate risk bounds quantify the rate of convergence of adversarial classification risk.
Paper introduces new loss functions for multi-class abstention learning.
problem Learning with multi-class classification and the ability to abstain.
method Developed new families of surrogate losses for abstention.
result Proved strong consistency guarantees for new surrogate losses.
Proposes a method to refine PDE-driven high-dimensional rare-event simulation.
problem Challenges in constructing accurate surrogates for rare-event simulation.
method Adaptive importance sampling framework that refines a locally constructed surrogate.
result Achieves accuracy comparable to true-model adaptive importance sampling with fewer high-fidelity evaluations.
We establish linear regret bounds for convex smooth losses using Fenchel-Young losses.
problem Establishing linear regret bounds for convex smooth losses.
method Constructing a convex smooth surrogate loss using Fenchel-Young losses generated by the convolutional negentropy.
result We derive a smooth loss with a linear surrogate regret bound.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
A new framework improves reinforcement learning algorithms with policy guarantees.
problem Designing efficient and stable reinforcement learning algorithms.
method A general framework (FMA-PG) based on functional mirror ascent that constructs surrogate functions enabling policy improvement guarantees.
result The proposed framework enables policy improvement guarantees that hold regardless of policy parameterization, and recovers important heuristics.