Develops a method to evaluate OPE robustness to hyperparameters and policies.
problem Difficulty in selecting and tuning OPE estimators due to limited experimental evaluations.
method Introduces IEOE (Interpretable Evaluation for Offline Evaluation) to assess robustness.
result Demonstrates improved evaluation of OPE estimators' reliability.
Procedure for evaluating RL policies in risky settings.
problem Identifying risky episodes in RL policy applications.
method Gumbel-Max SCMs for generating counterfactual trajectories.
result Identifies episodes with dramatic counterfactual reward differences.
Vision and language tasks often fail to test AI comprehensively.
problem Current vision and language tasks are flawed due to dataset and evaluation issues.
method Review of current state and proposal for improvement.
result State-of-the-art systems perform well due to dataset and evaluation flaws.
We consider evaluation methods for payoffs with an inherent financial risk as encountered for instance for portfolios held by pension funds and insurance companies. Pricing such payoffs in a way consistent to market prices typically involves combining actuarial techniques with methods from mathematical finance. We prop…
SIREN protocol corrects optimistic winner's scores in LLM evaluation.
problem Optimistic winner's scores in LLM evaluation due to adaptive benchmarking.
method SIREN protocol that freezes post-search shortlist, separates selection and evaluation, and uses bootstrap for uncertainty quantification.
result SIREN provides valid confidence intervals for procedure-performance curves and deployment conclusions.
Paper proposes a method to create more reliable confidence intervals for off-policy evaluations.
problem Creating reliable confidence intervals for off-policy evaluations.
method Proposes a deeply-debiasing procedure to construct efficient, robust, and flexible confidence intervals.
result Validated by theoretical results and numerical experiments, the method improves the reliability of off-policy evaluations.
The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. Ho…
Procgen Benchmark uses procedurally generated games to test reinforcement learning.
problem Lack of diverse and high-quality training environments for reinforcement learning.
method Developed 16 procedurally generated game-like environments and used them to benchmark reinforcement learning.
result Procedurally generated environments are essential for training and evaluating reinforcement learning agents.
In this paper, we consider the challenge of maximizing an unknown function f for which evaluations are noisy and are acquired with high cost. An iterative procedure uses the previous measures to actively select the next estimation of f which is predicted to be the most useful. We focus on the case where the function ca…
We design and analyse variations of the classical Thompson sampling (TS) procedure for Bayesian optimisation (BO) in settings where function evaluations are expensive, but can be performed in parallel. Our theoretical analysis shows that a direct application of the sequential Thompson sampling algorithm in either synch…
Paper develops robust policy evaluation for reinforcement learning with outlier and heavy-tailed rewards.
problem Outlier contamination and heavy-tailed rewards in reinforcement learning.
method Develops a fully online robust policy evaluation procedure and efficient statistical inference.
result Establishes the Bahadur-type representation of the estimator and develops an online inference procedure.
TUDataset provides benchmark datasets for graph learning.
problem Lack of meaningful benchmark datasets and standardized evaluation procedures for graph learning.
method Collection and standardization of 120 graph datasets from various applications.
result Standardized evaluation procedures and baseline experiments provided.
Procedure optimizes default thresholds to minimize financial loss in credit risk scenarios.
problem Finding the optimal default threshold to minimize financial loss in loan portfolios.
method Objective comparison and evaluation of default definitions using optimisation procedure.
result Loss minima can exist for a select range of credit risk profiles, suggesting loss optimisation of default thresholds is viable.
In this paper we examine a novel addition to the known methods for learning Bayesian networks from data that improves the quality of the learned networks. Our approach explicitly represents and learns the local structure in the conditional probability tables (CPTs), that quantify these networks. This increases the spac…
Outlier detection is a crucial part of robust evaluation for crowdsourceable assessment of Quality of Experience (QoE) and has attracted much attention in recent years. In this paper, we propose some simple and fast algorithms for outlier detection and robust QoE evaluation based on the nonconvex optimization principle…
Efficiently estimates uncertainty in deep networks.
problem Computational efficiency in uncertainty estimation for deep networks.
method Improved knowledge distillation procedure with novel target distribution and augmentation.
result State-of-the-art uncertainty estimates for in and out-of-distribution samples.
Rogue-Gym tests RL agents' generalization in procedurally generated dungeons.
problem Overfitting in RL agents in different environments.
method Proposes Rogue-Gym, a roguelike game for evaluating RL generalization.
result Some RL enhancements fail to improve generalization in Rogue-Gym.
The paper develops a new model to evaluate policies in complex temporal/spatial experiments.
problem Evaluating the impact of policies in experiments with temporal and spatial dependencies.
method Temporal/spatio-temporal Varying Coefficient Decision Process (VCDP) model, decomposing ATE into DE and IE.
result Effective estimation and inference of DE and IE with rigorous statistical analysis.
Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In thi…
A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
Stochastic models analyze traffic network performance.
problem Evaluate traffic system performance.
method Stochastic cell transmission models, preference functionals, Gaussian process regression.
result Illustrated in two case studies.
Simulation workflow is a top-level model for the design and control of simulation process. It connects multiple simulation components with time and interaction restrictions to form a complete simulation system. Before the construction and evaluation of the component models, the validation of upper-layer simulation work…
Challenge encourages reproducible deep learning methods.
problem Inconsistent training behavior in deep learning competitions.
method Evaluate methods based on their training procedures, retrain in controlled settings.
result Guaranteed reproducibility and generalization of methods.
New method reduces variance in subpopulation model performance estimates.
problem High variance in subpopulation performance metrics for small groups.
method Using an evaluation model to form model-based metric (MBM) estimates.
result MBMs produce more accurate and lower variance estimates for small subpopulations.
StableLM 2 1.6B is a new language model series with detailed evaluations and performance metrics.
problem Improving language models with smaller sizes and better performance.
method Detailed data and training procedure for StableLM 2 1.6B, including zero- and few-shot benchmarks and multilingual evaluations.
result StableLM 2 1.6B is the state-of-the-art open model under 2B parameters.
INT benchmark tests theorem proving agents' ability to generalize to unseen theorems.
problem Evaluating theorem proving agents' ability to generalize to unseen theorems.
method INT benchmark based on a theorem generation and proof procedure with adjustable knobs for measuring 6 types of generalization.
result MCTS can help agents prove new theorems.
Automates debiasing for large language model evaluations through Fisher random walk.
problem Rigorous and scalable evaluation of large language models.
method Semiparametric efficient estimator using Fisher random walk for weighted residual balancing.
result Efficient estimation of contextual preference scores for large language models.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and pro…
The paper proposes criteria and methods for evaluating and aggregating feature-based model explanations.
problem Lack of quantitative evaluation criteria for feature-based model explanations.
method Developed quantitative evaluation criteria (low sensitivity, high faithfulness, low complexity), devised a framework for aggregation, and derived a new aggregate Shapley value explanation function.
result A new aggregate Shapley value explanation function that minimizes sensitivity.
The study assesses ML model robustness under worst-case subpopulations.
problem ML model performance degradation under non-training population.
method Two-stage estimation procedure for evaluating worst-case robustness over subpopulations.
result The method certifies model robustness and prevents unreliable deployments.
Neural model predicts procedure names in stripped binaries.
problem Reverse engineering stripped executables with limited debug information.
method Combines static analysis with neural models to predict procedure names.
result Improves prediction accuracy by 28% and 100% over state-of-the-art models.
In sparse regression modeling via regularization such as the lasso, it is important to select appropriate values of tuning parameters including regularization parameters. The choice of tuning parameters can be viewed as a model selection and evaluation problem. Mallows' Cp type criteria may be used as a tuning param…
Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
Neural autoregressive models are explicit density estimators that achieve state-of-the-art likelihoods for generative modeling. The D-dimensional data distribution is factorized into an autoregressive product of one-dimensional conditional distributions according to the chain rule. Data completion is a more involved ta…
This research introduces a new strategy in cluster ensemble selection by using Independency and Diversity metrics. In recent years, Diversity and Quality, which are two metrics in evaluation procedure, have been used for selecting basic clustering results in the cluster ensemble selection. Although quality can improve …
MLModelScope streamlines ML/DL model evaluation and benchmarking.
problem Challenges in evaluating and benchmarking ML/DL models.
method Open-source, framework/hardware agnostic platform with distributed design.
result Demonstrates the impact of model evaluation pipelines and HW/SW choices.
We quantify content availability and user discovery opportunities in recommender systems.
problem Determining the maximum probability of recommending content to users.
method Stochastic reachability to compute upper bounds on recommendation likelihood.
result Reachability metrics can detect biases and diagnose user discovery limitations.
ECS evaluates synthetic CXR images' distributional fidelity.
problem Evaluating synthetic CXR images' distributional fidelity under privacy constraints.
method Characteristic function transforms of feature embeddings.
result ECS uncovers clinically relevant distributional discrepancies.
Optimizes loan recovery timing across various portfolios.
problem Comparing and evaluating bank's loan recovery decision rules.
method Simulation-based expert system considering time value of money and costs.
result Threshold optima exist across different risk scenarios and portfolio compositions.
The paper evaluates forecast accuracy of realized volatility measures in large cross-sections.
problem Forecast evaluation of realized volatility measures in large cross-sections of financial data.
method Equal predictive accuracy testing procedures, LASSO shrinkage, measurement error correction, cross-sectional jump component measures.
result The augmented HAR model outperforms the standard HAR model in forecasting realized volatility.
We propose the Margin Adaptation for Generative Adversarial Networks (MAGANs) algorithm, a novel training procedure for GANs to improve stability and performance by using an adaptive hinge loss function. We estimate the appropriate hinge loss margin with the expected energy of the target distribution, and derive princi…
Method creates synthetic remote sensing images for robust change detection.
problem Lack of annotated data for change detection in remote sensing.
method Procedural synthesis using game engines for generating realistic synthetic datasets.
result Improves deep learning model performance and convergence with limited real-world data.
Paper assesses uncertainty in safety evaluation of self-driving tech.
problem Estimating uncertainty in safety-critical event probabilities.
method Combines bootstrap method and likelihood ratio scheme for input uncertainty assessment.
result Demonstrates the impact of data variability on safety evaluation estimates.
We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machi…
This paper evaluates a method to improve representations using incomplete external evidence across tasks.
problem Increasing labelled data quality and quantity is challenging due to manual labelling errors and noise.
method Evidence Transfer method using incomplete categorical external evidence.
result Evidence Transfer proves effective and robust against different levels of incompleteness.
Policy evaluation is a crucial step in many reinforcement-learning procedures, which estimates a value function that predicts states' long-term value under a given policy. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evalua…
Paper proposes evaluation methods for climate change illustrations.
problem Lack of metrics to compare realism in conditional generative models.
method Adapted and assessed several existing metrics, including FID.
result FID with Inception-V3 embeddings correlates best with human realism.
Paper improves bootstrapping for off-policy reinforcement learning inference.
problem Improving bootstrapping for off-policy reinforcement learning inference.
method Proposes a bootstrapping FQE method for off-policy statistical inference and a subsampling procedure to improve runtime.
result Asymptotically efficient and distributionally consistent bootstrapping FQE method for off-policy inference.