New method improves consistency of reinforcement learning performance evaluations.
problem Inconsistent performance results in reinforcement learning due to flawed evaluation metrics.
method Proposes a new comprehensive evaluation methodology for reinforcement learning algorithms.
result Demonstrates improved reliability of performance measurements for reinforcement learning algorithms.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
IRT improves algorithm evaluation across datasets.
problem Evaluating the performance of algorithm portfolios.
method Modified IRT framework for evaluating algorithm portfolios across datasets.
result Richer characteristics of algorithm performance are revealed.
Evaluates change point detection algorithms on real-world data.
problem Insufficient evaluation of change point detection algorithms on real-world time series.
method Developed a data set of 37 time series from various domains, annotated by human experts, and evaluated 14 algorithms using consistency metrics.
result Demonstrates the need for better evaluation methods in change point detection.
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
New evaluation methods for causal models improve algorithm performance assessment.
problem Primitive evaluation techniques limit understanding of causal modeling algorithms.
method Use interventional measures and empirical data for evaluation.
result Evaluation techniques produce significantly different results than structural measures.
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical…
Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. \emph{Offline} evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "p…
Greedy algorithms which use only function evaluations are applied to convex optimization in a general Banach space X. Along with algorithms that use exact evaluations, algorithms with approximate evaluations are treated. A priori upper bounds for the convergence rate of the proposed algorithms are given. These bounds…
Generates multimodal safety-critical scenarios for robustness evaluation of decision-making algorithms.
problem Lack of comprehensive evaluation of neural network robustness under real-world scenarios.
method Proposes a flow-based multimodal scenario generator using weighted likelihood maximization and gradient-based sampling.
result Demonstrates improved testing efficiency and multimodal modeling capability compared to traditional methods.
Do-AIQ framework evaluates AI algorithms' quality using DOE.
problem Quality evaluation of AI mislabel detection algorithms.
method Design-of-experiment approach with high-dimensional constraint space design and surrogate modeling.
result Established framework for evaluating AI algorithm quality robustly.
Backward exploration reduces sample complexity in policy evaluation.
problem Empirical policy evaluation in reinforcement learning.
method Backward exploration algorithms from high-cost states.
result Reduced average-case sample complexity to O(logS). The paper introduces metrics to evaluate NILM algorithms' performance on unseen buildings.
problem Assessing NILM algorithms' performance on new, unseen buildings.
method Developed several metrics to evaluate NILM algorithms' generalization ability.
result Demonstrated the utility of the proposed metrics through two case studies.
We present and implement two algorithms for analytic asymptotic evaluation of the marginal likelihood of data given a Bayesian network with hidden nodes. As shown by previous work, this evaluation is particularly hard for latent Bayesian network models, namely networks that include hidden variables, where asymptotic ap…
Study improves off-policy evaluation from non-i.i.d. bandit samples.
problem Improving off-policy evaluation from non-independent bandit samples.
method Constructing an estimator from a standardized martingale difference sequence.
result Proposed estimator performs better than existing methods.
This paper evaluates algorithms for classification and outlier detection accuracies in temporal data. We focus on algorithms that train and classify rapidly and can be used for systems that need to incorporate new data regularly. Hence, we compare the accuracy of six fast algorithms using a range of well-known time-ser…
New algorithms improve policy evaluation in reinforcement learning.
problem Off-policy stability and on-policy efficiency issues in policy evaluation.
method Introduced novel algorithms using oblique projection method.
result Demonstrated both off-policy stability and on-policy efficiency.
Optimizes crowdsourced preference-based subjective evaluation with online learning.
problem Large-scale evaluation of generative media using crowdsourcing due to combinatorial explosion.
method Automatic optimization of pair combination selections and evaluation volumes with online learning.
result Optimizes evaluation by reducing pair combinations and allocating optimal evaluation volumes.
Paper applies NEAT for dynamic credit evaluation using streaming data.
problem Dynamic credit evaluation using streaming data.
method Neuroevolution of Augmenting Topologies (NEAT) with enhancements.
result NEAT effectively handles dynamic credit evaluation with streaming data.
PS-BAX uses posterior sampling to select evaluation points for efficient Bayesian algorithm execution.
problem Efficiently selecting evaluation points for expensive functions with limited evaluations.
method Posterior sampling to guide sequential selection of evaluation points.
result PS-BAX is faster, simpler, and more scalable than existing methods.
A new framework evaluates HTE estimators using relative error.
problem Lack of robust evaluation methods for HTE estimators.
method Proposes a relative error-based evaluation framework and neural network architecture to estimate nuisance parameters and robustly compare HTE estimators.
result Demonstrates reliable comparisons and improved HTE estimation through the proposed framework and learning algorithm.
We model and correct bias in sequential evaluation, improving ranking accuracy.
problem Sequential evaluation bias in online, irrevocable scoring.
method Modeling the rating process, posing as statistical inference, proposing an online algorithm.
result Near-linear time, online algorithm with guarantees in ranking metrics, information theoretically optimal.
This study evaluates clustering algorithms on high-dimensional data.
problem Comparing clustering algorithms on high-dimensional datasets.
method Evaluation of K-means, DBSCAN, and Spectral Clustering using PCA, t-SNE, UMAP, and multiple metrics.
result UMAP preprocessing improves clustering quality across all algorithms, with Spectral Clustering excelling.
In machine learning, the choice of a learning algorithm that is suitable for the application domain is critical. The performance metric used to compare different algorithms must also reflect the concerns of users in the application domain under consideration. In this work, we propose a novel probability-based performan…
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
Study limits of testing algorithms without assumptions, finding key performance bounds.
problem Understanding algorithm performance without making assumptions.
method Exploring fundamental limits of algorithm performance testing.
result Hardness results for evaluating algorithm performance and comparing multiple algorithms.
The paper proposes a technique to speed up evolutionary algorithms by using lower-cost approximations of the objective function.
problem Evolutionary algorithms require many evaluations to solve computationally expensive black-box optimization problems.
method The paper introduces a technique to choose an appropriate approximate function cost during the execution of the optimization algorithm.
result The proposed approach can reach the same objective value in less than half the time in certain cases.
Paper proposes a unified sparsity-based framework for evaluating algorithmic fairness.
problem Ensuring fairness in machine learning across diverse domains.
method Unified sparsity-based framework for evaluating fairness.
result Demonstrates broad applicability and effectiveness of the framework.
In real-world applications of reinforcement learning (RL), noise from inherent stochasticity of environments is inevitable. However, current policy evaluation algorithms, which plays a key role in many RL algorithms, are either prone to noise or inefficient. To solve this issue, we introduce a novel policy evaluation a…
ExperienceThinking optimizes hyperparameters quickly with smart pruning and knowledge use.
problem Efficiently optimizing hyperparameters in machine learning with limited evaluations.
method Two novel methods: search space pruning and knowledge utilization.
result ExperienceThinking outperforms classical algorithms in few evaluations.
Certified algorithms optimize functions with varying costs, providing error bounds.
problem Optimizing functions with varying evaluation costs and error bounds.
method Formalized as a min-max game, proposed certified MFDOO algorithm with cost complexity bound.
result Proposed certified MFDOO algorithm has near-optimal cost complexity for Lipschitz functions.
This paper provides lower bounds on the convergence rate of Derivative Free Optimization (DFO) with noisy function evaluations, exposing a fundamental and unavoidable gap between the performance of algorithms with access to gradients and those with access to only function evaluations. However, there are situations in w…
Discovery of causal relations from observational data is essential for many disciplines of science and real-world applications. However, unlike other machine learning algorithms, whose development has been greatly fostered by a large amount of available benchmark datasets, causal discovery algorithms are notoriously di…
Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effect…
A new algorithm optimizes time-varying functions with non-constant evaluation times.
problem Optimizing functions that change over time with varying evaluation times.
method Proposes a novel time-varying Bayesian optimization algorithm.
result Establishes a regret bound for the proposed algorithm.
New ranking algorithms are continually being developed and refined, necessitating the development of efficient methods for evaluating these rankers. Online ranker evaluation focuses on the challenge of efficiently determining, from implicit user feedback, which ranker out of a finite set of rankers is the best. Online …
Survey on RL reproducibility using real-world robots.
problem Difficulty in reproducing RL results due to algorithm variance, environment stochasticity, and hyper-parameters.
method Investigates issues leading to irreproducible RL research and standardizes evaluation approach.
result Shows how to manage and standardize evaluation of RL algorithms for unbiased comparison.
In reinforcement learning (RL) , one of the key components is policy evaluation, which aims to estimate the value function (i.e., expected long-term accumulated reward) of a policy. With a good policy evaluation method, the RL algorithms will estimate the value function more accurately and find a better policy. When th…
NAS helps find best neural network designs.
problem Designing optimal neural network architectures.
method Optimization algorithms and search spaces.
result Introduction to major advances in NAS for CNNs.
Paper uses LightGBM for mobile user credit assessment.
problem Improving credit evaluation methods for communication operators.
method Data preprocessing, feature engineering, multiple machine learning models integration.
result Established a suitable fusion model for operator user credit evaluation.
New methods for evaluating and optimizing policies in offline RL with unobserved confounders.
problem Evaluating and optimizing policies in the presence of unobserved confounders.
method Characterized settings and algorithms for consistent value estimates and lower bounds, with sample complexity guarantees.
result Proved local convergence guarantees for offline policy improvement.
We present the first differentially private algorithms for reinforcement learning, which apply to the task of evaluating a fixed policy. We establish two approaches for achieving differential privacy, provide a theoretical analysis of the privacy and utility of the two algorithms, and show promising results on simple e…
This work benchmarks MARL algorithms in cooperative tasks.
problem Lack of evaluation tasks and criteria for comparing MARL algorithms.
method Systematic evaluation of three MARL algorithm classes in diverse cooperative tasks.
result Insights into the effectiveness of different learning approaches.
We present a method to stop the evaluation of a decision making process when the result of the full evaluation is obvious. This trait is highly desirable for online margin-based machine learning algorithms where a classifier traditionally evaluates all the features for every example. We observe that some examples are e…
Study compares DSPy teleprompter algorithms for aligning LLM evaluations with human annotations.
problem Aligning LLM evaluation metrics with human annotations.
method Comparative analysis of five teleprompter algorithms within the DSPy framework.
result Certain teleprompters outperform others in detecting hallucinations.
Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
This paper reviews and proposes a new approach for evaluating internal cluster validation indices.
problem Selecting the best-performing unsupervised classification algorithm without external information.
method Examines and proposes a new evaluation approach for internal validation indices.
result Suggests a new evaluation approach for internal validation indices.
We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations base…