Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
Active testing reduces label costs for efficient model evaluation.
problem Real-world applications require expensive test labels, disconnecting from existing model evaluation methods.
method Derives acquisition strategies to select test points efficiently, addressing label bias and variance.
result Active testing improves model evaluation efficiency without sacrificing accuracy.
Active testing for large language models is made more efficient and accurate.
problem Efficient evaluation of large language models with limited labels.
method Cost-saving measures and in-context learning for constructing a surrogate model.
result Significantly more accurate evaluations of LLM performance compared to random data acquisition.
Study evaluates two-sample tests for validating generative models in high dimensions.
problem Validating the performance and efficiency of non-parametric two-sample tests for high-dimensional generative models.
method Proposes and evaluates the sliced Wasserstein distance, mean of Kolmogorov-Smirnov statistics, and novel sliced Kolmogorov-Smirnov statistic.
result One-dimensional-based tests provide comparable sensitivity to other multivariate metrics but with lower computational cost.
A new test improves feature evaluation in predictive models.
problem Evaluating the usefulness of features in supervised prediction problems.
method Proposes a novel conditional independence test using permutation testing.
result Achieves better predictive power than competing approaches.
New framework tests AVs as a black box, prioritizing rare failure modes.
problem Lack of rigorous and scalable testing methods for AVs.
method Developed a simulation testing framework that learns to identify and rank failure scenarios via adaptive importance-sampling methods.
result First independent evaluation of a full-stack commercial AV system (Comma AI's OpenPilot).
Proposes a modified Morgan-Pitman test for evaluating variances in machine learning models.
problem Limited ability to account for sampling variability in model selection.
method Enhances the classic Morgan-Pitman test for robustness in non-linear models with heavy-tailed distributions or outliers.
result Demonstrates the test's effectiveness and practical utility in model evaluation and selection.
When a feed-forward neural network (FNN) is trained for source ranging in an ocean waveguide, it is difficult evaluating the range accuracy of the FNN on unlabeled test data. A fitting-based early stopping (FEAST) method is introduced to evaluate the range error of the FNN on test data where the distance of source is u…
We present and evaluate the Fast (conditional) Independence Test (FIT) -- a nonparametric conditional independence test. The test is based on the idea that when P(X∣Y,Z)=P(X∣Y), Z is not useful as a feature to predict X, as long as Y is also a regressor. On the contrary, if $P(X \mid Y, Z) \neq P(X…
RENAL test evaluates generative models for time series data.
problem Evaluating generative models for time series data is challenging.
method RENAL test uses recurrent neural networks to transform time series into conditionally independent data pairs for goodness-of-fit testing.
result RENAL test outperforms existing methods in evaluating generative models for time series data.
Two modified tests improve the reliability of evaluating explanation methods.
problem Methodological concerns in evaluating explanation methods for saliency maps.
method Proposed modifications to the Model Parameter Randomisation Test (MPRT): Smooth MPRT and Efficient MPRT.
result Enhanced metric reliability, facilitating more trustworthy deployment of explanation methods.
Research proposes a test case generation system for deep learning models using dataset properties.
problem Automated generation of extensive test cases for deep learning models is challenging.
method Measures dataset quality and proposes a test case generation system guided by dataset properties.
result Systematic test case generation for deep learning models is effective.
New methods needed to evaluate uncertainty estimates in neural networks.
problem Evaluating uncertainty estimates in neural networks is flawed and inconsistent.
method Proposes a simulation-based testing approach to address flaws in current methods.
result Current methods for evaluating uncertainty estimates have significant flaws and cannot accurately compare different methods.
Study evaluates ML methods for two-sample testing with right-censored data.
problem Evaluating ML methods for two-sample testing with right-censored data.
method Developed and compared several ML-based methods with classical tests.
result Proposed methods outperform classical tests in terms of statistical power.
Evaluation and validation of complicated control systems are crucial to guarantee usability and safety. Usually, failure happens in some very rarely encountered situations, but once triggered, the consequence is disastrous. Accelerated Evaluation is a methodology that efficiently tests those rarely-occurring yet critic…
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in or…
Proposes a new framework for evaluating diagnostic models with multiple co-primary endpoints.
problem Overoptimistic assessments of predictive performance in automated medical testing devices.
method Multiple testing framework for diagnostic accuracy studies with co-primary endpoints, using a parametric simultaneous test procedure and Bayesian approach to determine optimal number of models.
result Our approach leads to a better final diagnostic model and increased statistical power.
This paper evaluates test selection methods for deep neural networks, revealing their limitations.
problem Challenges in testing deep learning systems due to high labeling costs.
method Analysis and empirical testing of 11 test selection methods on five datasets.
result Test selection methods can fail under certain conditions, leading to significant drops in test relative coverage.
A new test evaluates risk estimation accuracy using probability integral transform.
problem Measuring the accuracy of financial market risk estimations.
method Probability Integral Transform (PIT) of ex post realized returns against ex ante probability distributions.
result The new test shows the importance of capturing the dynamic of financial markets.
A semiparametric test evaluates instrument validity and complier characteristics.
problem Evaluating the validity of instruments and complier characteristics.
method Semiparametric test, doubly robust moment, machine learning update.
result Validates instrument validity and complier characteristics.
Study evaluates how well question-answering models generalize to new data types.
problem Generalization of question-answering models to new data types.
method Constructed new test sets from different domains and evaluated models' performance.
result Models show significant performance drops when tested on new data types.
Paper proposes a sequential statistical test for comparing imitation learning policies with near-optimal stopping.
problem Challenges in rigorously comparing imitation learning policies due to small sample sizes and potential p-hacking.
method Sequential statistical test that adapts the number of trials based on intermediate results, achieving near-optimal stopping.
result Reduces the number of evaluation trials by up to 32% compared to state-of-the-art baselines, saving significant time and effort.
DP synthetic data may inflate statistical test results, caution advised.
problem Inflated Type I errors in statistical tests on DP-synthetic data.
method Evaluation of Mann-Whitney U test, t-test, chi-squared test, and median test on DP-synthetic data generated from real-world and simulated datasets using various DP-synthetic data generation methods.
result A large portion of evaluation results showed inflated Type I errors, especially at low privacy levels.
Adaptive framework generates challenging adversarial scenarios for autonomous vehicles.
problem Lack of efficient and adaptable evaluation methods for autonomous vehicles.
method Adaptive evaluation framework using ensemble models and nonparametric Bayesian clustering.
result Adversarial scenarios significantly degrade tested autonomous vehicles' performance.
Study improves machine learning models for GI tract disease detection using comprehensive evaluations and cross-dataset testing.
problem Incomplete or incorrect evaluation of machine learning models for GI tract diseases.
method Comprehensive evaluations of five machine learning models using Global Features and Deep Neural Networks, introducing performance hexagons and cross-dataset testing.
result Demonstrates the need for more sophisticated performance metrics and evaluation methods to build generalizable models.
Statistical test evaluates if personalizing interventions is cost-effective.
problem Balancing the benefits of personalizing interventions with their potential costs.
method Developed a statistical hypothesis test to assess the performance of personalized interventions.
result The test shows that personalized interventions can outperform standard approaches under certain conditions.
Suitability filter detects model performance degradation in real-world deployment.
problem Ensuring model reliability in safety-critical domains without access to ground truth labels.
method Uses suitability signals to evaluate classifier performance on unlabeled user data.
result The suitability filter reliably detects performance deviations due to covariate shift.
IRT improves algorithm evaluation across datasets.
problem Evaluating the performance of algorithm portfolios.
method Modified IRT framework for evaluating algorithm portfolios across datasets.
result Richer characteristics of algorithm performance are revealed.
The paper proposes a new evaluation framework for causal inference models.
problem Challenges in estimating causal effects from observational data.
method Complements evaluation of causal inference models with statistical evidence and non-parametric tests.
result Eliminates the influence of a few instances or simulations on benchmarking results.
A new method reduces hyperparameter tuning evaluations by using sequential tests.
problem Time-consuming hyperparameter tuning in machine learning.
method Sequential Random Search (SQRS) extending regular random search.
result SQRS finds similarly well-performing parameter settings with fewer evaluations.
Paper proposes a unified framework for evaluating calibration of probabilistic models.
problem Evaluation of calibration for general probabilistic predictive models.
method Unified framework for calibration evaluation and tests for any probabilistic model.
result Generalization and reformulation of existing measures and tests.
Survey of Machine Learning Testing: Properties, Components, and Trends.
problem Challenges in testing machine learning models.
method Comprehensive review of 144 ML testing papers.
result Identification of research challenges and directions.
Weighted log-rank tests are arguably the most widely used tests by practitioners for the two-sample problem in the context of right-censored data. Many approaches have been considered to make weighted log-rank tests more robust against a broader family of alternatives, among them, considering linear combinations of wei…
A new protocol evaluates small machine learning improvements conservatively.
problem Uncertainty in small gains reported in machine learning papers.
method Paired bootstrap protocol with BCa confidence intervals and sign-flip permutation tests.
result Conservative evaluation reduces over-claiming of small improvements.
Embeddings of lab test codes improve mortality prediction and preserve ordinality.
problem Improving mortality prediction using lab test embeddings.
method Training embeddings for LOINC codes and their concatenations with abnormality symbols, evaluating performance on mortality prediction tasks.
result Embeddings of lab test codes improve mortality prediction and preserve ordinality.
Recently, adversarial deception becomes one of the most considerable threats to deep neural networks. However, compared to extensive research in new designs of various adversarial attacks and defenses, the neural networks' intrinsic robustness property is still lack of thorough investigation. This work aims to qualitat…
Corrects bias in LLM-as-a-judge evaluations using adaptive calibration.
problem Bias in LLM evaluations due to imperfect sensitivity and specificity.
method Plug-in framework with confidence intervals accounting for test and calibration dataset uncertainties.
result LML-based evaluation yields more reliable estimates than human-only evaluation.
Developing state-of-the-art approaches for specific tasks is a major driving force in our research community. Depending on the prestige of the task, publishing it can come along with a lot of visibility. The question arises how reliable are our evaluation methodologies to compare approaches? One common methodology to i…
LCIT tests conditional independence using latent representations.
problem Detecting conditional independencies in statistical and machine learning tasks.
method Generative framework for learning latent representations of target variables X and Y, then testing for remaining dependencies.
result LCIT outperforms state-of-the-art baselines consistently under different metrics and settings.
This paper tackles sandbagging in AI safety evaluations.
problem AI agents may hide dangerous capabilities to avoid being deactivated.
method Developed a simple model of strategic deception in sequential decision-making tasks.
result Demonstrated that optimal rational agents exhibit sandbagging behavior.
We conduct an extensive evaluation of price jump tests based on high-frequency financial data. After providing a concise review of multiple alternative tests, we document the size and power of all tests in a range of empirically relevant scenarios. Particular focus is given to the robustness of test performance to the …
This paper provides a new unimodality test with application in hierarchical clustering methods. The proposed method denoted by signature test (Sigtest), transforms the data based on its statistics. The transformed data has much smaller variation compared to the original data and can be evaluated in a simple proposed un…
GRETEL unifies GCE evaluation across various settings.
problem Lack of standardized evaluation for Graph Counterfactual Explanations.
method Unified framework for testing GCE methods in diverse settings.
result GRETEL promotes reproducible evaluations of GCE techniques.
The Restricted Boltzmann Machines (RBM) can be used either as classifiers or as generative models. The quality of the generative RBM is measured through the average log-likelihood on test data. Due to the high computational complexity of evaluating the partition function, exact calculation of test log-likelihood is ver…
Language grounded image understanding tasks have often been proposed as a method for evaluating progress in artificial intelligence. Ideally, these tasks should test a plethora of capabilities that integrate computer vision, reasoning, and natural language understanding. However, rather than behaving as visual Turing t…
Kernel test evaluates dynamical system data streams.
problem Evaluate if data streams from dynamical systems are from the same distribution.
method Proposes a novel kernel two-sample test for dynamical systems, addressing independence and autocorrelation challenges.
result Data-driven method with theoretical guarantees for anomaly detection.
This paper evaluates targeted data poisoning attacks by focusing on the hardest samples, improving evaluation and defense strategies.
problem The effectiveness of targeted data poisoning attacks is often overestimated due to average evaluation methods.
method The paper introduces metrics to identify the hardest and easiest to poison samples based on clean model information.
result The proposed metrics reliably stratify samples by poisoning vulnerability, enabling rigorous worst-case evaluation and proactive defense.
New framework for interpreting disaggregated fairness evaluations using causal models.
problem Misinterpretation of disaggregated fairness evaluations due to data representativeness and selection bias.
method Causal graphical models to characterize fairness properties and metric stability under different data generating processes.
result Disaggregated evaluations are unreliable without explicit assumptions regarding bias mechanisms.