This paper investigates task-level evaluation in few-shot learning models.
problem The reliability of evaluating and tuning models trained for individual tasks in few-shot learning is not well addressed.
method The paper measures accuracy of performance estimators, considers model selection strategies, and examines the reasons for evaluator failure.
result Cross-validation with a low number of folds is best for estimating model performance, while large number of folds is better for model selection.
Hierarchical framework for model evaluation on leaderboards
problem Uncertainty and variability in model performance across tasks
method Hierarchical framework with task-level and leaderboard-level rank prediction intervals
result Statistically valid and informative model rank intervals
This paper improves meta-learning by developing new PAC-Bayes bounds.
problem Meta-learning generalization gap across multiple tasks.
method Upper bounding convex functions linking environment and task-level losses.
result New PAC-Bayes bounds for meta-learning with improved algorithms.
Statistical test detects model degradations in optimized language models.
problem Detecting model degradations in optimized language models.
method Statistical hypothesis testing framework based on McNemar's test.
result Even small accuracy degradations (0.3%) can be attributed to actual degradations, not noise.
FedGTEA learns new tasks in federated learning with task embeddings and alignment.
problem Federated class-incremental learning with task-specific knowledge and model uncertainty.
method Cardinality-Agnostic Task Encoder (CATE) for Gaussian task embeddings, 2-Wasserstein distance for inter-task alignment.
result FedGTEA achieves superior classification performance and mitigates forgetting.
Traditionally, multitask learning (MTL) assumes that all the tasks are related. This can lead to negative transfer when tasks are indeed incoherent. Recently, a number of approaches have been proposed that alleviate this problem by discovering the underlying task clusters or relationships. However, they are limited to …
Quantization-aware training can recover accuracy lost by post-training quantization.
problem Post-training quantization (PTQ) can fail sharply at aggressive bitwidths.
method A unified geometric framework that explains PTQ failure and QAT recovery.
result QAT has a useful bias that steers iterates back into the basin.
Current end-to-end deep Reinforcement Learning (RL) approaches require jointly learning perception, decision-making and low-level control from very sparse reward signals and high-dimensional inputs, with little capability of incorporating prior knowledge. This results in prohibitively long training times for use on rea…
Pretrained time-series models outperform train-from-scratch baselines in financial return forecasting.
problem Financial return forecasting
method Pretrained time-series foundation models
result Pretrained TSFMs dominate the ranking distribution, accounting for 8 of 10 task-level wins.
Unified framework improves meta-learning generalization bounds.
problem Limited sharpness of existing meta-generalization bounds.
method Unified information-theoretic derivation for single-step bounds.
result Unified bounds exhibit tighter scaling and computational advantages.
New framework estimates demand responses across multiple contexts with limited price variation.
problem Estimating heterogeneous linear price-response functions across multiple contexts with limited price variation and confounding.
method Meta-learning framework that identifies conditional mean of task-specific causal demand parameters given a subset of task-specific observables.
result Improved recovery of demand responses relative to standard transfer-learning baselines.
RLMM extends psychometric models to larger tasks.
problem Sequential process data from interactive assessments are not well handled by conventional models.
method RLMM decouples person-level choice sensitivity from task-level value representation through a shared parametric action-value function.
result RLMM achieves higher estimation accuracy and lower runtime than MDP-MM in peg-solitaire simulations and AQUALAB gameplay logs.
LLM sandbox and persona dynamics create unethical reality gaps that shift risk to users.
problem Ethical issues arise from LLMs generating reality gaps that shift risk to uninformed users.
method Analyzes the ethical implications of LLM sandbox and persona dynamics, comparing them to financial regulation and compliance.
result Active generation of reality gaps is unethical as it shifts epistemic risk to users.
A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
Develops a method to evaluate OPE robustness to hyperparameters and policies.
problem Difficulty in selecting and tuning OPE estimators due to limited experimental evaluations.
method Introduces IEOE (Interpretable Evaluation for Offline Evaluation) to assess robustness.
result Demonstrates improved evaluation of OPE estimators' reliability.
This paper evaluates and validates cluster results using external and internal evaluation methods.
problem Evaluating and validating the quality of clustering results.
method External evaluation using Homogeneity, Correctness, and V-measure scores; internal evaluation using Silhouette Index and Sum of Square Errors.
result Validation of the number of clusters using dendrogram and statistical frequency distribution.
Optimizes crowdsourced preference-based subjective evaluation with online learning.
problem Large-scale evaluation of generative media using crowdsourcing due to combinatorial explosion.
method Automatic optimization of pair combination selections and evaluation volumes with online learning.
result Optimizes evaluation by reducing pair combinations and allocating optimal evaluation volumes.
We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machi…
Our research extends the Bilingual Evaluation Understudy (BLEU) evaluation technique for statistical machine translation to make it more adjustable and robust. We intend to adapt it to resemble human evaluation more. We perform experiments to evaluate the performance of our technique against the primary existing evalua…
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
The study addresses biases in evaluating molecular optimization methods and proposes methods to reduce these biases.
problem Biases in in silico evaluation of molecular optimization methods.
method Discussion and empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
result Empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
AXE evaluates explanations to avoid misleading Rashomon set model selection.
problem Evaluating explanations for Rashomon set models to avoid false selection.
method Proposed AXE method to evaluate explanation quality.
result AXE detects adversarial fairwashing with 100% success rate.
We consider evaluation methods for payoffs with an inherent financial risk as encountered for instance for portfolios held by pension funds and insurance companies. Pricing such payoffs in a way consistent to market prices typically involves combining actuarial techniques with methods from mathematical finance. We prop…
New method for evaluating LLMs reduces bias in open-ended evaluations.
problem Bias in Elo-based ratings of LLMs due to data redundancies.
method Proposes evaluation as a 3-player game and introduces novel solution concepts.
result Novel method leads to more robust and intuitive ratings.
Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accident…
New method improves consistency of reinforcement learning performance evaluations.
problem Inconsistent performance results in reinforcement learning due to flawed evaluation metrics.
method Proposes a new comprehensive evaluation methodology for reinforcement learning algorithms.
result Demonstrates improved reliability of performance measurements for reinforcement learning algorithms.
This paper considers the challenge of evaluating a set of classifiers, as done in shared task evaluations like the KDD Cup or NIST TREC, without expert labels. While expert labels provide the traditional cornerstone for evaluating statistical learners, limited or expensive access to experts represents a practical bottl…
Combines curvature descriptors with TDA for graph model evaluation.
problem Evaluating graph generative models efficiently and accurately.
method Combines graph curvature descriptors with topological data analysis.
result Robust, expressive descriptors for graph generative models.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In a…
Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
Study evaluates saliency maps on artificial data with different backgrounds.
problem Objective evaluation of saliency methods on artificial data with varying backgrounds.
method Developed a framework to generate artificial data with synthetic lesions and a known ground truth map, evaluated two data sets with different backgrounds (Perlin noise and 2D brain MRI slices).
result Heatmaps vary strongly between saliency methods and backgrounds.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
A new method for faster multi-objective optimization by evaluating objectives separately.
problem Finding the Pareto front of trade-offs between multiple objectives efficiently.
method Knowledge Gradient with decoupled evaluations, accounting for different costs.
result The method significantly outperforms existing approaches in terms of evaluation cost.
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
This work shows how evaluation metrics can be seen as fair gambles.
problem The relationship and evaluation of machine learning forecasts.
method Using game-theoretic probability, the authors show evaluation metrics as fair gambles.
result Standard evaluation metrics are fair gambler outcomes, with calibration and regret metrics on two dimensions.
Murakami-Ohtsuki-Yamada introduced an evaluation of certain oriented planar trivalent graphs with colored edges. This evaluation plays a key role in the evaluation of the colored HOMFLY polynomial of a link in 3-space and its Khovanov-Rozansky categorification. Our goal is is to give a generating series formula for the…
SureMap estimates model performance across subpopulations efficiently.
problem Estimating model performance across subpopulations with scarce data.
method Simultaneous Gaussian mean estimation with external data.
result High accuracy in both multi-task and single-task disaggregated evaluations.
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
Corrects bias in LLM-as-a-judge evaluations using adaptive calibration.
problem Bias in LLM evaluations due to imperfect sensitivity and specificity.
method Plug-in framework with confidence intervals accounting for test and calibration dataset uncertainties.
result LML-based evaluation yields more reliable estimates than human-only evaluation.
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Mark…
New method extends fitted Q-evaluation for distributional off-policy reinforcement learning.
problem Estimating return distribution in reinforcement learning using offline data.
method Developed a set of guiding principles and new FDE methods with theoretical justification.
result FDE methods outperform existing approaches in simulations and real-world games.
Learning knowledge representation is an increasingly important technology applicable in many domain-specific machine learning problems. We discuss the effectiveness of traditional Link Prediction or Knowledge Graph Completion evaluation protocol when embedding knowledge representation for categorised multi-relational d…
Geospatial ML models need special evaluation methods due to their unique challenges.
problem Evaluating geospatial machine learning models is challenging due to their specific characteristics.
method Delineated unique challenges and proposed concrete takeaways for improving geospatial model evaluations.
result Concrete takeaways for improving evaluations of geospatial model performance.