ModelRadar evaluates forecasting models across multiple aspects.
problem Evaluating forecasting models using single scores hides relevant performance variations.
method ModelRadar, a framework for aspect-based evaluation of univariate time series forecasting models.
result NHITS performs best overall but its superiority varies with forecasting conditions.
Due to the increased availability of online reviews, sentiment analysis had been witnessed a booming interest from the researchers. Sentiment analysis is a computational treatment of sentiment used to extract and understand the opinions of authors. While many systems were built to predict the sentiment of a document or…
Deep learning systems thrive on abundance of labeled training data but such data is not always available, calling for alternative methods of supervision. One such method is expectation regularization (XR) (Mann and McCallum, 2007), where models are trained based on expected label proportions. We propose a novel applica…
Graph-based semi-supervised learning for aspect term extraction.
problem Aspect term detection and sentiment analysis in reviews.
method Label spreading algorithm on graph sparsified by kNN for efficient classification.
result The approach achieves good results with limited labeled data.
Adversarial training improves BERT for ABSA tasks.
problem Lack of labeled data for ABSA.
method Adversarial training applied to BERT-PT model.
result BAT model outperforms post-trained BERT.
A new framework evaluates deep learning vs classical forecasting methods for time series predictions.
problem Current forecasting model evaluation metrics fail to capture model performance differences.
method Proposes a novel framework for evaluating univariate time series forecasting models from multiple perspectives.
result Deep learning models like NHITS outperform classical methods in multi-step ahead forecasting but not in anomaly handling.
MCU-Net combines U-Net and Monte Carlo Dropout for uncertainty in medical image segmentation.
problem Lack of uncertainty representation in deep learning methods for patient-centered healthcare decisions.
method MCU-Net framework using U-Net and Monte Carlo Dropout with four uncertainty metrics.
result MCU-Net maximizes automated performance and refers truly uncertain cases.
Hybrid Deep Embedding for aspect-level explanations in recommendations.
problem Challenges in personalization, dynamic explanations, and aspect-level granularity in recommendation systems.
method Proposes Hybrid Deep Embedding (HDE) to learn dynamic embeddings for user and item preferences, and aspect-level quality vectors.
result Demonstrates improved recommending performance and dynamic aspect-level explanations.
Proposes a method for ranking items across multiple aspects based on user feedback.
problem No principled solution exists for generating multiple item rankings over different aspects.
method Developed a directional multi-aspect ranking criterion using probabilistic multivariate tensor factorization.
result Demonstrated effectiveness of the proposed method through comprehensive experiments on real datasets.
System detects financial forecasts in tweets, achieving high precision.
problem Detecting financial forecasts in social media messages.
method Natural Language Processing and Machine Learning techniques for real-time analysis.
result Achieves over 90% precision for financial forecasts.
For large-scale industrial processes under closed-loop control, process dynamics directly resulting from control action are typical characteristics and may show different behaviors between real faults and normal changes of operating conditions. However, conventional distributed monitoring approaches do not consider the…
A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
Develops a method to evaluate OPE robustness to hyperparameters and policies.
problem Difficulty in selecting and tuning OPE estimators due to limited experimental evaluations.
method Introduces IEOE (Interpretable Evaluation for Offline Evaluation) to assess robustness.
result Demonstrates improved evaluation of OPE estimators' reliability.
This paper evaluates and validates cluster results using external and internal evaluation methods.
problem Evaluating and validating the quality of clustering results.
method External evaluation using Homogeneity, Correctness, and V-measure scores; internal evaluation using Silhouette Index and Sum of Square Errors.
result Validation of the number of clusters using dendrogram and statistical frequency distribution.
Optimizes crowdsourced preference-based subjective evaluation with online learning.
problem Large-scale evaluation of generative media using crowdsourcing due to combinatorial explosion.
method Automatic optimization of pair combination selections and evaluation volumes with online learning.
result Optimizes evaluation by reducing pair combinations and allocating optimal evaluation volumes.
We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the context of generating online product reviews. We compare human-based evaluators with a variety of automated evaluation procedures, including discriminative evaluators that measure how well machi…
Our research extends the Bilingual Evaluation Understudy (BLEU) evaluation technique for statistical machine translation to make it more adjustable and robust. We intend to adapt it to resemble human evaluation more. We perform experiments to evaluate the performance of our technique against the primary existing evalua…
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Cramming method evaluates learned policies from contextual bandits efficiently.
problem Evaluating final learned policies from contextual bandit algorithms.
method On-policy evaluation using a single pass of data, ensuring consistency and asymptotic normality.
result Cramming method reduces evaluation standard error by approximately 40% compared to off-policy methods.
Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
The study addresses biases in evaluating molecular optimization methods and proposes methods to reduce these biases.
problem Biases in in silico evaluation of molecular optimization methods.
method Discussion and empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
result Empirical investigation of bias reduction methods for predictor misspecification and sample reuse.
Paper addresses off-policy evaluation and learning with covariate shift.
problem Evaluating and training a new policy using historical data with a covariate shift.
method Derives efficiency bounds and proposes doubly robust estimators for OPE and OPL under covariate shift.
result Proposes estimators for off-policy evaluation and learning under covariate shift.
Proposes clustering as a new evaluation method for clinical knowledge embedding.
problem Traditional Link Prediction evaluation protocol loses information and harms model accuracy.
method Proposes Clustering Evaluation Protocol as an alternative.
result Experimental results show the proposed protocol can potentially replace Link Prediction.
AXE evaluates explanations to avoid misleading Rashomon set model selection.
problem Evaluating explanations for Rashomon set models to avoid false selection.
method Proposed AXE method to evaluate explanation quality.
result AXE detects adversarial fairwashing with 100% success rate.
We consider evaluation methods for payoffs with an inherent financial risk as encountered for instance for portfolios held by pension funds and insurance companies. Pricing such payoffs in a way consistent to market prices typically involves combining actuarial techniques with methods from mathematical finance. We prop…
New method for evaluating LLMs reduces bias in open-ended evaluations.
problem Bias in Elo-based ratings of LLMs due to data redundancies.
method Proposes evaluation as a 3-player game and introduces novel solution concepts.
result Novel method leads to more robust and intuitive ratings.
Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accident…
New method improves consistency of reinforcement learning performance evaluations.
problem Inconsistent performance results in reinforcement learning due to flawed evaluation metrics.
method Proposes a new comprehensive evaluation methodology for reinforcement learning algorithms.
result Demonstrates improved reliability of performance measurements for reinforcement learning algorithms.
This paper considers the challenge of evaluating a set of classifiers, as done in shared task evaluations like the KDD Cup or NIST TREC, without expert labels. While expert labels provide the traditional cornerstone for evaluating statistical learners, limited or expensive access to experts represents a practical bottl…
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
Combines curvature descriptors with TDA for graph model evaluation.
problem Evaluating graph generative models efficiently and accurately.
method Combines graph curvature descriptors with topological data analysis.
result Robust, expressive descriptors for graph generative models.
We present and prove properties of a new offline policy evaluator for an exploration learning setting which is superior to previous evaluators. In particular, it simultaneously and correctly incorporates techniques from importance weighting, doubly robust evaluation, and nonstationary policy evaluation approaches. In a…
Paper tackles efficient evaluation of natural stochastic policies in offline RL.
problem Efficiency issues in evaluating natural stochastic policies due to unknown evaluation policy.
method Derive efficiency bounds for tilting and modified treatment policies, propose nonparametric estimators.
result Proposed estimators attain efficiency bounds under lax conditions and enjoy partial double robustness.
CONE evaluates treatment assignment functions using networked observational data to mitigate hidden confounding bias.
problem Evaluate treatment assignment functions using networked observational data with hidden confounders.
method CONE framework that learns partial representations of latent confounders and combines them for counterfactual evaluation.
result Network information mitigates hidden confounding bias in counterfactual evaluation.
MLModelScope streamlines ML/DL model evaluation and benchmarking.
problem Challenges in evaluating and benchmarking ML/DL models.
method Open-source, framework/hardware agnostic platform with distributed design.
result Demonstrates the impact of model evaluation pipelines and HW/SW choices.
Study evaluates saliency maps on artificial data with different backgrounds.
problem Objective evaluation of saliency methods on artificial data with varying backgrounds.
method Developed a framework to generate artificial data with synthetic lesions and a known ground truth map, evaluated two data sets with different backgrounds (Perlin noise and 2D brain MRI slices).
result Heatmaps vary strongly between saliency methods and backgrounds.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
A new method for faster multi-objective optimization by evaluating objectives separately.
problem Finding the Pareto front of trade-offs between multiple objectives efficiently.
method Knowledge Gradient with decoupled evaluations, accounting for different costs.
result The method significantly outperforms existing approaches in terms of evaluation cost.
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
This work shows how evaluation metrics can be seen as fair gambles.
problem The relationship and evaluation of machine learning forecasts.
method Using game-theoretic probability, the authors show evaluation metrics as fair gambles.
result Standard evaluation metrics are fair gambler outcomes, with calibration and regret metrics on two dimensions.
SureMap estimates model performance across subpopulations efficiently.
problem Estimating model performance across subpopulations with scarce data.
method Simultaneous Gaussian mean estimation with external data.
result High accuracy in both multi-task and single-task disaggregated evaluations.
Murakami-Ohtsuki-Yamada introduced an evaluation of certain oriented planar trivalent graphs with colored edges. This evaluation plays a key role in the evaluation of the colored HOMFLY polynomial of a link in 3-space and its Khovanov-Rozansky categorification. Our goal is is to give a generating series formula for the…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
Corrects bias in LLM-as-a-judge evaluations using adaptive calibration.
problem Bias in LLM evaluations due to imperfect sensitivity and specificity.
method Plug-in framework with confidence intervals accounting for test and calibration dataset uncertainties.
result LML-based evaluation yields more reliable estimates than human-only evaluation.
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Mark…
New method extends fitted Q-evaluation for distributional off-policy reinforcement learning.
problem Estimating return distribution in reinforcement learning using offline data.
method Developed a set of guiding principles and new FDE methods with theoretical justification.
result FDE methods outperform existing approaches in simulations and real-world games.
Geospatial ML models need special evaluation methods due to their unique challenges.
problem Evaluating geospatial machine learning models is challenging due to their specific characteristics.
method Delineated unique challenges and proposed concrete takeaways for improving geospatial model evaluations.
result Concrete takeaways for improving evaluations of geospatial model performance.