This post introduces model calibration and evaluation measures, highlighting issues with a common measure.
problem Ensuring model confidence accurately reflects true outcomes.
method Explains common calibration definition, ECE, and its drawbacks.
result New evaluation measures needed for comprehensive model calibration.
Over the last decade there has been increasing concern about the biases embodied in traditional evaluation methods for Natural Language Processing/Learning, particularly methods borrowed from Information Retrieval. Without knowledge of the Bias and Prevalence of the contingency being tested, or equivalently the expecta…
The paper evaluates forecast accuracy of realized volatility measures in large cross-sections.
problem Forecast evaluation of realized volatility measures in large cross-sections of financial data.
method Equal predictive accuracy testing procedures, LASSO shrinkage, measurement error correction, cross-sectional jump component measures.
result The augmented HAR model outperforms the standard HAR model in forecasting realized volatility.
Improves comparison of F-measures for imbalanced datasets.
problem Comparing F-measures for classification algorithms on imbalanced data.
method Two improvements to existing F-measure comparison methods.
result Enhanced accuracy in comparing F-measures for classification.
A personalized learning system needs a large pool of items for learners to solve. When working with a large pool of items, it is useful to measure the similarity of items. We outline a general approach to measuring the similarity of items and discuss specific measures for items used in introductory programming. Evaluat…
This research evaluates measures of dependence for financial time-series data.
problem Accurately preparing time series data and selecting an appropriate measure of dependence is challenging.
method Review and establishment of a comprehensive analysis framework for shaping time-series data and evaluating measures of dependence.
result A method, framework, and example for selecting and evaluating a suitable measure of dependence are presented.
Study proposes worst+gap measure for better DG evaluation.
problem Lack of comprehensive exploration of average measure in DG evaluation.
method Introduced worst+gap measure and compared it with average measure.
result Worst+gap measure provides a more accurate approximation of true DG performance.
The paper critiques and expands on common evaluation metrics in machine learning.
problem The common evaluation metrics like Precision, Recall, F-Measure, and Rand Accuracy are biased and misleading.
method The paper introduces new measures like Informedness, Markedness, and Correlation to better reflect the quality of predictions.
result A system that performs worse in terms of Informedness can appear better using common measures like Precision and Recall.
Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of im…
The paper addresses evaluating survival predictions using discrimination measures, finding a robust method to convert distributions to risks.
problem Evaluating survival distribution predictions with discrimination measures is challenging and often leads to unfair comparisons.
method The paper surveys existing methods and recommends summing over the predicted cumulative hazard as the most robust method to convert distributions to risks.
result Summing over the predicted cumulative hazard is the most robust method to convert distribution predictions to risk predictions.
This paper evaluates and validates cluster results using external and internal evaluation methods.
problem Evaluating and validating the quality of clustering results.
method External evaluation using Homogeneity, Correctness, and V-measure scores; internal evaluation using Silhouette Index and Sum of Square Errors.
result Validation of the number of clusters using dendrogram and statistical frequency distribution.
The paper proposes a dynamic risk measure approach for evaluating defined-contribution pension funds.
problem Periodic evaluation of defined-contribution pension funds to manage risk and improve projections.
method Dynamic risk measure criterion, model-free reinforcement learning, Lee-Carter mortality model.
result Periodic evaluations lead to more risk-averse strategies, while mortality improvements encourage risk-seeking behaviors.
Time series are ubiquitous, and a measure to assess their similarity is a core part of many computational systems. In particular, the similarity measure is the most essential ingredient of time series clustering and classification systems. Because of this importance, countless approaches to estimate time series similar…
This paper tackles deep clustering evaluation challenges in high-dimensional data.
problem Evaluation of deep clustering methods is problematic due to the curse of dimensionality and variations in embedding spaces.
method Develops a theoretical framework to highlight the ineffectiveness of internal validation measures and proposes a systematic approach to applying clustering validity indices in deep learning.
result The proposed framework reduces misguidance from improper use of clustering validity indices in deep learning.
Paper proposes a new method to evaluate joint risk under uncertainty.
problem Evaluating joint risk of multiple insurance risks under dependence uncertainty.
method Axiomatic approach to scalar and vector-valued distortion joint risk measures.
result Established a new scalar distortion joint risk measure with positive homogeneity.
The use of alternative measures to evaluate classifier performance is gaining attention, specially for imbalanced problems. However, the use of these measures in the classifier design process is still unsolved. In this work we propose a classifier designed specifically to optimize one of these alternative measures, nam…
The paper evaluates joint life insurance risk under dependence uncertainty using copulas and convex risk measures.
problem Evaluating risk of joint life insurance products under uncertainty in dependence structure.
method Monotonicity of risk evaluation with concordance order, linear programming for bounds, and numerical analysis.
result Bounds for mean, Value-at-Risk, and Expected Shortfall computed using linear programs.
This paper evaluates forecast quality in electricity markets beyond traditional accuracy measures.
problem Traditional accuracy measures fail to reflect the economic value of electricity price forecasts.
method Investigates four quality dimensions: accuracy, dispersion, association, and extremum identification.
result Dispersion- and association-based measures better capture forecast economic value.
This paper introduces new risk measures for evaluating losses with varying time horizons.
problem Capturing horizon risk and cash non-additivity in risk evaluation.
method Uses BSDEs and shortfall approaches to develop h-generalized shortfall risk measures.
result Introduces hq-entropic risk measures as a new family of fully-dynamic risk measures.
Stochastic optimization problems often involve the expectation in its objective. When risk is incorporated in the problem description as well, then risk measures have to be involved in addition to quantify the acceptable risk, often in the objective. For this purpose it is important to have an adjusted, adapted and eff…
The design of machine learning systems often requires trading off different objectives, for example, prediction error and energy consumption for deep neural networks (DNNs). Typically, no single design performs well in all objectives; therefore, finding Pareto-optimal designs is of interest. The search for Pareto-optim…
The report evaluates heuristics for learning timescale graphical event models.
problem Lack of heuristics for determining hyper-parameters in timescale graphical event models.
method Proposed and evaluated different heuristics for hyper-parameter determination and refined an existing distance measure.
result Conclusions about the applicability of different heuristics on synthetic data.
Generative Adversarial Networks (GANs) are an elegant mechanism for data generation. However, a key challenge when using GANs is how to best measure their ability to generate realistic data. In this paper, we demonstrate that an intrinsic dimensional characterization of the data space learned by a GAN model leads to an…
This research evaluates generalization measures in deep learning.
problem Understanding why deep learning models generalize well despite small training error.
method Empirical evaluation of generalization bounds and measures.
result Generalization measures should be evaluated using distributional robustness.
Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a…
Confidence intervals improve evaluation of binary prediction rules in data mining.
problem Uncertainty in performance measures estimation from finite datasets.
method Asymptotic normal approximations for confidence intervals, with a blurring correction.
result Improved finite sample coverage probabilities and general performance measures inference.
A new measure normalizes clustering accuracy to evaluate algorithms better.
problem Evaluation of clustering algorithms is challenging due to limitations of existing measures.
method Proposes a new, normalised clustering accuracy measure.
result The new measure identifies worst-case scenarios and is more interpretable.
A new measure DCSI quantifies separability for density-based clustering.
problem Quantifying meaningful clusters in data sets.
method Developed a new separability measure DCSI based on separation and connectedness.
result Correctly identifies touching or overlapping classes that do not correspond to meaningful density-based clusters.
A new state-sum formula for the evaluation of the Yang-Mills measure in the Kauffman bracket skein algebra of a closed surface is derived. The formula extends the Kauffman bracket to diagrams that lie in surfaces other than the plane. It also extends Turaev's shadow world invariant of links in a circle bundle over a su…
Efficient medoid-based Silhouette method speeds up clustering evaluation.
problem Difficulty in evaluating clustering results due to many measures and data set dependence.
method Direct optimization of medoid-based Silhouette, combining PAM algorithm improvements.
result 10464x speedup in clustering evaluation for large datasets.
New methods evaluate data representations by complexity of low-loss predictor learning.
problem Evaluating quality of data representations for downstream tasks.
method Surplus Description Length (SDL) and ε Sample Complexity (εSC) methods.
result Methods measure the information needed to approximate optimal predictor up to specified tolerance.
New protocol evaluates synthetic data for temporal consistency.
problem Synthetic data generators can produce invalid timestamps and trajectories.
method Characterize datasets by four properties, then measure timestamp validity and dynamics.
result Temporal fidelity must be measured, not inferred from static data.
Gemini uses inexpensive measurements to correct biases in expensive property evaluations.
problem Accurate estimation of materials properties using expensive measurements is hindered in scientific discovery campaigns.
method Gemini is a data-driven model that corrects systematic biases between property evaluation methods using inexpensive measurements.
result Gemini reduces the number of expensive evaluations needed for Bayesian optimization in materials discovery.
Defining similarity measures is a requirement for some machine learning methods. One such method is case-based reasoning (CBR) where the similarity measure is used to retrieve the stored case or set of cases most similar to the query case. Describing a similarity measure analytically is challenging, even for domain exp…
New measures quantify diversity of latent representations using metric space magnitude.
problem Evaluating the diversity of latent representations in machine learning models.
method Developed magnitude-based measures for latent representations, stable under data perturbations.
result Demonstrated superior performance across various domains and tasks.
Transformer models improve financial sentiment measurement.
problem Capturing nuanced sentiment from financial news articles.
method Transformer-based language models for sentiment classification and aggregation.
result Transformer models outperform traditional dictionary-based methods in sentiment classification.
One often finds in the literature connections between measures of fairness and measures of feature importance employed to interpret trained classifiers. However, there seems to be no study that compares fairness measures and feature importance measures. In this paper we propose ways to evaluate and compare such measure…
This paper presents a novel clustering concept that is based on jointly learned nonlinear transforms (NTs) with priors on the information loss and the discrimination. We introduce a clustering principle that is based on evaluation of a parametric min-max measure for the discriminative prior. The decomposition of the pr…
Unified framework for risk evaluation under uncertainty.
problem Risk assessment under multiple economic scenarios.
method Axiomatic framework for generalized risk measures.
result Characterization of worst-case, coherent, and robust risk measures.
Enhances resilience evaluation by using dynamic convex risk measures.
problem Capturing the full risk profile of financial positions under adverse conditions.
method Introduces a new resilience evaluation method using dynamic convex risk measures.
result Shows that the resilience evaluation can distinguish between positions with the same expected recovery but different conditional risk profiles.
Proposes a new method to rank risky investments based on Omega measure.
problem Evaluating and ranking risky investment projects.
method Introduces an investment certainty equivalence approach and uses the Omega measure.
result Proposed method ranks projects differently from conventional risk-adjusted discount rate (RADR) approach.
Clustering is an essential data mining tool that aims to discover inherent cluster structure in data. For most applications, applying clustering is only appropriate when cluster structure is present. As such, the study of clusterability, which evaluates whether data possesses such structure, is an integral part of clus…
VB-Score evaluates AI systems without ground truth, revealing robustness.
problem Evaluating AI systems without ground truth labels, especially for entity-centric tasks.
method VB-Score uses variance-bounded evaluation, constraint relaxation, and Monte Carlo sampling.
result VB-Score reveals robustness differences not seen by conventional frameworks.
This paper reviews PU learning evaluation methods and provides practical recommendations.
problem Evaluating PU learning methods when only positive and unlabelled data are available.
method Critical review of 51 articles proposing PU classifiers and alternative predictive accuracy measures.
result Practical recommendations for improving PU learning evaluation.
Clustering evaluation measures are frequently used to evaluate the performance of algorithms. However, most measures are not properly normalized and ignore some information in the inherent structure of clusterings. We model the relation between two clusterings as a bipartite graph and propose a general component-based …
CCE improves anomaly detection metrics by measuring both confidence and consistency.
problem Existing anomaly detection metrics lack discriminative power, hyperparameter dependency, and robustness to perturbations.
method CCE uses Bayesian estimation to quantify uncertainty and constructs global and event-level confidence and consistency scores.
result CCE demonstrates strict boundedness, robustness, and linear time complexity.
A hybrid impurity measure balances theoretical soundness and computational efficiency.
problem Developing a robust impurity measure for decision trees.
method Integrates Tsallis entropy with an exponential polarization component.
result Simple parametric measures outperform ITC, but ITC variants are competitive with strong theoretical guarantees.
Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be…