The paper critiques current time series classification evaluation methods.
problem Current performance evaluation methods in time series classification are criticized.
method No specific new method proposed, but a critical analysis of existing methods.
result Suggests a need for discussion and reflection on TSC performance evaluation.
Interactive tool helps choose and understand classification metrics.
problem Common metrics for binary classification have limitations.
method Graphical application to visualize and explore evaluation metrics.
result Promotes careful attention to interpretation of metrics.
Study finds AUC is most consistent across different prevalence in binary classification.
problem Consistency of model evaluation metrics across varying prevalence in binary classification.
method Analysis of 156 data scenarios with 18 metrics, 5 models, and a random guess model.
result AUC has the smallest variance in evaluating individual models and ranking of models.
Temporal information impacts only a fraction of time series datasets, skewing benchmark evaluations.
problem Temporal information's impact on time series classification is often overestimated.
method Permutation tests on UCR archive to identify datasets where temporal info is irrelevant.
result Many tabular datasets perform well without temporal info, skewing benchmark evaluations.
This paper reviews and proposes a new approach for evaluating internal cluster validation indices.
problem Selecting the best-performing unsupervised classification algorithm without external information.
method Examines and proposes a new evaluation approach for internal validation indices.
result Suggests a new evaluation approach for internal validation indices.
QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
Proposes MCC-F1 curve for better binary classification evaluation.
problem Misleading performance evaluations with ROC and PR curves for imbalanced data.
method Introduces MCC-F1 curve combining MCC and F1 score.
result MCC-F1 curve provides clearer classifier differentiation.
This paper reviews metrics for evaluating multi-class classification models.
problem Evaluating and comparing multi-class classification models.
method Review and analysis of metrics.
result Promising multi-class metrics are highlighted and their usages are demonstrated.
New scoring rules improve probabilistic classification model evaluation.
problem Traditional scoring rules misalign with the preference for correct classifications.
method Introduces Penalized Brier Score (PBS) and Penalized Logarithmic Loss (PLL) to modify proper scoring rules.
result PBS and PLL better identify optimal checkpoints and early stopping points, leading to superior F1 scores.
Proposes a new framework for uncertainty evaluation in ML classification models.
problem Uncertainty evaluation for ML classification models not addressed by existing metrological guidelines.
method Develops a metrological framework based on probability mass functions and summary statistics.
result Extends the GUM to uncertainty for nominal properties, applicable to ML classification models.
Generative model evaluates text emotion intensity, outperforming classification.
problem Limitations of discrete emotion classification in applied domains.
method Fine-tuning generative language models to output continuous emotion intensity scores.
result Generative model outperforms classification baselines and reveals generalization capabilities.
Paper benchmarks adversarial robustness methods on image classification.
problem Vulnerability of deep neural networks to adversarial examples.
method Established a comprehensive benchmark with robustness curves.
result Found important findings on adversarial attack and defense methods.
Critiques binary classification evaluation methods, advocating for proper scoring rules.
problem The dominance of top-K metrics and fixed-threshold evaluations in machine learning.
method Introduces a decision-theoretic framework mapping evaluation metrics to their use cases, and implements a clipped Brier score variant.
result Demonstrates the clinical utility of proper scoring rules through a Python package, exttt{briertools}.
Improves classifier evaluation by aligning with Total Classification Cost.
problem Lack of consensus on evaluation metrics and class imbalance issues.
method Introduces Weighted Accuracy (WA) and a reweighting framework for cost-sensitive scenarios.
result WA aligns with Total Classification Cost (TCC) minimization under realistic conditions.
This thesis evaluates text-based vs audio-based classification of mental health interviews.
problem Classifying psychiatric illness using text-based methods.
method Design and evaluate a text classification network on mental health interviews, using belabBERT.
result Text-based classification is a strong alternative to audio-based methods.
Paper proposes a unified framework for evaluating calibration of probabilistic models.
problem Evaluation of calibration for general probabilistic predictive models.
method Unified framework for calibration evaluation and tests for any probabilistic model.
result Generalization and reformulation of existing measures and tests.
Improves ROC/AUC for multi-class classification.
problem Lack of sensible plots, sensitivity to imbalanced data, inability to specify mis-classification cost, and lack of evaluation uncertainty quantification.
method Factorizes multi-class ROC into a one-dimensional vector representation for visualization and summary.
result Provides a binary AUC-equivalent summary and mis-classification weights specification.
Evaluates deep learning models in histopathology for robustness and classification strategies.
problem Lack of comprehensive evaluation of histopathology models beyond accuracy.
method Developed a new methodology to evaluate models on five histopathology datasets, including vision transformers and CNNs.
result Identified insights into cancer classification strategies and robustness against stain variations.
Study evaluates consistency of LLMs in binary text classification, providing systematic guidance.
problem Lack of reliable methods for evaluating large language model (LLM) binary text classification.
method Adapting psychometric principles, the study determines sample size requirements, develops metrics for invalid responses, and evaluates intra- and inter-rater reliability.
result LLMs demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with smaller models outperforming larger counterparts.
Improves comparison of F-measures for imbalanced datasets.
problem Comparing F-measures for classification algorithms on imbalanced data.
method Two improvements to existing F-measure comparison methods.
result Enhanced accuracy in comparing F-measures for classification.
This review explores resampling techniques for imbalanced binary classification.
problem Imbalanced classes lead to poor prediction results in classification.
method Classical, cost-sensitive, and Neyman-Pearson paradigms with resampling techniques and classification methods.
result Complex dynamics among resampling techniques, base methods, metrics, and imbalance ratios.
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
Comparison of decision curve analysis and cost curves for model evaluation.
problem Evaluating classification performance across different operating contexts.
method Comparison of Decision Curve Analysis (DCA) and Cost Curves.
result DCA and Cost Curves are closely related, with Brier curves being more generally applicable.
This paper evaluates algorithms for classification and outlier detection accuracies in temporal data. We focus on algorithms that train and classify rapidly and can be used for systems that need to incorporate new data regularly. Hence, we compare the accuracy of six fast algorithms using a range of well-known time-ser…
Method evaluates classification uncertainty with adaptively chosen features.
problem Finding a balance between model efficiency and fairness.
method Adaptively selects features for equalized coverage in classification.
result Valid and effective method demonstrated on simulated and real data.
Researchers compute Bayes error for classification models using normalizing flows.
problem Evaluating the inherent difficulty of classification problems.
method Invertible transformations and Gaussian base distributions to compute Bayes error.
result State-of-the-art models can achieve near-optimal accuracy but not always.
Two different formulas for macro F1 lead to significant differences in classification evaluation.
problem Evaluation discrepancies in binary, multi-class, and multi-label classification problems.
method Comparison of two formulas for macro F1 metric.
result The two formulas can result in up to a 0.5 difference and different classifier rankings.
Proposes a method to evaluate meta-learning performance based on task similarity.
problem Meta-learning performance evaluation ignores task similarity, leading to biased results.
method Generative approach using Latent Dirichlet Allocation to analyze task similarity.
result The proposed method provides an insightful evaluation of meta-learning algorithms, matching common intuition.
Screening methods improve model performance on 2 out of 10 datasets.
problem Evaluating usefulness of screening methods in feature selection.
method Evaluated 3 regression and 5 classification learners on 10 datasets using 7 screening methods.
result Screening methods improved model performance on 2 out of 10 datasets.
Defines explanations for classifier outcomes using causal concepts.
problem Understanding classifier outcomes in a causal context.
method Proposes a new definition of explanation based on causality, compares it with existing notions, and evaluates it experimentally.
result Experimental evaluation shows the new definition's effectiveness on financial datasets.
A new network-based method for high-level data classification without normalization.
problem Evaluation and classification of data with hidden attribute interactions.
method A new network-building methodology based on attribute-attribute interactions, avoiding normalization.
result Improvement in accuracy of high-level classification using betweenness centrality.
New MTSC archive adds 30 multivariate time series datasets for evaluation.
problem Lack of multivariate time series datasets for rigorous evaluation.
method Forming the first MTSC archive with 30 datasets, equal lengths, no missing data, and train/test splits.
result Addresses the need for a comprehensive evaluation of multivariate time series classification algorithms.
This paper proposes a use of an ordinal classifier to evaluate the financial solidity of non-life insurance companies as strong, moderate, weak, and insolvency. This study constructed an efficient classification model that can be used by regulators to evaluate the financial solidity and to determine the priority of fur…
Image classification has become one of the main tasks in the field of computer vision technologies. In this context, a recent algorithm called CapsNet that implements an approach based on activity vectors and dynamic routing between capsules may overcome some of the limitations of the current state of the art artificia…
Evaluates six ETSC algorithms on various datasets.
problem Comparing ETSC algorithms for different data characteristics.
method Empirical comparison of six ETSC algorithms on public and new datasets.
result Provides insights into algorithm performance on real-life applications.
CAT framework improves AI medical screening fairness and reliability.
problem Imbalanced data, varying performance across cohorts, and patient-level inconsistencies in traditional metrics.
method CAT framework introduces patient-level assessment, entropy-based distribution weighting, and cohort-weighted sensitivity and specificity.
result Enhanced predictive reliability, fairness, and interpretability of AI-driven medical screening models.
This study integrates cost-sensitive and causal classification methods.
problem Improving classification model performance in business decision-making.
method A unifying evaluation framework for cost-sensitive and causal classification.
result Conventional classification is a specific case of causal classification.
This paper evaluates methods to improve classification on imbalanced datasets.
problem Class imbalance in classification problems.
method Combination of data augmentation and ensemble learning methods.
result Combinations of data augmentation methods with ensemble learning can significantly improve classification performance.
Paper clarifies unary vs binary AV approaches and evaluates their performance.
problem Improperly treating authorship verification as a binary classification task.
method Literature review and evaluation of unary and binary AV methods.
result Improper binary classification leads to flawed AV methods.
Pairwise ranking aligns subjective clinical evaluations with objective indicators.
problem Aligning subjective clinical evaluations with objective indicators for improved diagnosis.
method Pairwise ranking methods to align subjective evaluations with objective indicators.
result The resulting score improves classification accuracy and provides a nuanced severity assessment.
This paper reviews PU learning evaluation methods and provides practical recommendations.
problem Evaluating PU learning methods when only positive and unlabelled data are available.
method Critical review of 51 articles proposing PU classifiers and alternative predictive accuracy measures.
result Practical recommendations for improving PU learning evaluation.
Early classification improves satellite-based crop type identification.
problem Accurate early identification of crop types from satellite imagery.
method End-to-end trainable recurrent neural network with an additional stopping probability based on previously seen data.
result The model can distinguish crop types before the end of the vegetative period.
Improves off-policy evaluation weights for balanced state-action pair distribution.
problem Imbalance in importance sampling weights for off-policy evaluation of contextual bandits.
method Balanced Off-Policy Evaluation (B-OPE) method that minimizes imbalance to desired counterfactual distribution of state-action pairs.
result Experimental evidence shows B-OPE improves offline policy evaluation in both discrete and continuous action spaces.
Study reveals class-dependent effects in perturbation-based feature attribution metrics for time series classification.
problem Varying effectiveness of perturbation-based metrics across different classes in time series models.
method Systematic empirical analysis across multiple datasets, model architectures, and perturbation strategies.
result Perturbation-based metrics show varying effectiveness across classes, with some metrics performing better for certain classes.
Time series are ubiquitous, and a measure to assess their similarity is a core part of many computational systems. In particular, the similarity measure is the most essential ingredient of time series clustering and classification systems. Because of this importance, countless approaches to estimate time series similar…
Framework for reproducible AD classification experiments using MRI and PET data.
problem Difficulty in reproducing and objectively comparing machine learning methods for Alzheimer's disease.
method Automatic conversion of datasets, modular preprocessing pipelines, feature extraction, and classification methods; evaluation framework.
result FDG PET outperformed T1 MRI, and classifiers trained on ADNI generalized well to other datasets.
Hierarchical Softmax approximates class probabilities for large datasets efficiently.
problem Computational inefficiency of Softmax for large-scale classification tasks.
method Used Hierarchical Softmax to approximate class probabilities efficiently.
result Hierarchical Softmax performance degrades as the number of classes increases.
We study the classification performance of Kronecker-structured models in two asymptotic regimes and developed an algorithm for separable, fast and compact K-S dictionary learning for better classification and representation of multidimensional signals by exploiting the structure in the signal. First, we study the clas…