ALICE combines feature selection and inter-rater agreeability for ML model insights.
problem Improving interpretability of black box machine learning models.
method Integrates feature selection and inter-rater agreeability into a user-friendly Python library.
result Initial experiments on customer churn modeling show promising insights.
Framework assesses autograders' reliability and biases.
problem Mixed reliability and biases in autograders for LLM evaluation.
method Bayesian GLMs to model evaluation outcomes.
result Explicit quantification of scoring differences and biases.
Quantifying the degree of atrophy is done clinically by neuroradiologists following established visual rating scales. For these assessments to be reliable the rater requires substantial training and experience, and even then the rating agreement between two radiologists is not perfect. We have developed a model we call…
Machine learning improves EEG pathology classification.
problem Automating clinical EEG analysis using machine learning.
method Developed a comprehensive feature-based framework and compared it to deep neural networks.
result Feature-based framework achieves accuracies similar to deep neural networks.
An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and rely on annotation schemes with low …
Automatically evaluates image quality based on human judgment.
problem Difficulty in rigorously evaluating generated image quality.
method Generative model embeddings, human labels regression, and statistical matching.
result 66% accuracy in predicting human scores of image realism.
Study evaluates consistency of LLMs in binary text classification, providing systematic guidance.
problem Lack of reliable methods for evaluating large language model (LLM) binary text classification.
method Adapting psychometric principles, the study determines sample size requirements, develops metrics for invalid responses, and evaluates intra- and inter-rater reliability.
result LLMs demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with smaller models outperforming larger counterparts.
k-Rater reliability corrects under-reporting of aggregated data reliability.
problem Under-reporting of data reliability in aggregated ratings.
method k-Rater reliability (kRR) as a multi-rater generalization of IRR.
result kRR provides a more accurate measure of reliability for aggregated datasets.
Study evaluates five LLMs for financial report analysis, revealing performance differences and variability.
problem Lack of understanding in reliability, consistency, and transparency of LLMs in financial analysis.
method Human evaluation, automated similarity metrics, and behavioral diagnostics applied to five transformer-based LLMs over U.S. 10-K filings.
result No single LLM consistently dominates across all evaluation perspectives, highlighting variability and need for interpretability.
We derive the mapping between two of the most pervasive utility functions, the mean square error (MSE) and the concordance correlation coefficient (CCC, ρc). Despite its drawbacks, MSE is one of the most popular performance metrics (and a loss function); along with lately ρc in many of the sequence prediction…
Study benchmarks automated sleep staging against human scorers, achieving human-level performance.
problem Lack of standardized comparison between human and automated sleep staging.
method Developed multi-scored datasets and a framework to compare multiple human scorers' consensus.
result Many automated methods can match human scorers' performance, with SimpleSleepNet achieving high F1 scores.
Infants' spontaneous and voluntary movements mirror developmental integrity of brain networks since they require coordinated activation of multiple sites in the central nervous system. Accordingly, early detection of infants with atypical motor development holds promise for recognizing those infants who are at risk for…
Solves a 60-year-old question on agreement measures in statistics.
problem The challenge of measuring agreement between two raters or measures.
method Developed a new algorithm to minimize diagonals in contingency tables, formulated the minimum feasible agreement, and studied the lower limit of maximum feasible agreement.
result Formulated the lower limit of Cohen's kappa and two statistics for agreement analysis.
SMART is an open source web application designed to help data scientists and research teams efficiently build labeled training data sets for supervised machine learning tasks. SMART provides users with an intuitive interface for creating labeled data sets, supports active learning to help reduce the required amount of …
An explicit solution found for maximizing/minimizing agreement in a 2x2 table.
problem Maximizing or minimizing agreement between clusterings with given marginals.
method Formal framework for several agreement measures, explicit solution for 2x2 table.
result An explicit solution for the 2x2 case.
Improves reliability of medical diagnosis uncertainty estimates.
problem Label uncertainty in medical diagnosis.
method Post-hoc alpha-calibration method for neural network classifiers. result Significantly enhances reliability of uncertainty estimates.
This work introduces significativity indices for agreement values between classifiers.
problem Evaluating the quality of agreement measures between classifiers.
method Proposes general approach and two specific indices for evaluation.
result Introduces two new significativity indices for evaluating agreement values.
SNAP improves robust computation by emphasizing trustworthy items and downweighting outliers.
problem Improving robustness in computation, especially in high-dimensional settings.
method SNAP assigns weights based on mutual agreement, suppressing outlier contributions.
result SNAP ensures outliers contribute negligibly to computations, even in high-dimensional settings.
Discriminatory trade liberalization policies are becoming more popular among world economies. Countries are motivated to enter for regional trade agreements to capture faster economic growth for alleviating poverty. In developing economies like most of the member countries of the Association of South East Asian Nations…
LFD method improves text classification by making features clearer and less label-leaking.
problem Creating interpretable text representations that are both predictive and understandable.
method LFD method: proposes lexical and semantic features from contrastive text pairs, screens candidates using κ, and selects features by residual gain. result LFD features achieve higher human-human and human-LLM agreement than baseline concepts and are less label-leaking.
Formula found for minimum ARI between clusterings of fixed sizes.
problem Understanding the lowest possible agreement between clusterings.
method Explicit formula derivation for minimum ARI.
result A specific pair of clusterings achieving the minimum ARI is provided.
In unsupervised machine learning, agreement between partitions is commonly assessed with so-called external validity indices. Researchers tend to use and report indices that quantify agreement between two partitions for all clusters simultaneously. Commonly used examples are the Rand index and the adjusted Rand index. …
In many machine learning problems, labeled training data is limited but unlabeled data is ample. Some of these problems have instances that can be factored into multiple views, each of which is nearly sufficent in determining the correct labels. In this paper we present a new algorithm for probabilistic multi-view lear…
In Bipartite Correlation Clustering (BCC) we are given a complete bipartite graph G with `+' and `-' edges, and we seek a vertex clustering that maximizes the number of agreements: the number of all `+' edges within clusters plus all `-' edges cut across clusters. BCC is known to be NP-hard. We present a novel approx…
Proposes a strategy to train models with minimal labeled data.
problem Scarce and expensive labeled data for medical tasks.
method Recursive training strategy to use image-level annotations for pixel-level segmentation.
result Improved segmentation of intracranial hemorrhage in CT scans.
Model selection is a problem that has occupied machine learning researchers for a long time. Recently, its importance has become evident through applications in deep learning. We propose an agreement-based learning framework that prevents many of the pitfalls associated with model selection. It relies on coupling the t…
A new method for combining multiple data views in supervised learning.
problem Combining multiple data views in supervised learning, especially in biology and medicine.
method Cooperative learning combines squared error loss with an agreement penalty to encourage predictions from different data views to agree.
result Cooperative learning achieves higher predictive accuracy on simulated and real multiomics data.
The retinal vascular condition is a reliable biomarker of several ophthalmologic and cardiovascular diseases, so automatic vessel segmentation may be crucial to diagnose and monitor them. In this paper, we propose a novel method that combines the multiscale analysis provided by the Stationary Wavelet Transform with a m…
Computable contracts simplify financial transactions and reduce legal costs.
problem Difficulty in querying, executing, and analyzing text-based financial contracts.
method Develop a Contract Definition Language and illustrate use cases.
result Substantial improvements in customer experience and cost reduction.
Generalization and reliability of multilingual translation often highly depend on the amount of available parallel data for each language pair of interest. In this paper, we focus on zero-shot generalization---a challenging setup that tests models on translation directions they have not been optimized for at training t…
We apply an asymmetric version of Kirman's herding model to volatile financial markets. In the relation between returns and agent concentration we use the square root law proposed by Zhang. This can be derived by extending the idea of a critical mean field theory suggested by Plerou et al. We show that this model is eq…
Study finds simple model-agreement scores perform well in various error estimation scenarios.
problem Evaluating model performance on unseen distributions using disparate scoring functions.
method Rigorously studied popular scoring functions (confidence, local manifold smoothness, model agreement) independently of mechanism choice.
result Simple model-agreement scores outperform confidence- and smoothness-based scores in realistic settings with compromised training data.
Study optimizes deep learning models for sleep stage classification.
problem Time-consuming and inconsistent manual sleep stage scoring.
method Investigated architectural choices in encoder-predictor architectures for polysomnography recordings.
result Robust architectures improve sleep stage classification performance.
Researchers develop multi-agent systems for quadcopters to collaborate in missions.
problem Enable multiple quadcopters to work together in remote sensing tasks.
method Agent dynamics, network topologies, collective behaviors, agreement protocol, equations of motion for quadcopters.
result Multi-agent systems can successfully collaborate in remote sensing missions.
Ensemble learning is a powerful approach to construct a strong learner from multiple base learners. The most popular way to aggregate an ensemble of classifiers is majority voting, which assigns a sample to the class that most base classifiers vote for. However, improved performance can be obtained by assigning weights…
Study shows more data improves model explanations, aiding reliable knowledge extraction.
problem Challenges in deriving reliable knowledge from machine learning models due to the Rashōmon effect.
method Examined the influence of sample size on explanations from models in a Rashōmon set using SHAP.
result Explanations from <128 samples are highly variable, but agreement improves with more data.
There is considerable debate whether the domestic political institutions (specifically, the country s level of democracy) of the host developing country toward foreign investors are effective in establishing the credibility of commitments are still underway, researchers have also analyzed the effect of international in…
Unified framework for policy learning using weak supervision.
problem High-quality supervision is often infeasible or expensive in practice.
method Treat weak supervision as imperfect peer information and evaluate policies based on correlated agreement.
result Substantial performance improvements, especially in complex or noisy environments.
MAS scores cluster size consistency from points, robust to label changes.
problem Desired uniformity in cluster sizes, stability under label perturbations.
method Mass Agreement Score (MAS) measures point-centric cluster size consistency, robust to label changes.
result MAS yields similar scores for partitions with similar bulk structure, sensitive to genuine redistribution of cluster mass.
In important applications involving multi-task networks with multiple objectives, agents in the network need to decide between these multiple objectives and reach an agreement about which single objective to follow for the network. In this work we propose a distributed decision-making algorithm. The agents are assumed …
Co-learning BO improves global optimization with limited samples.
problem Suboptimal solutions in Bayesian optimization due to poor surrogate accuracy.
method Build multiple Gaussian process models to complement each other and reduce prediction errors.
result CLBO achieves more efficient global optimization with fewer samples.
Model explains money creation under regulatory constraints.
problem Understanding money creation dynamics under regulatory pressures.
method Agent-based model of secured interbank network.
result Excess liquidity and repurchase agreements emerge due to regulatory constraints.
New index improves anomaly detection in correlated time series data.
problem Challenges in evaluating cluster quality for anomaly detection.
method Introduced Synchronized Anomaly Agreement Index (SAAI) to assess cluster quality.
result Maximizing SAAI improves anomaly detection accuracy by 0.23 compared to SSC and by 0.32 compared to X-Means.
The paper uses Black-Scholes model to analyze political support and coalition agreements.
problem Determining the minimum support level for a minor party in a pre-electoral coalition.
method Modeling political support as a stochastic process with a deterministic growth rate and applying Black-Scholes option pricing theory.
result The minimum support level for a minor party to gain a representative in a pre-electoral coalition.
This paper examines how regional trade agreements affect global trade relationships.
problem The relationship between regional trade agreements and global trade purity.
method Defined and decomposed synthesized trade resistance, separated natural and artificial factors, used expectation maximization algorithm to optimize parameters, and quantified trade purity indicator.
result Regional trade agreements contribute to the relative prosperity of EU and NAFTA countries, but weaken the role of trade unions and accelerate multilateral trade liberalization.
ValueBlindBench tests LLM-generated investment rationales for validity before returns are known.
problem Delayed-ground-truth evaluation of LLM-generated investment rationales.
method Agreement-gated stress testing protocol to validate LLM-judged rationales.
result ValueBlindBench prevents overclaims and identifies flawed financial constructs.
New algorithms for collaborative learning in uncertain, decentralized environments.
problem Byzantine collaborative learning in uncertain, decentralized environments.
method Two asynchronous solutions to averaging agreement, each optimal according to some dimension.
result New algorithms achieve optimal Byzantine resilience and collaborative learning.
Paper develops framework for valuing and assessing credit risk in renewable PPAs.
problem Renewable PPAs expose both parties to counterparty credit risk.
method Modelled joint dynamics of electricity prices and renewable output, incorporated default probabilities.
result Provides transparent metric for PPA valuation under counterparty risk.