In correlation clustering, we are given objects together with a binary similarity score between each pair of them. The goal is to partition the objects into clusters so to minimise the disagreements with the scores. In this work we investigate correlation clustering as an active learning problem: each similarity sc…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “score disagreement”
The issue of disagreements amongst human experts is a ubiquitous one in both machine learning and medicine. In medicine, this often corresponds to doctor disagreements on a patient diagnosis. In this work, we show that machine learning models can be trained to give uncertainty scores to data instances that might result…
Our work investigates disagreement in neural network feature attribution methods.
problem Disagreement among feature attribution methods for neural networks.
method Investigates the fundamental and distributional behavior of feature attribution methods.
result Illustrates the impact of scaling and encoding techniques on explanation quality.
Proposes a new method to measure epistemic uncertainty in Bayesian neural networks.
problem Measuring epistemic uncertainty in Bayesian neural networks for out-of-distribution detection.
method Proposes measuring disagreement between logits and their pre-softmax counterparts as an epistemic uncertainty measure.
result Proposed epistemic uncertainty scores outperform mutual information and equal predictive entropy performance.
Bayes-consistent disagreement discrepancy loss improves model robustness.
problem Distribution shift in real-world neural network deployment.
method Introducing a novel disagreement loss that is Bayes consistent.
result Proves existing surrogates for disagreement discrepancy are not Bayes consistent.
This paper characterizes and explains the disagreement between two graph embedding methods.
problem Understanding why two popular graph embedding methods produce different results.
method End-to-end analysis of ASE-LSE latent subspaces, proving conditions for agreement and disagreement.
result No maximal-disagreement graph exists; disagreement is strictly below its theoretical ceiling.
The study reveals a linear relationship between source and target domain classification errors based on disagreement.
problem Evaluating model performance under distribution shift with limited labeled data.
method Developed a theoretical foundation for analyzing disagreement in high-dimensional random features regression.
result The disagreement-on-the-line phenomenon occurs when classification error under the source domain is a linear function of the target domain.
Disagreement is an essential element of science and life in general. The language of probabilities and statistics is often used to describe disagreements quantitatively. In practice, however, we want much more than that. We want disagreements to be resolved. This leaves us with a substantial knowledge gap which is ofte…
Traditional multi-view learning approaches suffer in the presence of view disagreement,i.e., when samples in each view do not belong to the same class due to view corruption, occlusion or other noise processes. In this paper we present a multi-view learning approach that uses a conditional entropy criterion to detect v…
Disagreement-based approaches generate multiple classifiers and exploit the disagreement among them with unlabeled data to improve learning performance. Co-training is a representative paradigm of them, which trains two classifiers separately on two sufficient and redundant views; while for the applications where there…
This study finds ESG rating disagreement reduces corporate productivity, especially in certain types of firms.
problem The impact of ESG rating disagreement on corporate productivity.
method Analysis of A-share listed companies data from 2015 to 2022 using XGBoost regression and SHAP.
result ESG rating disagreement reduces corporate productivity, especially in certain types of firms.
Online monitor detects classifier drift and adapts predictions.
problem Silent degradation of classifier accuracy under distributional shift.
method Sliding-window KS statistic with calibrated alarm thresholds.
result 86.6% valid detection across various shift conditions.
Improves reliability of medical diagnosis uncertainty estimates.
problem Label uncertainty in medical diagnosis.
method Post-hoc -calibration method for neural network classifiers.
result Significantly enhances reliability of uncertainty estimates.
Inside the Mind of Investors During the COVID-19 Pandemic: Evidence from the StockTwits Dataq-fin.ST
Study investor sentiment and disagreement on StockTwits during COVID-19.
problem Understanding investor beliefs and sentiment during the pandemic.
method Analysis of social media data (StockTwits) for investor messages.
result Sentiment and disagreement sharply decreased in early March 2020, followed by a reversal.
Framework assesses autograders' reliability and biases.
problem Mixed reliability and biases in autograders for LLM evaluation.
method Bayesian GLMs to model evaluation outcomes.
result Explicit quantification of scoring differences and biases.
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
SGD-trained models' disagreement predicts test error.
problem Estimating test error of deep networks.
method Empirical testing and theoretical analysis of SGD ensembles.
result SGD ensembles' disagreement correlates with test error.
When might human input help (or not) when assessing risk in fairness domains? Dressel and Farid (2018) asked Mechanical Turk workers to evaluate a subset of defendants in the ProPublica COMPAS data for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by human…
Learning with noisy labels is one of the hottest problems in weakly-supervised learning. Based on memorization effects of deep neural networks, training on small-loss instances becomes very promising for handling noisy labels. This fosters the state-of-the-art approach "Co-teaching" that cross-trains two deep neural ne…
This paper improves active learning by using robust divergences for committee disagreement.
problem Active learning with high measurement costs.
method Query by committee with Bregman divergence (including Kullback-Leibler divergence as a special case).
result The proposed method is more robust and performs as well as or better than conventional methods.
New algorithm improves active learning in agnostic pool-based classification.
problem Efficient active learning in the agnostic setting with minimized sample complexity.
method Solves an experimental design problem to determine a distribution over examples for label requests.
result Achieves sample complexity bounds never worse than best disagreement coefficient-based bounds, sometimes significantly smaller.
Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable t…
Structured Credal Learningstat.ME
Structured credal learning separates covariate shift and label disagreement.
problem Uncertainty in real-world learning tasks due to covariate shift and noisy labels.
method Introduces a structured credal learning framework that explicitly separates these sources.
result Geometric bounds and decomposition reveal how covariate shifts affect label disagreement contributions.
The paper analyzes how multiple classifiers' disagreement and polarization affect overall accuracy.
problem Improving accuracy through ensembling multiple classifiers.
method The paper derives an upper bound for polarization, proposes a neural polarization law, and presents a tight upper bound for the error of majority vote classifiers.
result Disagreement and polarization among classifiers are linearly correlated with the target, and polarization is nearly constant for a dataset.
Solves a 60-year-old question on agreement measures in statistics.
problem The challenge of measuring agreement between two raters or measures.
method Developed a new algorithm to minimize diagonals in contingency tables, formulated the minimum feasible agreement, and studied the lower limit of maximum feasible agreement.
result Formulated the lower limit of Cohen's kappa and two statistics for agreement analysis.
We provide new results for noise-tolerant and sample-efficient learning algorithms under -concave distributions. The new class of -concave distributions is a broad and natural generalization of log-concavity, and includes many important additional distributions, e.g., the Pareto distribution and -distribution.…
GRANITE: A Generalized Regional Framework for Identifying Agreement in Feature-Based Explanationsstat.ML
GRANITE unifies feature-based explanation methods to reduce disagreement.
problem Disagreement among feature-based explanation methods.
method GRANITE partitions feature space into regions minimizing interaction and distribution influences.
result Unified and consistent feature explanations.
Study compares and contrasts various ML explanation methods, highlighting their disagreements and similarities.
problem Understanding and quantifying the differences between various machine learning explanation methods.
method Synthesized and visualized various explanation methods for global and local aspects of ML models.
result There is substantial agreement on the top features but less on specific rankings, and tree interpreter is comparable to SHAP for feature effects.
EXAGREE selects a stakeholder-aligned model to reduce conflicting explanations in machine learning.
problem Conflicting explanations from different attribution methods limit the adoption of machine learning models in safety-critical domains.
method EXAGREE is a two-stage framework that selects a Stakeholder-Aligned Explanation Model (SAEM) from a set of similar-performing models, maximizing Stakeholder-Machine Agreement (SMA).
result EXAGREE achieves simultaneous gains in faithfulness, plausibility, and fairness over baselines while preserving task accuracy.
Paper tackles ESG rating disagreement in sustainable investing portfolios.
problem Lack of alignment between ESG ratings from different agencies affects investment decisions.
method Proposes a nonlinear optimization model reformulated as a convex quadratic program to address ESG rating disagreement.
result The proposed model can effectively manage ESG rating disagreement and improve investment decisions.
AI agents in experimental markets exhibit behavioral patterns that aggregate into market dynamics.
problem Understanding AI trading behavior and its impact on market dynamics.
method Experimental asset markets populated by AI agents trained on Large Language Models (LLMs).
result AI agents' behavior leads to market dynamics similar to human traders, including bubbles.
AutoDIME automates design of multi-agent environments for RL.
problem Designing multi-agent environments for reinforcement learning is challenging.
method Developed intrinsic teacher rewards for multi-agent settings and evaluated them in various tasks.
result Value disagreement was found to be most consistent and effective across tasks.
New method gives provable error bounds for neural nets under distribution shift.
problem Proving reliable error bounds for neural networks under distribution shift.
method Optimizing a classifier to disagree with another, using a new 'disagreement loss'.
result Valid error bounds with comparable accuracy to competitive methods.
This paper studies Learning from Observations (LfO) for imitation learning with access to state-only demonstrations. In contrast to Learning from Demonstration (LfD) that involves both action and state supervision, LfO is more practical in leveraging previously inapplicable resources (e.g. videos), yet more challenging…
Bayesian methods improve OoD detection in deep networks.
problem Detecting Out-of-Distribution (OoD) inputs in deep neural networks.
method Three Bayesian inference approaches applied to VAE weights.
result Improved OoD detection scores over benchmarks.
New method reduces version space for CNNs, improving active learning performance.
problem Sampling bias in active learning hinders optimal hypothesis finding in neural networks.
method Version space reduction through prior mass reduction and diameter reduction, proposing a new Gibbs-vote disagreement method.
result Diameter-based querying method reduces version space more effectively than prior mass reduction and other methods.
Zero-shot understanding of accidents from surveillance videos using vision-language models
problem Accident understanding from surveillance videos
method Three-stage pipeline with vision-language similarity, metadata-driven multi-prompt reasoning, and entropy-gated pairwise adjudicator
result Substantial improvement in harmonic-mean score over baseline
Ensembling improves performance when classifiers disagree more than average.
problem When do ensembles provide significant performance improvements in classification tasks?
method Theoretical and empirical analysis of ensemble improvement rate and disagreement-error ratio.
result Ensembling improves performance significantly when the disagreement rate is large relative to the average error rate.
New method pools labels from similar data items to improve learning from small samples.
problem Learning from small, human-annotated samples with potential disagreement among annotators.
method Proposes neighborhood-based pooling for sharing labels across similar data items.
result Improves learning from small, noisy samples by pooling labels from similar items.
The paper tackles fair correlation clustering with fairness constraints.
problem Minimizing disagreements while adhering to fairness constraints for clustering.
method Two variants of fairness constraints are considered: equal distribution and relative bounds. Approximation algorithms are developed for these constraints.
result Approximation algorithms for fair correlation clustering with theoretical guarantees and empirical validation.
CreDRO learns credal ensembles via distributionally robust optimization, improving EU quantification.
problem Quantifying predictive epistemic uncertainty in credal models.
method Distributionally robust optimization to capture EU from training randomness and potential distribution shifts.
result Empirically, CreDRO outperforms existing credal methods on various tasks.
A test assesses the calibration of set-based epistemic uncertainty representations.
problem Evaluating the accuracy of set-based representations of epistemic uncertainty in machine learning.
method Proposes a novel statistical test to determine if a convex combination of predictions is calibrated, allowing instance-level variability.
result Demonstrates the benefits of capturing instance-level variability on synthetic and real-world experiments.
The paper explores how Shapley value for a feature can vary based on model outcomes and feature distribution.
problem The uniqueness of Shapley value in explaining model predictions.
method Analyzes the relationship between feature distribution and Shapley value, and compares Shapley values for different model outcomes.
result Shapley value for a feature depends on more than just its mean and can vary significantly based on model outcome.
We introduce a new and improved characterization of the label complexity of disagreement-based active learning, in which the leading quantity is the version space compression set size. This quantity is defined as the size of the smallest subset of the training data that induces the same version space. We show various a…
We show that the disagreement coefficient of certain smooth hypothesis classes is , where is the dimension of the hypothesis space, thereby answering a question posed in \cite{friedman09}.
We study agnostic active learning, where the goal is to learn a classifier in a pre-specified hypothesis class interactively with as few label queries as possible, while making no assumptions on the true function generating the labels. The main algorithms for this problem are {\em{disagreement-based active learning}}, …
This paper relaxes the common prior assumption in the public and private information game of Morris and Shin (2000, 2004). For the generalized game, where the agent's prior expectations are heterogenous, it derives a sharp condition for the emergence of unique/multiple equilibria. This condition indicates that unique e…
Out-of-distribution (OOD) detection approaches usually present special requirements (e.g., hyperparameter validation, collection of outlier data) and produce side effects (e.g., classification accuracy drop, slower energy-inefficient inferences). We argue that these issues are a consequence of the SoftMax loss anisotro…