Paper proposes an unbiased classifier from triplet comparison data.
problem Learning a classifier from triplet comparison data.
method Empirical risk minimization framework with an unbiased estimator.
result The proposed method achieves better performance than baseline methods.
Binary feedback outperforms ordinal comparisons in ranking recovery.
problem Challenges the conventional wisdom that ordinal comparisons offer richer information.
method Proposes a parametric framework for modeling ordinal paired comparisons, binarizing ordinal data, and proving faster convergence rates for binary comparisons.
result Binarizing ordinal data significantly improves ranking recovery accuracy and exhibits a substantial performance gap.
Comparison data arises in many important contexts, e.g. shopping, web clicks, or sports competitions. Typically we are given a dataset of comparisons and wish to train a model to make predictions about the outcome of unseen comparisons. In many cases available datasets have relatively few comparisons (e.g. there are on…
Optimal privacy-preserving ranking from noisy comparisons.
problem Protecting individual privacy in ranking from noisy comparisons.
method Differentially private ranking algorithms under edge and individual differential privacy.
result Achieved minimax optimal rates of convergence under privacy constraints.
A new method learns from pairwise comparisons to predict sensitive data without making strong assumptions.
problem Predicting sensitive data like annual income from unlabeled data with unknown target correspondence.
method Utilizes pairwise comparison data to learn a regression model without strong assumptions.
result The learned model converges to optimal with optimal parametric rate for uniformly distributed targets.
New method improves ABC for Bayesian model comparison.
problem Comparing complex models with observed data.
method Approximate Bayesian Computation with posterior density estimation.
result Efficiently assigns high posterior probabilities to ground-truth models.
Improves comparison of F-measures for imbalanced datasets.
problem Comparing F-measures for classification algorithms on imbalanced data.
method Two improvements to existing F-measure comparison methods.
result Enhanced accuracy in comparing F-measures for classification.
The paper designs tests for comparing ranked preference data and finds significant differences.
problem Comparing pairwise comparison and ranking data in various applications.
method Developed two-sample tests for pairwise comparison and ranking data, proving upper and lower bounds.
result Upper and lower bounds show tightness of the proposed tests, and significant differences in preferences were found.
This paper aims to present a general idea of method comparison of Credit Scoring techniques. Any scorecard can be made in various methods based on variable transformations in the logistic regression model. To make a comparison and come up with the proof that one technique is better than another is a big challenge due t…
New model reduces bias in crowdsourced pairwise comparisons.
problem Crowdsourced pairwise comparisons are biased due to perceptual factors.
method factorBT model accounts for irrelevant factors affecting worker answers.
result factorBT produces more accurate rankings than previous models.
In this study, a pairwise comparison matrix is generalized to the case when coefficients create Lie group G, non necessarily abelian. A necessary and sufficient criterion for pairwise comparisons matrices to be consistent is provided. Basic criteria for finding a nearest consistent pairwise comparisons matrix (extend…
New model for pairwise comparisons without stochastic transitivity.
problem Suboptimal performance of models assuming stochastic transitivity in real-world scenarios.
method Proposes a general family of statistical models using a skew-symmetric matrix.
result Achieves minimax-rate optimality and adapts to data sparsity.
SC improves robustness in model comparison for misspecified models.
problem Model misspecification challenges in amortized Bayesian inference.
method Parameter posterior-based methods augmented with SC training.
result SC improves robustness under model misspecification.
Paper establishes comparison theorems for large-margin learning.
problem Data piling issue in high-dimension and low-sample size SVM.
method Large-margin unified machines (LUM) loss functions.
result New comparison theorems for all LUM loss functions.
New algorithms learn from comparisons to classify data robustly to noise.
problem Learning robust classifiers from noisy data efficiently.
method Introducing comparison queries to active learning, providing noise-tolerant classifiers.
result First time and query efficient algorithms for robust learning under bounded noise.
TripletBoost learns classifiers from noisy triplet comparisons.
problem Learning from comparison-based data.
method Aggregate weak classifiers from weakly learned triplets, then boost.
result Theoretical guarantees and empirical competitiveness.
Study on deep neural networks for reward modeling with pairwise comparison data.
problem Reward modeling with deep neural networks in non-parametric settings.
method Established a non-asymptotic regret bound for deep reward estimators, introduced a margin-type condition.
result Improved regret bound for deep reward estimators, highlighting the importance of clear human beliefs.
ROVAE uses noisy pairwise comparisons to disentangle factors in VAEs.
problem Disentangling factors in VAEs requires an inductive bias.
method Robust Ordinal VAE (ROVAE) incorporates noisy pairwise ordinal comparisons to disentangle factors.
result ROVAE outperforms existing methods and is more robust to noisy comparisons.
Paper tackles clustering with ordinal comparisons, achieving near-optimal results.
problem Clustering with ordinal comparisons when similarity measures are not available.
method Two-step procedure: estimate similarity matrix from comparisons, then apply SDP clustering.
result Near-optimal recovery of planted clustering using near-optimal number of comparisons.
Unified framework for robust ordinal embedding from contaminated comparisons.
problem Inconsistent relative comparisons in ordinal data.
method Jointly identifies and corrects contaminated comparisons, learning embeddings directly.
result Unified framework alleviates sub-optimality and provides a robust solution.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
This paper proposes a new method for solving the well-known rank aggregation problem from pairwise comparisons using the method of low-rank matrix completion. The partial and noisy data of pairwise comparisons is transformed into a matrix form. We then use tools from matrix completion, which has served as a major compo…
Enhances AI models with human feedback for noisy data.
problem Improving AI model alignment with human feedback in noisy environments.
method Two-stage SL+LHF framework connecting machine learning with human feedback.
result The LNCA ratio identifies conditions for SL+LHF superiority over pure SL.
Active sampling algorithm improves accuracy of inferred scores from pairwise comparisons.
problem Inference of accurate scores from time-consuming pairwise comparisons.
method Approximate message passing and expected information gain maximization.
result ASAP offers the highest accuracy of inferred scores compared to existing methods.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
DECS tool assesses swap rates of DEXes and Fusion outperforms competitors.
problem Lack of unbiased swap rate comparisons in decentralized finance.
method Swap transaction monitoring and simulation techniques.
result 1inch Classic and Fusion consistently outperform competitors in swap rates.
Modeling preference rankings with salient features to explain irrational choices.
problem Estimating rankings from noisy pairwise comparisons with irrational choices.
method Salient feature preference model with maximum likelihood estimation.
result Strong performance of maximum likelihood estimation on synthetic and real data.
Rank aggregation systems collect ordinal preferences from individuals to produce a global ranking that represents the social preference. Rank-breaking is a common practice to reduce the computational complexity of learning the global ranking. The individual preferences are broken into pairwise comparisons and applied t…
New oracle uses uncertainty for active classification with noisy feedback.
problem Improving query complexity in interactive binary classifier learning.
method Proposes a new pairwise comparison oracle that considers uncertainty and an adaptive labeling algorithm.
result Demonstrates improved performance and efficiency compared to existing methods.
Study compares crowdsourcing with lab experiments using comparison-based psychophysics.
problem Improving data quality in crowdsourcing psychophysics experiments.
method Comparison-based psychophysics, machine learning for triplet prediction.
result Accuracy of crowdsourcing psychophysics close to lab experiments.
Develops a method to infer partial rankings from sparse comparisons.
problem Challenges in ranking items with limited and noisy comparisons.
method Nonparametric Bayesian approach for learning partial rankings.
result Finds partial rankings that distinguish meaningful differences only when data supports it.
We examine several recently suggested methods for the detection of long-range correlations in data series based on similar ideas as the well-established Detrended Fluctuation Analysis (DFA). In particular, we present a detailed comparison between the regular DFA and two recently suggested methods: the Centered Moving A…
New methods resolve conflicting treatment effect estimates in health tech assessments.
problem Conflicting conclusions from different sponsors analyzing the same data.
method Arbitrated indirect treatment comparisons (ArMAIC) targeting a common target population.
result Estimates treatment effects in a common target population, resolving the MAIC paradox.
Pairwise comparison data arises in many domains, including tournament rankings, web search, and preference elicitation. Given noisy comparisons of a fixed subset of pairs of items, we study the problem of estimating the underlying comparison probabilities under the assumption of strong stochastic transitivity (SST). We…
Paper accelerates MM algorithm for faster inference of ranking scores from comparison data.
problem Inference of Bradley-Terry model parameters from comparison data.
method Developed and analyzed MM algorithm for maximum likelihood and Bayesian estimation, proposed an accelerated version.
result Accelerated MM algorithm achieves faster convergence rates compared to classical MM algorithm.
A common problem in machine learning is to rank a set of n items based on pairwise comparisons. Here ranking refers to partitioning the items into sets of pre-specified sizes according to their scores, which includes identification of the top-k items as the most prominent special case. The score of a given item is defi…
We consider machine learning in a comparison-based setting where we are given a set of points in a metric space, but we have no access to the actual distances between the points. Instead, we can only ask an oracle whether the distance between two points i and j is smaller than the distance between the points i an…
Paper establishes statistical inference for pairwise comparison models.
problem Statistical inference for pairwise comparison models when the number of subjects diverges.
method Identifies Fisher information matrix as a weighted graph Laplacian for asymptotic normality.
result Near-optimal asymptotic normality result for maximum likelihood estimator.
Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
A faster algorithm for ranking from pairwise comparisons.
problem Efficiently ranking individuals or objects from pairwise comparisons.
method An alternative and simpler iterative algorithm for ranking that converges faster.
result The new algorithm is over 100 times faster in some cases.
Evidence Networks simplify Bayesian model comparison for complex models.
problem Bayesian model comparison challenges with intractable likelihoods or priors.
method Loss functions and neural networks for fast, amortized estimation of Bayes factors.
result Evidence Networks provide accurate and scalable Bayes factor estimation.
We performed an empirical comparison of ICA and PCA algorithms by applying them on two simulated noisy time series with varying distribution parameters and level of noise. In general, ICA shows better results than PCA because it takes into account higher moments of data distribution. On the other hand, PCA remains quit…
Novel CNN-based gaze scanpath comparison distinguishes experts from novices in dental radiograph interpretation.
problem Distinguishing expertise in dental radiograph interpretation based on gaze behavior.
method Convolutional neural networks (CNN) process scene information at the fixation level, using image patches as input to compare gaze scanpaths.
result 93% accuracy in distinguishing experts from novices using image patch features.
Paper tackles learning mixture of RUMs from partial data.
problem Learning a mixture of Random Utility Models (RUMs) from pairwise comparisons.
method PCA-based spectral clustering to reduce mixture to single component.
result Algorithm correctly clusters data from a mixture of RUMs with high probability.
MIRA scores assess conditional distribution accuracy using joint samples.
problem Assessing the accuracy of candidate conditional distributions.
method Analytic expression for Mira score based on equal probability mass regions.
result Mira enables Bayesian model comparison by quantifying alignment with true process.
GNNRank uses neural networks to learn global rankings from competition match data.
problem Learning global rankings from pairwise comparisons in directed graphs.
method Proposes GNNRank, a trainable GNN-based framework with digraph embedding and new objectives.
result GNNRank achieves competitive and superior performance compared to baselines.
Paper proposes a new method to compare classifiers across multiple datasets.
problem Comparing classifiers over multiple datasets with multiple criteria.
method Adopting decision theory, the paper introduces generalized stochastic dominance for ranking classifiers.
result Generalized stochastic dominance can be used to rank classifiers and statistically tested.
Enhances Bayesian model comparison with a probabilistic framework for meta-uncertainty.
problem Uncertainty in posterior model probabilities (PMPs) when derived from finite data.
method Develops a fully probabilistic approach to quantify and represent meta-uncertainty over PMPs.
result Demonstrates utility in various BMC contexts, including regression, MCMC, and neural networks.