Active sampling algorithm improves accuracy of inferred scores from pairwise comparisons.
problem Inference of accurate scores from time-consuming pairwise comparisons.
method Approximate message passing and expected information gain maximization.
result ASAP offers the highest accuracy of inferred scores compared to existing methods.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
MIRA scores assess conditional distribution accuracy using joint samples.
problem Assessing the accuracy of candidate conditional distributions.
method Analytic expression for Mira score based on equal probability mass regions.
result Mira enables Bayesian model comparison by quantifying alignment with true process.
Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
New method uses model comparison signals to improve LLM evaluation accuracy.
problem Limited benchmark sizes and model stochasticity in evaluating LLMs' mathematical reasoning.
method Combines standard labeled outcomes with model comparison signals to design a statistically efficient evaluation framework.
result Semiparametric estimator achieves the semiparametric efficiency bound and substantially improves ranking accuracy.
Proposes HTM for aggregating ranked data considering user accuracy.
problem Aggregating ranked data from heterogeneous users with varying accuracy levels.
method Heterogeneous Thurstone Model (HTM) with alternating gradient descent algorithm.
result Algorithm converges linearly and outperforms existing methods.
New proposed models are often compared to state-of-the-art using statistical significance testing. Literature is scarce for classifier comparison using metrics other than accuracy. We present a survey of statistical methods that can be used for classifier comparison using precision, accounting for inter-precision corre…
Rank aggregation systems collect ordinal preferences from individuals to produce a global ranking that represents the social preference. Rank-breaking is a common practice to reduce the computational complexity of learning the global ranking. The individual preferences are broken into pairwise comparisons and applied t…
This paper presents a detailed comparison of a recently proposed algorithm for optimizing decision trees, tree alternating optimization (TAO), with other popular, established algorithms. We compare their performance on a number of classification and regression datasets of various complexity, different size and dimensio…
Rank regression from pairwise comparisons requires many comparisons to accurately learn model parameters.
problem Learning model parameters for rank regression from noisy pairwise comparisons.
method Uniform random pairwise comparisons to estimate model parameters with a given accuracy.
result Learning model parameters requires a number of comparisons proportional to dNlog3N/ε2. We propose an experimental comparison between Deep Echo State Networks (DeepESNs) and gated Recurrent Neural Networks (RNNs) on multivariate time-series prediction tasks. In particular, we compare reservoir and fully-trained RNNs able to represent signals featured by multiple time-scales dynamics. The analysis is perfo…
Rank aggregation based on pairwise comparisons over a set of items has a wide range of applications. Although considerable research has been devoted to the development of rank aggregation algorithms, one basic question is how to efficiently collect a large amount of high-quality pairwise comparisons for the ranking pur…
New models ensure monotonicity in preference learning, improving accuracy especially with limited data.
problem Failure of widely used preference learning models to maintain monotonicity.
method Proposed Linear Generalized Bradley-Terry models with Diffusion Priors.
result New models improve accuracy, especially with limited data.
Enhances AI models with human feedback for noisy data.
problem Improving AI model alignment with human feedback in noisy environments.
method Two-stage SL+LHF framework connecting machine learning with human feedback.
result The LNCA ratio identifies conditions for SL+LHF superiority over pure SL.
Novel CNN-based gaze scanpath comparison distinguishes experts from novices in dental radiograph interpretation.
problem Distinguishing expertise in dental radiograph interpretation based on gaze behavior.
method Convolutional neural networks (CNN) process scene information at the fixation level, using image patches as input to compare gaze scanpaths.
result 93% accuracy in distinguishing experts from novices using image patch features.
Binary feedback outperforms ordinal comparisons in ranking recovery.
problem Challenges the conventional wisdom that ordinal comparisons offer richer information.
method Proposes a parametric framework for modeling ordinal paired comparisons, binarizing ordinal data, and proving faster convergence rates for binary comparisons.
result Binarizing ordinal data significantly improves ranking recovery accuracy and exhibits a substantial performance gap.
Develops a fast, accurate method for comparing networks.
problem Comparing networks with repeated observations and varying sizes/sparsity.
method A novel two-sample hypothesis testing method with theoretical guarantees.
result Outperforms existing tools in speed and accuracy, power-optimal.
Study compares federated learning and coreset approaches for privacy in distributed machine learning.
problem Measuring privacy in distributed machine learning approaches.
method Comparison of federated learning and coreset approaches using membership inference attack.
result Federated learning offers better privacy than coreset, but with higher communication cost.
Transformer learns to estimate negative binomial parameters efficiently.
problem Parameter estimation for over-dispersed count data in large screens.
method Pre-trained transformer trained on synthetic data generation to invert parameter to count transformation.
result Method of moments provides faster, more efficient, and better-calibrated estimates.
SC improves robustness in model comparison for misspecified models.
problem Model misspecification challenges in amortized Bayesian inference.
method Parameter posterior-based methods augmented with SC training.
result SC improves robustness under model misspecification.
The article explains how to estimate confusion matrices for classifiers using unlabeled data.
problem Estimating sensitivity and specificity of binary medical diagnostic tests without gold standard tests.
method Modifying diagnostic test solutions to estimate confusion matrices for classifiers on unlabeled data.
result The approach can be used to estimate accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
New scoring rules compare probabilistic top lists in classification.
problem Evaluation of probabilistic top lists in classification.
method Elicitability through symmetric proper scoring rules.
result Brier score provides a well-suited metric for comparison.
Automatic cover detection -- the task of finding in an audio database all the covers of one or several query tracks -- has long been seen as a challenging theoretical problem in the MIR community and as an acute practical problem for authors and composers societies. Original algorithms proposed for this task have prove…
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
In this paper, prediction for linear systems with missing information is investigated. New methods are introduced to improve the Mean Squared Error (MSE) on the test set in comparison to state-of-the-art methods, through appropriate tuning of Bias-Variance trade-off. First, the use of proposed Soft Weighted Prediction …
Vecchia approximations provide the best accuracy-runtime trade-off for Gaussian process approximations.
problem High computational cost of Gaussian processes for large data sets.
method Systematic comparison of different Gaussian process approximations.
result Vecchia approximations consistently provide the best accuracy-runtime trade-off.
Proposes a machine learning predictor for survey data.
problem Limited integration of machine learning in traditional surveys.
method Predictor supported by machine learning algorithms, analyzing departures from model assumptions.
result Machine learning predictors are a good alternative, even under small departures from model assumptions.
Unified view on random walk and Weisfeiler-Leman kernels, improving accuracy.
problem Improving graph kernel methods for better classification accuracy.
method Define and analyze walk-based node refinement methods, relate to Weisfeiler-Leman test, and introduce new walk-based kernels.
result Walk-based kernels are as expressive as Weisfeiler-Leman subtree kernel but support non-strict neighborhood comparison.
Researchers study fairness-accuracy tradeoffs in predictive models for multiple groups.
problem Understanding the tradeoff between fairness and accuracy in models serving multiple demographic groups.
method Characterizing the fairness-accuracy (FA) Pareto frontier, approximating it from limited data, and bounding the worst-case gap.
result Derivation of worst-case-optimal estimators and uniform finite-sample bounds for the entire FA frontier.
Study compares machine learning models and BERT on SQuAD dataset.
problem Comparing performance of machine learning models and BERT on SQuAD dataset.
method Comparative analysis of machine learning models and BERT on SQuAD dataset.
result BERT model outperforms other models on SQuAD dataset but requires more time.
Bispectral OT improves dataset comparison by preserving intrinsic coherence.
problem Ignoring intrinsic coherence in dataset comparisons using pairwise geometric distances.
method Introduces Bispectral Optimal Transport, a symmetry-aware extension of discrete OT.
result Transport plans computed with Bispectral OT achieve greater class preservation accuracy.
A comparison of SLDS and LSTM for pedestrian behavior prediction shows SLDS works better with shorter sequences.
problem Time-critical pedestrian behavior prediction in autonomous vehicles.
method Comparison of a switching linear dynamical system (SLDS) and a three-layered bi-directional LSTM neural network.
result SLDS achieves higher accuracy with shorter sequences (10 samples) compared to LSTM's 80% accuracy with 100 samples.
A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results). Past work has shown that under the BTL model, the widely-used maximum-likeli…
In recent work on both generative and discriminative score to log-likelihood-ratio calibration, it was shown that linear transforms give good accuracy only for a limited range of operating points. Moreover, these methods required tailoring of the calibration training objective functions in order to target the desired r…
Active learning optimizes correlation clustering by querying the most informative pairwise comparisons.
problem Efficiently clustering data with limited pairwise similarity information.
method Developed principled active learning approach using information-theoretic acquisition functions.
result Significantly outperforms existing baselines in clustering accuracy and query efficiency.
Multi-step ahead forecasting is still an open challenge in time series forecasting. Several approaches that deal with this complex problem have been proposed in the literature but an extensive comparison on a large number of tasks is still missing. This paper aims to fill this gap by reviewing existing strategies for m…
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about whic…
Simultaneously recorded electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be used to non-invasively measure the spatiotemporal dynamics of the human brain. One challenge is dealing with the artifacts that each modality introduces into the other when the two are recorded concurrently, for…
Recent progress in the development of efficient computational algorithms to price financial derivatives is summarized. A first algorithm is based on a path integral approach to option pricing, while a second algorithm makes use of a neural network parameterization of option prices. The accuracy of the two methods is es…
Online Streaming Feature Selection (OSFS) is a sequential learning problem where individual features across all samples are made available to algorithms in a streaming fashion. In this work, firstly, we assert that OSFS's main assumption of having data from all the samples available at runtime is unrealistic and introd…
s-OTDD compares datasets efficiently without training, robust to class variations.
problem Efficiently compare datasets without training or class variations.
method Moment Transform Projection (MTP) and sliced optimal transport.
result s-OTDD correlates with optimal transport and transfer learning performance.
A method compares AI corrections to a base model for explaining predictions.
problem Creating explanations for AI predictions.
method Introduces a surrogate model to correct a simpler base model and provides criteria for accuracy and fidelity.
result Induces neighborhoods of instances with ideal accuracy and fidelity.
Random Projection (RP) technique has been widely applied in many scenarios because it can reduce high-dimensional features into low-dimensional space within short time and meet the need of real-time analysis of massive data. There is an urgent need of dimensionality reduction with fast increase of big genomics data. Ho…
Paper introduces new metrics for evaluating model accuracy.
problem Improving model accuracy and calibration.
method Developed two second-order accuracy metrics with integral and numerical representations.
result Validates model calibration settings and reveals distortions.
Usually one compares the accuracy of two competing classifiers via null hypothesis significance tests (nhst). Yet the nhst tests suffer from important shortcomings, which can be overcome by switching to Bayesian hypothesis testing. We propose a Bayesian hierarchical model which jointly analyzes the cross-validation res…
Novel method for Bayesian model comparison using deep learning.
problem Comparing complex models in science with intractable likelihood functions.
method Simulation-based, purely deep learning approach that amortizes model fitting costs.
result Achieves excellent results in accuracy, calibration, and efficiency.
Estimates density ratio for two-sample comparison using tree models.
problem Comparing two distributions given i.i.d. observations.
method Additive tree models with balancing loss for density ratio estimation.
result Bayesian inference provides uncertainty quantification for density ratio.