arXiv research

ELICA helps analysts extract relevant information during elicitation meetings.

Framework for eliciting fairness constraints from stakeholders.

We discuss equivalent axiomatic characterizations of distortion risk measures, and give a novel and concise proof of the characterization of elicitable distortion risk measures. Elicitability has recently been discussed as a desirable criterion for risk measures, motivated by statistical considerations of forecasting. …

This paper develops a new method for eliciting more flexible metrics, improving fairness and applicability.

ELICA helps analysts understand unfamiliar domains by extracting relevant terms.

Proposes a method to select fair performance metrics through metric elicitation.

A property, or statistical functional, is said to be elicitable if it minimizes expected loss for some loss function. The study of which properties are elicitable sheds light on the capabilities and limitations of point estimation and empirical risk minimization. While recent work asks which properties are elicitable, …

This paper tackles sample elicitation for learning systems, introducing a method to incentivize truthful samples.

Method combines deep learning and elicitability for solving complex stochastic equations.

This paper improves prediction in small data sets by eliciting expert knowledge about feature similarities.

Constructs new elicitable risk measures with multiplicative scoring functions.

Gradient optimization improves preference elicitation for large item spaces.

Robustifies elicitable functionals to handle small distribution misspecifications.

Study generalizes property elicitation to imprecise probabilities.

A statistical functional, such as the mean or the median, is called elicitable if there is a scoring function or loss function such that the correct forecast of the functional is the unique minimizer of the expected score. Such scoring functions are called strictly consistent for the functional. The elicitability of a …

Proposes a new framework for risk-sensitive RL using deep nets.

Study creates web interface to elicit user-preferred metrics.

A method for eliciting expert beliefs using preferential questions and normalizing flows.

Given a binary prediction problem, which performance metric should the classifier optimize? We address this question by formalizing the problem of Metric Elicitation. The goal of metric elicitation is to discover the performance metric of a practitioner, which reflects her innate rewards (costs) for correct (incorrect)…

The paper analyzes elicitability of return risk measures and their scoring functions.

Proposes method for eliciting non-parametric joint priors using normalizing flows.

This thesis formalizes metric selection for machine learning applications.

The risk of a financial position is usually summarized by a risk measure. As this risk measure has to be estimated from historical data, it is important to be able to verify and compare competing estimation procedures. In statistical decision theory, risk measures for which such verification and comparison is possible,…

Develops a simulation-based method to translate expert knowledge into prior distributions for Bayesian models.

Paper establishes identifiability and elicitability of tail risk measures.

In this note, we comment on the relevance of elicitability for backtesting risk measure estimates. In particular, we propose the use of Diebold-Mariano tests, and show how they can be implemented for Expected Shortfall (ES), based on the recent result of Fissler and Ziegel (2015) that ES is jointly elicitable with Valu…

This paper attempts to provide a decision-theoretic foundation for the measurement of economic tail risk, which is not only closely related to utility theory but also relevant to statistical model uncertainty. The main result is that the only risk measures that satisfy a set of economic axioms for the Choquet expected …

This paper explores how to choose scoring rules for estimating properties with parametric assumptions.

New algorithm achieves faster multicalibration in online settings.

Framework uses IRL and RL to elicit and optimize risk preferences robustly to noise.

New method allows backtesting of systemic risk forecasts.

We propose a cost-effective framework for preference elicitation and aggregation under the Plackett-Luce model with features. Given a budget, our framework iteratively computes the most cost-effective elicitation questions in order to help the agents make a better group decision. We illustrate the viability of the fram…

Formulates a Dueling Bandits problem for eliciting Kemeny rankings.

New technique reduces language biases in large language models.

Paper proposes incentives for federated learning to ensure truthful contributions.

A framework for eliciting utility functions from investor preferences.

In human-in-the-loop machine learning, the user provides information beyond that in the training data. Many algorithms and user interfaces have been designed to optimize and facilitate this human--machine interaction; however, fewer studies have addressed the potential defects the designs can cause. Effective interacti…

t-Distributed Stochastic Neighbor Embedding (t-SNE) is one of the most widely used dimensionality reduction methods for data visualization, but it has a perplexity hyperparameter that requires manual selection. In practice, proper tuning of t-SNE perplexity requires users to understand the inner working of the method a…

Providing accurate predictions is challenging for machine learning algorithms when the number of features is larger than the number of samples in the data. Prior knowledge can improve machine learning models by indicating relevant variables and parameter values. Yet, this prior knowledge is often tacit and only availab…

Motivated by an application of eliciting users' preferences, we investigate the problem of learning hemimetrics, i.e., pairwise distances among a set of $n$ items that satisfy triangle inequalities and non-negativity constraints. In our application, the (asymmetric) distances quantify private costs a user incurs when s…

Study uses property elicitation to understand how fairness regularizers affect optimal decisions.

PPT optimizes transformer behavior by steering its latent posterior using prior samples.

In this paper we propose an approach to preference elicitation that is suitable to large configuration spaces beyond the reach of existing state-of-the-art approaches. Our setwise max-margin method can be viewed as a generalization of max-margin learning to sets, and can produce a set of "diverse" items that can be use…

Crowdsourced wisdom improves causal learning.

Generative Adversarial Regression (GAR) learns risk scenarios robustly across policies.

Platform uses queries to elicit investor preferences for portfolio trades, improving allocation efficiency.

ContextBench benchmarks methods for generating linguistically fluent inputs that activate specific latent features in language models.

New scoring rules compare probabilistic top lists in classification.