ELICA helps analysts extract relevant information during elicitation meetings.
problem Challenges in eliciting requirements due to analyst's lack of domain knowledge.
method ELICA uses a novel information extraction algorithm combining WFSTs and SVMs, presented in an interactive GUI.
result ELICA effectively extracts relevant information in real-time and facilitates interactive and dynamic process.
Interactive tool improves prediction accuracy in small datasets.
problem Challenges in machine learning with small data sets and tacit expert knowledge.
method Interactive visualization and user model to elicit feature relevance.
result User model significantly improves prediction accuracy and prior knowledge elicitation.
Method combines deep learning and elicitability for solving complex stochastic equations.
problem Solving McKean-Vlasov FBSDEs with common noise.
method Combines Picard iterations, elicitability, and deep learning.
result Validated on systemic-risk model and extended to quantile-mediated interactions.
We propose a decomposition technique to reduce user cognitive load in constructive preference elicitation.
problem Learning user preferences in large combinatorial decision problems.
method Part-wise inference and feedback over partial configurations.
result Significantly reduced user cognitive load and up to exponentially less computational demand.
A method to prevent overfitting by modeling user knowledge in interactive machine learning.
problem Overfitting due to user reinforcement of noisy patterns in training data.
method User modelling based on rational behavior to correct overfitting.
result Improves predictive performance in sentiment analysis tasks.
Bayesian method helps decision-makers find preferred solutions in multi-objective optimization.
problem Identifying preferred solutions from the Pareto set in multi-objective optimization problems.
method Bayesian model to estimate decision-maker's utility function based on pairwise comparisons, guided by a principled elicitation strategy.
result Superior performance in finding high-utility solutions with a small number of queries.
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria. These criteria can be difficult to formalize, even when it is easy for an analyst to know a good clustering when she sees one. We presen…
A good clustering can help a data analyst to explore and understand a data set, but what constitutes a good clustering may depend on domain-specific and application-specific criteria. These criteria can be difficult to formalize, even when it is easy for an analyst to know a good clustering when they see one. We presen…
New algorithm achieves faster multicalibration in online settings.
problem Achieving fair predictions in online settings with limited interaction.
method Generalized multicalibration to arbitrary bounded hypothesis classes and introduced swap multicalibration. Proposed an oracle-efficient algorithm achieving faster multicalibration error.
result Oracle-efficient algorithm achieves T 1 / ( r + 1 ) T^{1/(r+1)} T 1/ ( r + 1 ) ℓ r \ell_r ℓ r -swap multicalibration error with high probability, improving on previous bounds. Crowdsourced wisdom improves causal learning.
problem Improving causal learning through collective intelligence.
method Crowdsourcing, expert knowledge elicitation, aggregation techniques, and LLMs.
result Collective contributions enhance global causal structure.
Gradient optimization improves preference elicitation for large item spaces.
problem Computational infeasibility of EVOI for large item spaces in recommender systems.
method Continuous formulation of EVOI as a differentiable network, optimized using gradient methods.
result Gradient-based EVOI optimization achieves state-of-the-art performance and scalability.
This paper develops a new method for eliciting more flexible metrics, improving fairness and applicability.
problem Limited flexibility in existing metric elicitation strategies for reflecting user preferences.
method Develops a strategy for eliciting quadratic metrics based on predictive rates, requiring only relative preference feedback.
result Achieves near-optimal query complexity and broadens the use cases for metric elicitation.
Constructs new elicitable risk measures with multiplicative scoring functions.
problem Defining new elicitable risk measures with specific properties.
method Constructs new elicitable risk measures using a multiplicative scoring function.
result Encompasses and allows construction of novel elicitable risk measures.
The paper proposes a method to improve prediction accuracy by querying expert knowledge sequentially.
problem Prediction in high-dimensional settings with limited samples and costly expert consultation.
method Formulates knowledge elicitation as a probabilistic inference process, sequentially querying experts to improve predictions.
result The method shows improved prediction accuracy with minimal expert effort.
New game design method for better trait inference.
problem Inferring latent psychological traits from human behavior.
method Formulated as a mutual information maximization problem, solved using variational lower bound optimization.
result Designed games successfully distinguish among players with different traits, outperforming traditional methods.
A property, or statistical functional, is said to be elicitable if it minimizes expected loss for some loss function. The study of which properties are elicitable sheds light on the capabilities and limitations of point estimation and empirical risk minimization. While recent work asks which properties are elicitable, …
Proposes a method to select fair performance metrics through metric elicitation.
problem Choosing fair performance metrics in multiclass classification with multiple sensitive groups.
method Metric elicitation strategy that requires only relative preference feedback and is robust to noise.
result Elicits group-fair performance metrics for multiclass classification problems.
We discuss equivalent axiomatic characterizations of distortion risk measures, and give a novel and concise proof of the characterization of elicitable distortion risk measures. Elicitability has recently been discussed as a desirable criterion for risk measures, motivated by statistical considerations of forecasting. …
This paper tackles sample elicitation for learning systems, introducing a method to incentivize truthful samples.
problem Eliciting credible training samples for complex distributions from humans is challenging.
method Introduces a deep learning aided method to incentivize truthful samples from self-interested and rational agents.
result Achieves approximate incentive compatibility in eliciting truthful samples via accurate estimation of f f f -divergence function. Elicit performance metrics from classifier comparisons.
problem Discover the performance metric a practitioner prefers for binary classification.
method Formalize and exploit geometric properties of confusion matrices for efficient metric elicitation.
result Provably efficient algorithms for eliciting linear and linear-fractional metrics from pairwise feedback.
Robustifies elicitable functionals to handle small distribution misspecifications.
problem Determining uniquely optimal forecasts under distributional misspecification.
method Integrates statistical robustness into elicitable functionals using Kullback-Leibler divergence.
result Robust elicitable functionals admit unique solutions at the boundary of uncertainty regions.
Game teaches ASD teenagers to imitate gestures.
problem Improving social interactions of ASD teenagers.
method Adapted robot-based imitation game with machine learning for pose detection.
result Preliminary results show game can elicit expected responses.
Study generalizes property elicitation to imprecise probabilities.
problem Minimizing risk over imprecise probability distributions.
method Maximin risk minimization over a set of imprecise probabilities.
result Conditions for elicitability of IP-properties.
A statistical functional, such as the mean or the median, is called elicitable if there is a scoring function or loss function such that the correct forecast of the functional is the unique minimizer of the expected score. Such scoring functions are called strictly consistent for the functional. The elicitability of a …
Study creates web interface to elicit user-preferred metrics.
problem Eliciting classification metrics that align with user preferences.
method Developed a web-based interface and conducted a user study.
result Users preferred metrics that align with their task and context.
Robo-advisors estimate clients' risk aversion using interactive questionnaires.
problem Estimating risk aversion of non-expert clients using adaptive questionnaires.
method Model risk aversion with cost functions and spectral risk measures. Use inverse reinforcement learning to design questions maximizing distinguishing power.
result Designing questions by maximizing distinguishing power achieves satisfactory accuracy in learning risk aversion with fewer than 50 questions.
A method for eliciting expert beliefs using preferential questions and normalizing flows.
problem Eliciting high-dimensional probability distributions from noisy judgments.
method Normalizing flows based on preferential questions with a novel functional prior.
result The method allows for the inference of arbitrarily flexible densities from preferential judgments.
Framework for eliciting fairness constraints from stakeholders.
problem Complex, nuanced fairness requirements not captured by simple definitions.
method Algorithm for learning accurate models subject to elicited fairness constraints.
result Provably convergent and oracle efficient algorithm with generalization bounds.
Cost-effective framework for eliciting and aggregating preferences.
problem Eliciting preferences efficiently under budget constraints.
method Iterative computation of cost-effective questions using Plackett-Luce model and various information criteria.
result Carefully designed information criteria lead to more accurate predictions with fewer questions.
The paper analyzes elicitability of return risk measures and their scoring functions.
problem Elicitability of return risk measures and their scoring functions.
method Dual representation results for convex and geometrically convex return risk measures, axiomatic characterizations of Orlicz premia, and construction of strictly consistent scoring functions.
result Orlicz premia are the only elicitable return risk measures under different sets of conditions.
Proposes method for eliciting non-parametric joint priors using normalizing flows.
problem Learning complex non-parametric joint priors for model parameters.
method Expert elicitation combined with normalizing flows for generative modeling.
result Framework supports elicitation of both parametric and non-parametric priors.
This thesis formalizes metric selection for machine learning applications.
problem Selecting the right performance metric for machine learning models.
method Metric Elicitation framework using pairwise preference feedback.
result Novel strategies for eliciting various classification metrics robust to noise.
The risk of a financial position is usually summarized by a risk measure. As this risk measure has to be estimated from historical data, it is important to be able to verify and compare competing estimation procedures. In statistical decision theory, risk measures for which such verification and comparison is possible,…
Optimizes trading in CFMMs and exchanges using deep learning.
problem Optimizing trading strategies in CFMMs and exchanges.
method Develops a model accounting for interaction between CFMMs and exchanges, employs deep Galerkin method to solve dynamic programming equation.
result Optimal strategy outperforms naïve strategies and is not prone to price slippage.
Develops a simulation-based method to translate expert knowledge into prior distributions for Bayesian models.
problem Effective incorporation of expert knowledge into prior distributions for diverse model structures.
method Simulation-based stochastic gradient descent to learn hyperparameters of parametric priors from expert knowledge.
result Method is adaptable to various elicitation techniques and independent of model structure.
Paper establishes identifiability and elicitability of tail risk measures.
problem Identifying and measuring tail risk measures accurately.
method Establishes identifiability and elicitability of tail risk measures using generators and quantiles.
result Joint identifiability and elicitability of tail risk measures and quantiles.
This paper improves prediction in small data sets by eliciting expert knowledge about feature similarities.
problem Improving predictive models from small high-dimensional data sets.
method Eliciting expert knowledge about pairwise feature similarities and using sequential decision making techniques.
result Improvement in predictive performance on both simulated and real data.
In this note, we comment on the relevance of elicitability for backtesting risk measure estimates. In particular, we propose the use of Diebold-Mariano tests, and show how they can be implemented for Expected Shortfall (ES), based on the recent result of Fissler and Ziegel (2015) that ES is jointly elicitable with Valu…
Framework uses IRL and RL to elicit and optimize risk preferences robustly to noise.
problem Eliciting and optimizing risk preferences in noisy environments.
method Adaptive Bayesian IRL for elicitation, model-free RL for optimization, using quantile networks.
result Framework achieves convergence rate of O ( exp ( − c m + O ( m log m ) ) ) O(\exp(-cm+O(\sqrt{m\log m}))) O ( exp ( − c m + O ( m log m ))) . The paper introduces a method to incorporate expert opinion on observable quantities into statistical models.
problem Tackling the challenge of integrating expert knowledge on observable quantities into statistical models.
method The approach involves updating a prior belief using a loss function that reflects expert opinion on observable quantities.
result The method allows for a flexible specification of expert opinion and is straightforward to implement.
New method allows backtesting of systemic risk forecasts.
problem Systemic risk measures are not elitable and identifiable, making backtesting impossible.
method Introduces multi-objective elicitability and Diebold--Mariano type tests.
result Proposes a traffic-light approach for backtesting.
New method ranks European countries' innovation performance considering criterion interactions.
problem Lack of consensus on weighting composite innovation indicators.
method Hierarchical-SMAA-Choquet integral approach to rank and benchmark innovation performance.
result Robust measurement of innovation performances in Europe with a hierarchy of interacting composite indicators.
Formulates a Dueling Bandits problem for eliciting Kemeny rankings.
problem Eliciting preferences to find a Kemeny ranking.
method Formulates the problem as a Dueling Bandits problem, considering sampling with and without replacement.
result Approximation bounds and algorithms for finding PAC solutions with sample complexity.
Paper proposes incentives for federated learning to ensure truthful contributions.
problem Ensuring truthful contributions from decentralized users in federated learning.
method Introduces a scoring rule based framework to incentivize truthful reporting of local hypotheses at a Bayesian Nash Equilibrium.
result Proposed solution verified using MNIST and CIFAR-10 datasets, showing decreasing scores for low-quality hypotheses.
A framework for eliciting utility functions from investor preferences.
problem Hard elicitation of specific utility functions in portfolio selection.
method Preference-fitting method using probability-wealth pairs and PHARA approximation.
result Fitted utility function converges to the optimal one as more data is used.
Interactive learning with hindsight instruction feedback achieves better performance than traditional methods.
problem Learning with expert supervision of optimal responses is often impractical or requires expert knowledge.
method Study of interactive learning with hindsight instruction feedback, introducing LORIL algorithm.
result LORIL algorithm achieves T \sqrt{T} T regret, improving over baselines in domains with low-rank instruction-response distribution. This paper improves decision support in multi-objective planning by better eliciting user preferences.
problem Determining optimal policies from user preference profiles in multi-objective decision making.
method Extending Gaussian process and pairwise comparison methods to multi-objective scenarios, proposing new ordered preference elicitation strategies.
result Proposed elicitation strategies outperform existing methods and users prefer ranking.
Study uses property elicitation to understand how fairness regularizers affect optimal decisions.
problem Understanding how fairness regularizers change the optimal decision in predictive algorithms.
method Property elicitation to analyze the relationship between loss, regularization, and optimal decision.
result Necessary and sufficient condition for when a property changes with the addition of a regularizer.