New algorithm finds optimal sample complexity for pure exploration with multiple good answers.
problem Determining the optimal number of samples needed to explore multiple good answers in a bandit problem.
method Derive lower bound using game equilibrium, extend Track-and-Stop algorithm to multiple answers.
result New algorithm has asymptotic sample complexity matching the derived lower bound.
Unified framework for ranking-and-selection with multiple correct answers and non-answerable estimates
problem Fixed-precision ranking-and-selection in structured settings with non-unique answers and non-answerable estimates
method Unified framework based on answer-wise acceptance sets, restricted generalized likelihood ratio stopping, and answer-pitfall decomposition
result Unified recipe performs well across a broad range of pure-exploration problems
Model improves BERT for answering multiple-choice questions in large texts.
problem Improving machine comprehension of large text corpora for question answering.
method Developed a model using BERT with a semantic similarity attention layer to extract key sentences.
result Outperforms leading models in MovieQA challenge with 87.79% test accuracy.
We propose a probabilistic model to aggregate the answers of respondents answering multiple-choice questions. The model does not assume that everyone has access to the same information, and so does not assume that the consensus answer is correct. Instead, it infers the most probable world state, even if only a minority…
This paper presents the current state of a work in progress, whose objective is to better understand the effects of factors that significantly influence the performance of Latent Semantic Analysis (LSA). A difficult task, which consists in answering (French) biology Multiple Choice Questions, is used to test the semant…
We propose a method to efficiently learn diverse strategies in reinforcement learning for query reformulation in the tasks of document retrieval and question answering. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to…
The paper optimizes LLM accuracy by stopping early based on consistent answers.
problem Improving LLM accuracy in math and reasoning problems.
method Bayesian stopping policy to save on sampling costs, tracking only the L-1 most frequent answer counts.
result The L=3 stopping policy is sufficient for asymptotic optimality and significantly reduces inference costs.
End-to-end KBQA system learns from multiple reasoning paths without labeled paths.
problem Lack of labeled reasoning paths limits KBQA system performance.
method End-to-end KBQA system using multiple reasoning paths.
result Demonstrates strong performance on various KBQA datasets.
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a qu…
We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-speci…
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
This work uses conformal prediction to quantify uncertainty in large language models for multiple-choice questions.
problem Ensuring robustness and reliability of large language models in high-stakes applications.
method Conformal prediction applied to multi-choice question answering.
result Uncertainty estimates from conformal prediction are closely related to prediction accuracy.
Goldman (Invent. Math. 85(2) (1986) 263) and Turaev (Ann. Sci. Ecole Norm. Sup. (4) 24 (6)(1991) 635) found a Lie bialgebra structure on the vector space generated by non-trivial free homotopy classes of loops on an orientable surface. Chas (Combinatorial Lie bialgebras of curves on surfaces, Topology 43 (2004) 543), b…
Prefix consistency improves model reliability by weighting answers based on their reproducibility.
problem Improving the reliability of large language models' reasoning traces.
method Use prefix consistency to weight candidate answers based on their reproducibility during regeneration.
result Prefix consistency is the best correctness predictor, reaching Standard MV plateau accuracy with up to 21x fewer tokens.
Most research in reading comprehension has focused on answering questions based on individual documents or even single paragraphs. We introduce a neural model which integrates and reasons relying on information spread within documents and across multiple documents. We frame it as an inference problem on a graph. Mentio…
We prove that the total curvature of any planar graph with nonnegative combinatorial curvature is an integral multiple of 121. As a corollary, this answers a question proposed by T. Réti.
Can humans impute missing data with similar proficiency as machines? This is the question we aim to answer in this paper. We present a novel idea of converting observations with missing data in to a survey questionnaire, which is presented to crowdworkers for completion. We replicate a multiple imputation framework by …
G-SHAP generates multiple types of explanations for machine learning models.
problem Understanding model predictions and their differences across groups.
method Generalization of SHAP method to produce additional types of explanations.
result G-SHAP produces explanations for classification, intergroup differences, and model failure.
CITE algorithm provides anytime-valid certification of model outputs.
problem Challenges in controlling error levels in LLM self-consistency.
method Certification by Intersection-union Testing with E-processes (CITE) algorithm.
result Provable control of false certification at any prescribed level under arbitrary stopping rules.
Machine learning ensemble improves accuracy by considering minority answers as more likely true.
problem Ensemble methods often rely on majority voting, which can fail when the majority is wrong.
method Proposes Bayesian Truth Serum for classification problems, detecting surprising majority answers.
result Better classification performance achieved by considering minority answers as more likely true.
New method learns to answer questions from correct demonstrations without assuming bounded complexity of the demonstrator.
problem Learning to generate answers from correct demonstrations with multiple correct answers.
method Formalizes imitation learning in contextual bandits, relying on reward model complexity, not policy complexity.
result Achieves nearly optimal performance with sample complexity logarithmic in reward class cardinality.
A new approach reduces the size of prediction sets in CP, making it more efficient.
problem Large and costly prediction sets in standard CP.
method Expand correctness criterion to include inferred admissible answers, and use prediction cascades to prune implausible labels.
result Empirically effective in natural language processing and drug discovery.
Crowdsourcing platforms emerged as popular venues for purchasing human intelligence at low cost for large volume of tasks. As many low-paid workers are prone to give noisy answers, a common practice is to add redundancy by assigning multiple workers to each task and then simply average out these answers. However, to fu…
Improves dialogue state tracking across multiple domains.
problem Incomplete domain ontology limits DST models' adaptability.
method Model DST as Q&A, using evolving knowledge graph.
result 5.80% and 12.21% relative improvement on datasets.
The paper proposes a new method to evaluate LLM agent responses using ECDF clustering.
problem The standard evaluation of LLM agent responses via majority voting obscures response quality and distribution.
method The paper introduces a novel evaluation framework based on ECDF of cosine similarities and clustering of ECDFs using distances and k-medoids algorithm. result ECDF clustering reveals interpretable group structures in LLM responses, offering insights into agent settings.
Paper introduces SDM for detecting LLM hallucinations, improving on entropy tests.
problem Challenges of Large Language Models (LLMs) with non-factual, nonsensical responses.
method Joint clustering on sentence embeddings to measure semantic divergence between prompts and responses.
result SDM framework detects deeper form of arbitrariness in LLM responses.
Log-Normal Multiplicative Dynamics improves low-precision training of neural networks.
problem Training large neural networks with low precision is unstable.
method Derive a Bayesian learning rule with log-normal posterior distributions and multiplicative updates.
result LMD achieves stable and accurate training for Vision Transformer and GPT-2.
We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in t…
A method for learning with autoregressive chain-of-thoughts.
problem Learning prompt-to-answer mappings from sequence-to-next-token generators.
method Iterating a fixed, time-invariant generator for multiple steps to generate a chain-of-thought, then taking the final token as the answer.
result Universal representability and computationally tractable chain-of-thought learning for a simple base class.
ForecastQA creates a new QA task for event forecasting from text data.
problem Forecasting future events from unstructured text data.
method Formulated a restricted-domain, multiple-choice QA task for event forecasting.
result Best model achieves 60.1% accuracy, lagging behind human performance by about 19%
New method connects CAT(0) spaces to hyperbolic spaces.
problem Injecting sublinear Morse boundaries into Gromov boundaries.
method Developed curtain machinery to characterize sublinear Morse properties.
result Continuous injection of sublinear Morse boundaries into Gromov boundaries.
Optimal query allocation improves extractive QA efficiency with LLMs.
problem Efficiency and reliability in extractive question answering with LLMs.
method Learning-to-Defer framework that allocates queries to specialized models.
result Enhanced answer reliability with reduced computational overhead.
Inferring the correct answers to binary tasks based on multiple noisy answers in an unsupervised manner has emerged as the canonical question for micro-task crowdsourcing or more generally aggregating opinions. In graphon estimation, one is interested in estimating edge intensities or probabilities between nodes using …
SE-KGE embeds spatial data into KGs for better spatial reasoning.
problem Spatially explicit KG embeddings for geographic tasks.
method Location-aware KG embedding model SE-KGE.
result SE-KGE outperforms baselines on DBGeo dataset.
MT-SGD samples from multiple target distributions using gradient descent.
problem Sampling from multiple unnormalized target distributions.
method Proposes MT-SGD, a flow of intermediate distributions to sample from multiple target distributions.
result Asymptotic analysis shows MT-SGD reduces to multiple-gradient descent for multi-objective optimization.
The paper applies communication theory to improve language model reranking.
problem Ensuring safety and accuracy in language model outputs.
method Drawing parallels between communication theory and language model reranking, the authors propose a protocol to improve reliability.
result The proposed protocol can achieve asymptotically error-free performance in noisy communication scenarios.
In many contexts, we have access to aggregate data, but individual level data is unavailable. For example, medical studies sometimes report only aggregate statistics about disease prevalence because of privacy concerns. Even so, many a time it is desirable, and in fact could be necessary to infer individual level chara…
In one dimension, the theory of the G-normal distribution is well-developed, and many results from the classical setting have a nonlinear counterpart. Significant challenges remain in multiple dimensions, and some of what has already been discovered is quite nonintuitive. By answering several classically-inspired que…
SCROLLS benchmarks long text NLP tasks, improving existing models.
problem Short NLP benchmarks ignore long texts; SCROLLS addresses this.
method Handpicked long-text datasets for summarization, QA, and inference tasks.
result Improvement potential on SCROLLS tasks, as indicated by initial baselines.
When considering answering important questions with data, unsupervised data offers extensive insight opportunity and unique challenges. This study considers student survey data with a specific goal of clustering students into like groups with underlying concept of identifying different poverty levels. Fuzzy logic is co…
In 1973, J. Cheeger and J. Simons raised the following question that still remains open and is known as the Rational Simplex Problem: Given a geodesic simplex in the spherical 3-space so that all of its interior dihedral angles are rational multiples of π, is it true that its volume is a rational multiple of the volu…
Infinitely many 3D shapes have multiple ways to be filled with special surfaces.
problem Understanding how many ways 3D shapes can be filled with special surfaces.
method Examined 3D shapes supported by planar open books and found multiple ways to fill them with special surfaces.
result Found infinitely many 3D shapes that can be filled with multiple, non-homeomorphic special surfaces.
Let M and N be two closed (not necessarily orientable) surfaces, and f a continuous map from M to N. By definition, the minimal multiplicity MMR[f] of the map f denotes the minimal integer k having the following property: f can be deformed into a map g such that the number |g^{-1}(c)| of preimages of any point c in N u…
Study knot invariants to answer questions about slice genus and clasp numbers.
problem Whether the difference between the four-dimensional clasp number and the slice genus can be arbitrarily large.
method Equivariant singular instanton theory and Chern--Simons functional.
result Answers a conjecture by Livingston about slicing numbers and provides a lower bound for the unoriented slice genus.
Paper proposes PSIPS for identifying Pareto set with correlated objectives.
problem Identifying the best answer among items with multiple conflicting metrics.
method Posterior sampling in stopping and sampling rules for structure and correlation.
result PSIPS is asymptotically optimal and demonstrates good empirical performance.
CROQ optimizes LLM decision-making by narrowing down choices and improving accuracy.
problem Uncertainty in LLM outputs poses risks in high-stakes domains.
method Conformal prediction (CP) and optimization (CP-OPT) to minimize prediction set sizes.
result CROQ improves LLM accuracy, especially with CP-OPT.
DiffQue estimates relative difficulty of questions in CQA services.
problem Estimating relative difficulty of questions in community Q&A services.
method Network-aided edge directionality prediction.
result DiffQue outperforms state-of-the-art methods by significant margins.
Method certifies edge predictions with cloud-level reliability.
problem Ensuring reliability of edge intelligence models.
method Conformal alignment-based cascading mechanism.
result Certifies conditional coverage with user control over risk level.