Crowdsourced science questions improve exam accuracy.
problem Generating high-quality, domain-specific multiple choice questions is challenging.
method Leveraging a corpus and existing questions, the method suggests document selection and answer distractors.
result Crowdsourced questions are indistinguishable from original questions and improve exam accuracy.
This paper presents the current state of a work in progress, whose objective is to better understand the effects of factors that significantly influence the performance of Latent Semantic Analysis (LSA). A difficult task, which consists in answering (French) biology Multiple Choice Questions, is used to test the semant…
Model improves BERT for answering multiple-choice questions in large texts.
problem Improving machine comprehension of large text corpora for question answering.
method Developed a model using BERT with a semantic similarity attention layer to extract key sentences.
result Outperforms leading models in MovieQA challenge with 87.79% test accuracy.
This work uses conformal prediction to quantify uncertainty in large language models for multiple-choice questions.
problem Ensuring robustness and reliability of large language models in high-stakes applications.
method Conformal prediction applied to multi-choice question answering.
result Uncertainty estimates from conformal prediction are closely related to prediction accuracy.
Optimizes exam weights for better student assessment.
problem Designing accurate exams with generic question scores.
method Uses machine learning algorithms to adjust exam weights.
result Significant error reduction in exam scores.
R2DE assesses new exam questions quickly and accurately.
problem Assessing newly generated exam questions efficiently and accurately.
method R2DE uses NLP to estimate difficulty and discrimination of questions based on their text.
result R2DE can estimate question difficulty and discrimination quickly and accurately.
Model aggregates answers with peer predictions, inferring world states.
problem Aggregating answers from multiple respondents without assuming consensus correctness.
method Probabilistic model incorporating respondent signals and predictions.
result Model infers world states and respondent expertise, outperforming other models.
New method uses Multiple Choice Learning for speech separation.
problem Ambiguous task of assigning model predictions to ground truth signals.
method Uses Multiple Choice Learning (MCL) instead of Permutation Invariant Training (PIT).
result MCL matches PIT performance but is computationally advantageous.
Study presents MMC model for better fitting multiple choice data.
problem Improving accuracy of latent trait estimates in IRT models.
method Fit autoencoders to MMC model, demonstrating better fit than nominal response model.
result MMC model outperforms traditional IRT models in fit.
rMCL improves on MCL by preserving diversity in predictions for regression problems.
problem Multimodal density estimation in regression settings with multiple targets.
method rMCL uses a learned scoring scheme based on Voronoi tessellations to maintain diversity among predictions.
result rMCL outperforms existing MCL variants in sound source localization tasks.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
aMCL uses annealing to improve hypothesis diversity in ambiguous tasks.
problem Limitations of Winner-takes-all in predicting plausible hypotheses.
method Combines simulated annealing with Multiple Choice Learning (MCL).
result Enhanced exploration of hypothesis space during training.
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
ForecastQA creates a new QA task for event forecasting from text data.
problem Forecasting future events from unstructured text data.
method Formulated a restricted-domain, multiple-choice QA task for event forecasting.
result Best model achieves 60.1% accuracy, lagging behind human performance by about 19%
CROQ optimizes LLM decision-making by narrowing down choices and improving accuracy.
problem Uncertainty in LLM outputs poses risks in high-stakes domains.
method Conformal prediction (CP) and optimization (CP-OPT) to minimize prediction set sizes.
result CROQ improves LLM accuracy, especially with CP-OPT.
New CMCL method reduces errors in deep neural network tasks.
problem Overconfidence in ensemble methods for deep neural networks.
method Confident Multiple Choice Learning (CMCL) with new loss, architecture, and training method.
result CMCL reduces top-1 error rates by 14.05% on CIFAR and 6.60% on SVHN.
Reasoning models outperform LLMs on CFA exams.
problem Previous research showed LLMs failing CFA exams; reasoning models show promise.
method Evaluated state-of-the-art reasoning models on CFA exams using pass/fail criteria.
result Most reasoning models pass all three CFA levels; Gemini 3.0 Pro achieves highest scores.
Paper classifies brain signals using eigenvalues for 2D and 3D educational content questions.
problem Classifying brain signals for 2D and 3D educational content questions.
method Eigenvalues of covariance matrix used as features; KNN and SVM classifiers applied.
result No significant difference in learning, memory retention, and recall between 2D and 3D educational content.
We propose a generic model for multiple choice situations in the presence of herding and compare it with recent empirical results from a Web-based music market experiment. The model predicts a phase transition between a weak imitation phase and a strong imitation, `fashion' phase, where choices are driven by peer press…
New method quantifies uncertainty in fine-tuned LLMs using LoRA ensembles.
problem Uncertainty in fine-tuned LLMs and how to trust their predictions.
method Posterior approximations using low-rank adaptation ensembles.
result Unexpected retention of acquired knowledge during fine-tuning in overfitting regime.
TimeMCL forecasts diverse time series futures using neural networks and WTA loss.
problem Forecasting multiple plausible time series futures.
method Multiple Choice Learning (MCL) framework with Winner-Takes-All (WTA) loss.
result TimeMCL efficiently predicts diverse time series futures at low computational cost.
FactTest assesses LLM factuality with Type I error control.
problem Lack of rigorous factuality verification for LLMs.
method Formulates factuality testing as hypothesis testing, ensuring Type I and II error control.
result Improves model accuracy by over 40% in abstaining from unknown questions.
Item response theory (IRT) models for categorical response data are widely used in the analysis of educational data, computerized adaptive testing, and psychological surveys. However, most IRT models rely on both the assumption that categories are strictly ordered and the assumption that this ordering is known a priori…
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
Develops CPL for optimal prediction set length and validity.
problem Balancing conditional validity and length efficiency in conformal prediction.
method Conformal Prediction with Length-Optimization (CPL).
result Achieves optimal prediction set length while maintaining conditional validity.
Proposes a new method to minimize non-singleton predictions in conformal prediction.
problem Large prediction sets in conformal prediction are costly and inefficient.
method Introduces a new nonconformity score to minimize non-singleton sets and provides an algorithm to compute it efficiently.
result The proposed Singleton-Optimized Conformal Prediction (SOCOP) method increases singleton frequency by over 20% compared to standard scores, with minimal impact on average set size.
This paper optimizes ad bids and daily budgets for multiple campaigns in pay-per-click advertising.
problem Optimizing ad bids and daily budgets for multiple campaigns in pay-per-click advertising.
method Formulated as a combinatorial semi-bandit problem, solved using Gaussian Processes and four algorithms.
result Regret upper bounded as O(sqrt{T}), where T is the time horizon.
LLMs struggle to generate random numbers from statistical distributions, leading to biased results in applications.
problem LLMs' inability to generate random numbers accurately from specified distributions.
method Dual-protocol design: Batch Generation and Independent Requests, benchmarking 11 models across 15 distributions.
result Sampling fidelity degrades with distributional complexity and horizon, leading to systematic biases in downstream applications.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
GraphITE estimates individual effects of graph-structured treatments.
problem Estimating individual effects of complex treatment structures.
method Graph neural networks and Hilbert-Schmidt Independence Criterion regularization.
result GraphITE outperforms baselines in estimating treatment effects for large numbers of treatments.
New optimization method improves generalization across various tasks.
problem Improving zeroth-order optimization for better generalization.
method Exponential tilting objective to connect zeroth-order optimization with sharpness-aware minimization.
result Achieves better generalization compared to vanilla zeroth-order baselines.
A trainable gate optimizes neural network selection and pruning.
problem Optimizing neural networks for specific tasks.
method Introduces a trainable gate function to make discrete selection problems differentiable.
result Efficiently optimizes arbitrary neural networks across various tasks.
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of H-consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings. result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
Market confidence is essential for successful investing. By incorporating multi-market into the evolutionary minority game, we investigate the effects of investor beliefs on the evolution of collective behaviors and asset prices. When there exists another investment opportunity, market confidence, including overconfide…
Hybrid QC system for Bengali questions using smart data balancing.
problem Classifying factoid questions in Bengali.
method Two-stage approach with 1D CNN for coarse classification and SGD for fine classification.
result Effectiveness of smart data balancing technique in improving classification accuracy.
Paper tackles medical question similarity using domain-relevant embeddings.
problem Identifying same-question pairs in medical contexts.
method Semi-supervised pre-training of a neural network on medical question-answer pairs.
result Our model achieves 82.6% accuracy on medical question similarity task.
Paper evaluates Arabic question similarity, 9 teams participated.
problem Predicting semantic similarity between Arabic questions.
method 9 teams participated in a shared task to predict semantic similarity.
result 9 teams participated in the task, results made publicly available.
Paper proposes a neural network for generating better questions from text.
problem Automatic generation of relevant questions from sentences and paragraphs.
method Adaptive copying recurrent neural network model with a copying mechanism added to a bidirectional LSTM architecture.
result The model outperforms state-of-the-art methods in question generation metrics.
Paper tackles Arabic question similarity, outperforming state-of-the-art.
problem Detecting semantically similar questions in Arabic is challenging.
method Utilizes contextualized word representations (ELMo embeddings) trained on MSA and dialectic sentences, combined with a pairwise similarity layer.
result Achieves 93% F1-score on Modern Standard Arabic benchmark and 82% on dialectical benchmark.
Online surveys have the potential to support adaptive questions, where later questions depend on earlier responses. Past work has taken a rule-based approach, uniformly across all respondents. We envision a richer interpretation of adaptive questions, which we call dynamic question ordering (DQO), where question order …
MAC Net improves natural language question answering with data-driven reasoning.
problem Natural Language Question Answering requires complex reasoning.
method MAC Net architecture separates memory and control for iterative reasoning.
result MAC Net achieves high efficiency and interpretability in NLP tasks.
Unified QuesNet learns comprehensive representations for diverse test questions.
problem Lack of labeled data for test questions in online learning systems.
method Unified framework and two-level hierarchical pre-training algorithm for unsupervised learning of heterogeneous question representations.
result QuesNet effectively learns comprehensive question representations and outperforms existing methods.
James's octonionic Stiefel spaces questions answered partially.
problem Two fundamental questions about octonionic Stiefel spaces.
method Partial answers to James's questions about octonionic Stiefel spaces.
result Partial answers to James's questions about octonionic Stiefel spaces.
AI tested on 10 math questions from research.
problem Assessing AI's ability to solve research-level math problems.
method Shared 10 math questions not previously publicly available.
result Answers to questions are known to authors but encrypted.
Predicts student performance in interactive online question pools using GNNs.
problem Predicting student performance in interactive online question pools with evolving knowledge.
method Proposes R^2GCN, a GNN model for heterogeneous networks to predict student performance.
result Achieves higher accuracy in student performance prediction than traditional methods.
Proposes CNN Inception + Gate model for visual question answering.
problem Deep understanding of images and texts for visual question answering.
method Proposes a CNN Inception + Gate model for learning textual representations.
result Improves question representations and overall accuracy in visual question answering.
DiffQue estimates relative difficulty of questions in CQA services.
problem Estimating relative difficulty of questions in community Q&A services.
method Network-aided edge directionality prediction.
result DiffQue outperforms state-of-the-art methods by significant margins.
Algorithm transfers visual concepts to answer out-of-vocabulary questions.
problem Leveraging off-the-shelf visual and linguistic data for out-of-vocabulary answers in visual question answering.
method Unsupervised task discovery for learning task conditional visual classifier, then transferring to visual question answering models.
result Algorithm generalizes to out-of-vocabulary answers successfully.