We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-speci…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “multiple-choice datasets”
aMCL uses annealing to improve hypothesis diversity in ambiguous tasks.
problem Limitations of Winner-takes-all in predicting plausible hypotheses.
method Combines simulated annealing with Multiple Choice Learning (MCL).
result Enhanced exploration of hypothesis space during training.
New method quantifies uncertainty in fine-tuned LLMs using LoRA ensembles.
problem Uncertainty in fine-tuned LLMs and how to trust their predictions.
method Posterior approximations using low-rank adaptation ensembles.
result Unexpected retention of acquired knowledge during fine-tuning in overfitting regime.
New method uses Multiple Choice Learning for speech separation.
problem Ambiguous task of assigning model predictions to ground truth signals.
method Uses Multiple Choice Learning (MCL) instead of Permutation Invariant Training (PIT).
result MCL matches PIT performance but is computationally advantageous.
Study presents MMC model for better fitting multiple choice data.
problem Improving accuracy of latent trait estimates in IRT models.
method Fit autoencoders to MMC model, demonstrating better fit than nominal response model.
result MMC model outperforms traditional IRT models in fit.
rMCL improves on MCL by preserving diversity in predictions for regression problems.
problem Multimodal density estimation in regression settings with multiple targets.
method rMCL uses a learned scoring scheme based on Voronoi tessellations to maintain diversity among predictions.
result rMCL outperforms existing MCL variants in sound source localization tasks.
This paper presents the current state of a work in progress, whose objective is to better understand the effects of factors that significantly influence the performance of Latent Semantic Analysis (LSA). A difficult task, which consists in answering (French) biology Multiple Choice Questions, is used to test the semant…
LoRA-MCL improves language models by generating diverse sentence continuations.
problem Language models struggle with generating diverse, plausible sentence continuations.
method Low-Rank Adaptation combined with Multiple Choice Learning (MCL) to handle ambiguity.
result LoRA-MCL generates high-diversity and relevant outputs in various tasks.
Item response theory (IRT) models for categorical response data are widely used in the analysis of educational data, computerized adaptive testing, and psychological surveys. However, most IRT models rely on both the assumption that categories are strictly ordered and the assumption that this ordering is known a priori…
Ensemble methods are arguably the most trustworthy techniques for boosting the performance of machine learning models. Popular independent ensembles (IE) relying on naive averaging/voting scheme have been of typical choice for most applications involving deep neural networks, but they do not consider advanced collabora…
FAT ALBERT: Finding Answers in Large Texts using Semantic Similarity Attention Layer based on BERTcs.CL
Model improves BERT for answering multiple-choice questions in large texts.
problem Improving machine comprehension of large text corpora for question answering.
method Developed a model using BERT with a semantic similarity attention layer to extract key sentences.
result Outperforms leading models in MovieQA challenge with 87.79% test accuracy.
Of Songs and Men: a Model for Multiple Choice with Herdingphysics.data-an
We propose a generic model for multiple choice situations in the presence of herding and compare it with recent empirical results from a Web-based music market experiment. The model predicts a phase transition between a weak imitation phase and a strong imitation, `fashion' phase, where choices are driven by peer press…
TimeMCL forecasts diverse time series futures using neural networks and WTA loss.
problem Forecasting multiple plausible time series futures.
method Multiple Choice Learning (MCL) framework with Winner-Takes-All (WTA) loss.
result TimeMCL efficiently predicts diverse time series futures at low computational cost.
This work uses conformal prediction to quantify uncertainty in large language models for multiple-choice questions.
problem Ensuring robustness and reliability of large language models in high-stakes applications.
method Conformal prediction applied to multi-choice question answering.
result Uncertainty estimates from conformal prediction are closely related to prediction accuracy.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
ForecastQA creates a new QA task for event forecasting from text data.
problem Forecasting future events from unstructured text data.
method Formulated a restricted-domain, multiple-choice QA task for event forecasting.
result Best model achieves 60.1% accuracy, lagging behind human performance by about 19%
GraphITE estimates individual effects of graph-structured treatments.
problem Estimating individual effects of complex treatment structures.
method Graph neural networks and Hilbert-Schmidt Independence Criterion regularization.
result GraphITE outperforms baselines in estimating treatment effects for large numbers of treatments.
This paper optimizes ad bids and daily budgets for multiple campaigns in pay-per-click advertising.
problem Optimizing ad bids and daily budgets for multiple campaigns in pay-per-click advertising.
method Formulated as a combinatorial semi-bandit problem, solved using Gaussian Processes and four algorithms.
result Regret upper bounded as O(sqrt{T}), where T is the time horizon.
Develops CPL for optimal prediction set length and validity.
problem Balancing conditional validity and length efficiency in conformal prediction.
method Conformal Prediction with Length-Optimization (CPL).
result Achieves optimal prediction set length while maintaining conditional validity.
The main objective of exams consists in performing an assessment of students' expertise on a specific subject. Such expertise, also referred to as skill or knowledge level, can then be leveraged in different ways (e.g., to assign a grade to the students, to understand whether a student might need some support, etc.). S…
CROQ optimizes LLM decision-making by narrowing down choices and improving accuracy.
problem Uncertainty in LLM outputs poses risks in high-stakes domains.
method Conformal prediction (CP) and optimization (CP-OPT) to minimize prediction set sizes.
result CROQ improves LLM accuracy, especially with CP-OPT.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
A problem faced by many instructors is that of designing exams that accurately assess the abilities of the students. Typically these exams are prepared several days in advance, and generic question scores are used based on rough approximation of the question difficulty and length. For example, for a recent class taught…
We propose a probabilistic model to aggregate the answers of respondents answering multiple-choice questions. The model does not assume that everyone has access to the same information, and so does not assume that the consensus answer is correct. Instead, it infers the most probable world state, even if only a minority…
Constructing a Portfolio Optimization Benchmark Framework for Evaluating Large Language Modelsq-fin.PM
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
Architecture optimization, which is a technique for finding an efficient neural network that meets certain requirements, generally reduces to a set of multiple-choice selection problems among alternative sub-structures or parameters. The discrete nature of the selection problem, however, makes this optimization difficu…
New optimization method improves generalization across various tasks.
problem Improving zeroth-order optimization for better generalization.
method Exponential tilting objective to connect zeroth-order optimization with sharpness-aware minimization.
result Achieves better generalization compared to vanilla zeroth-order baselines.
In this paper, we have proposed a brain signal classification method, which uses eigenvalues of the covariance matrix as features to classify images (topomaps) created from the brain signals. The signals are recorded during the answering of 2D and 3D questions. The system is used to classify the correct and incorrect a…
Reasoning models outperform LLMs on CFA exams.
problem Previous research showed LLMs failing CFA exams; reasoning models show promise.
method Evaluated state-of-the-art reasoning models on CFA exams using pass/fail criteria.
result Most reasoning models pass all three CFA levels; Gemini 3.0 Pro achieves highest scores.
Proposes a new method to minimize non-singleton predictions in conformal prediction.
problem Large prediction sets in conformal prediction are costly and inefficient.
method Introduces a new nonconformity score to minimize non-singleton sets and provides an algorithm to compute it efficiently.
result The proposed Singleton-Optimized Conformal Prediction (SOCOP) method increases singleton frequency by over 20% compared to standard scores, with minimal impact on average set size.
FactTest assesses LLM factuality with Type I error control.
problem Lack of rigorous factuality verification for LLMs.
method Formulates factuality testing as hypothesis testing, ensuring Type I and II error control.
result Improves model accuracy by over 40% in abstaining from unknown questions.
LLMs struggle to generate random numbers from statistical distributions, leading to biased results in applications.
problem LLMs' inability to generate random numbers accurately from specified distributions.
method Dual-protocol design: Batch Generation and Independent Requests, benchmarking 11 models across 15 distributions.
result Sampling fidelity degrades with distributional complexity and horizon, leading to systematic biases in downstream applications.
Study of estimation errors in surrogate loss minimizers, providing stronger guarantees than existing methods.
problem Estimation errors in surrogate loss minimizers for various hypothesis sets.
method Detailed study of -consistency estimation error bounds, proving general theorems for distribution-dependent and independent settings.
result Explicit bounds for zero-one and adversarial losses, showing enhancements under distributional assumptions.
Market confidence is essential for successful investing. By incorporating multi-market into the evolutionary minority game, we investigate the effects of investor beliefs on the evolution of collective behaviors and asset prices. When there exists another investment opportunity, market confidence, including overconfide…
Dependent MMD coresets help compare multiple related datasets.
problem Comparing multiple related datasets for insights into model generalization.
method Dependent MMD coresets for collections of datasets.
result Dependent MMD coresets facilitate comparison and understanding of multiple related datasets.
SCARY dataset generates complex causal scenarios for causality research.
problem Lack of complexity in existing causal datasets.
method Synthetic dataset with 40 scenarios, three seeds, and two data generation mechanisms.
result Provides a valuable resource for realistic causal discovery.
MusPy is a toolkit for symbolic music generation, providing tools for dataset management and analysis.
problem Facilitating the creation and analysis of symbolic music datasets.
method Development of an open-source Python library (MusPy) with features for dataset management, data I/O, preprocessing, and model evaluation. Demonstrated through statistical analysis and cross-dataset generalizability experiments.
result MusPy's dataset analysis reveals varying degrees of cross-genre representation across different music datasets.
In this paper, we propose a data collaboration analysis method for distributed datasets. The proposed method is a centralized machine learning while training datasets and models remain distributed over some institutions. Recently, data became large and distributed with decreasing costs of data collection. If we can cen…
StyleDiff compares unlabeled datasets using disentangled image spaces.
problem Mismatches between development and real-world datasets lead to inaccurate predictions.
method Uses disentangled image spaces and focuses on attributes to compare datasets.
result Accurately detects and presents differences between datasets.
In this paper, we propose a novel method for generating a synthetic dataset obeying Gaussian distribution. Compared to the commonly used benchmark datasets with unknown distribution, the synthetic dataset has an explicit distribution, i.e., Gaussian distribution. Meanwhile, it has the same characteristics as the benchm…
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
This paper introduces a new dataset called "ToyADMOS" designed for anomaly detection in machine operating sounds (ADMOS). To the best our knowledge, no large-scale datasets are available for ADMOS, although large-scale datasets have contributed to recent advancements in acoustic signal processing. This is because anoma…
In this paper, we disseminate a new handwritten digits-dataset, termed Kannada-MNIST, for the Kannada script, that can potentially serve as a direct drop-in replacement for the original MNIST dataset. In addition to this dataset, we disseminate an additional real world handwritten dataset (with images), which we …
Fairness GANstat.ML
In this paper, we introduce the Fairness GAN, an approach for generating a dataset that is plausibly similar to a given multimedia dataset, but is more fair with respect to protected attributes in allocative decision making. We propose a novel auxiliary classifier GAN that strives for demographic parity or equality of …
MTL method uses unlabeled data with pseudo labels to improve classification with disjoint datasets.
problem Improving classification performance with disjoint labeled datasets using unlabeled data.
method Proposes MTL-SA method to select and augment unlabeled data with confident pseudo labels and close distribution to labeled data.
result Extensive experiments show the effectiveness of MTL-SA method in improving classification performance.
The study examines dataset usage patterns in machine learning research.
problem Lack of attention to dataset dynamics in machine learning research.
method Analysis of dataset usage patterns across machine learning subcommunities and time periods (2015-2020).
result Increasing concentration on fewer and fewer datasets, significant adoption from other tasks, and concentration across the field on datasets introduced by elite institutions.
Method embeds numeric tabular datasets into a shared vector space for similarity and retrieval.
problem Lack of meaningful representation for numeric tabular datasets in large language models.
method Structured exploratory data analysis descriptors, sentence transformer embedding, CCA for cross-dataset alignment.
result Total P@1 score of 0.9 across 15 datasets, robust nearest-neighbor retrieval and cluster structure.
This paper presents the recently published Cerema AWP (Adverse Weather Pedestrian) dataset for various machine learning tasks and its exports in machine learning friendly format. We explain why this dataset can be interesting (mainly because it is a greatly controlled and fully annotated image dataset) and present base…