We propose a new probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings. We devise an active learning/adaptive testing scheme based on a greedy minimization of expected model entro…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,982 papers · 148 categories
Trend · papers per month
48 results for “test questions”
Unified QuesNet learns comprehensive representations for diverse test questions.
problem Lack of labeled data for test questions in online learning systems.
method Unified framework and two-level hierarchical pre-training algorithm for unsupervised learning of heterogeneous question representations.
result QuesNet effectively learns comprehensive question representations and outperforms existing methods.
Study evaluates how well question-answering models generalize to new data types.
problem Generalization of question-answering models to new data types.
method Constructed new test sets from different domains and evaluated models' performance.
result Models show significant performance drops when tested on new data types.
Turing test relevance questioned in age of AI.
problem Relevance of Turing test in AI age.
method Examining cultural and technological context.
result Turing test may not fully capture AI capabilities.
The paper proposes incentivizing human annotators with 'golden questions' to improve data quality.
problem Ensuring high-quality human annotations for training large language models.
method A principal-agent model is used to incentivize annotators with bonuses based on the maximum likelihood estimators (MLE) of their annotations. Hypothesis testing is applied to monitor the annotators' performance.
result The hypothesis testing rate for the principal-agent model is of , highlighting the importance of 'golden questions' for monitoring annotators.
Paper analyzes biases in video QA datasets, showing models can answer 37-48% questions correctly without multimodal context.
problem Question answering biases in video QA datasets can lead to model overfitting and poor generalization.
method Analyzed popular video question answering datasets, conducted ablation studies on biases from annotators and question types.
result Pretrained language models can answer 37-48% questions correctly without multimodal context, far exceeding random guess baseline.
First Proofcs.AI
AI tested on 10 math questions from research.
problem Assessing AI's ability to solve research-level math problems.
method Shared 10 math questions not previously publicly available.
result Answers to questions are known to authors but encrypted.
Assessing symmetry of financial returns seriesphysics.data-an
Testing symmetry of a probability distribution is a common question arising from applications in several fields. Particularly, in the study of observables used in the analysis of stock market index variations, the question of symmetry has not been fully investigated by means of statistical procedures. In this work a di…
New dataset tests mental rotation from single images, improving model understanding of 3D scenes.
problem Understanding how a scene looks from a different viewpoint using a single image.
method Created CLEVR-MRT dataset, explored neural architectures for volumetric scene representations.
result Demonstrated the effectiveness of volumetric representations in answering mental rotation questions.
Framework evaluates medical image classification methods using a probabilistic model and a Twenty Questions paradigm.
problem Lack of detailed understanding of how machine learning methods make decisions in medical imaging.
method Inspired by the Turing Test, uses a probabilistic model and a Twenty Questions paradigm to evaluate classification methods based on task-specific concepts.
result Probabilistic model exposes dataset and method biases, reducing the number of queries needed for confident performance evaluation.
FactTest assesses LLM factuality with Type I error control.
problem Lack of rigorous factuality verification for LLMs.
method Formulates factuality testing as hypothesis testing, ensuring Type I and II error control.
result Improves model accuracy by over 40% in abstaining from unknown questions.
We develop a simple test for deviations from power law tails, which is based on the asymptotic properties of the empirical distribution function. We use this test to answer the question whether great natural disasters, financial crashes or electricity price spikes should be classified as dragon kings or 'only' as black…
Community-based Question Answering (CQA) sites play an important role in addressing health information needs. However, a significant number of posted questions remain unanswered. Automatically answering the posted questions can provide a useful source of information for online health communities. In this study, we deve…
Improved KSD test for faster GoF testing.
problem Slow and computationally intractable KSD tests.
method Nyström acceleration for KSD estimation.
result Asymptotic properties preserved by Nyström acceleration.
New robustness test for kernel goodness-of-fit tests.
problem Lack of robustness in existing kernel goodness-of-fit tests.
method Proposes a new robust kernel goodness-of-fit test using kernel Stein discrepancy (KSD) balls.
result First robust kernel goodness-of-fit test addressing both qualitative and quantitative robustness.
Optimal ability estimation in adaptive testing with binary responses.
problem Estimating a continuous ability parameter from sequential binary responses.
method Adaptive selection of questions to maximize Fisher information, updating estimate using method-of-moments, and deciding accuracy with a test statistic.
result Fisher-tracking strategy achieves optimal performance in fixed-confidence and fixed-budget regimes.
FinanceBench benchmarks LLMs on financial QA, revealing limitations.
problem Evaluating LLMs' performance on financial question answering.
method Developed a comprehensive test suite (FinanceBench) with 10,231 questions, tested 16 models, and manually reviewed answers.
result Existing LLMs have significant limitations for financial QA, especially GPT-4-Turbo.
Study uses neural networks to predict stock prices and tests market efficiency.
problem Predicting stock prices from historical data.
method Used Recurrent Neural Networks and Multilayer Perceptrons, compared normalization techniques.
result Found that neural networks can predict stock prices accurately and challenged the efficient-market hypothesis.
Drawing on some recent results that provide the formalism necessary to definite stationarity for infinite random graphs, this paper initiates the study of statistical and learning questions pertaining to these objects. Specifically, a criterion for the existence of a consistent test for complex hypotheses is presented,…
New framework to test neural network representation similarity measures.
problem Disagreements among dissimilarity measures in neural networks.
method Statistical testing framework to evaluate measures based on functional behavior.
result Current metrics have different weaknesses; a classical baseline performs surprisingly well.
Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?stat.ML
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a qu…
Paper tackles Arabic question similarity, outperforming state-of-the-art.
problem Detecting semantically similar questions in Arabic is challenging.
method Utilizes contextualized word representations (ELMo embeddings) trained on MSA and dialectic sentences, combined with a pairwise similarity layer.
result Achieves 93% F1-score on Modern Standard Arabic benchmark and 82% on dialectical benchmark.
In this paper we settle Thurston's old question of whether the Weber-Seifert dodecahedral space is non-Haken, a problem that has been a benchmark for progress in computational 3-manifold topology over recent decades. We resolve this question by combining recent significant advances in normal surface enumeration, new he…
Simpler, faster algorithm for uniformity testing in the shuffle model.
problem Testing uniformity of data in the shuffle model with privacy constraints.
method Simplified analysis and use of privacy amplification via shuffling.
result An algorithm with the same guarantees but simpler and more streamlined.
New framework limits testing algorithmic stability under computational constraints.
problem Testing algorithmic stability is computationally hard.
method Unified framework for quantifying stability hardness.
result Exhaustive search is the only universally valid mechanism for certifying stability.
Hypothesis testing for graphs has been an important tool in applied research fields for more than two decades, and still remains a challenging problem as one often needs to draw inference from few replicates of large graphs. Recent studies in statistics and learning theory have provided some theoretical insights about …
New attacks show ML models can be compromised even when targeting one concept.
problem Vulnerability of ML models to multi-concept attacks.
method Developed novel multi-concept attack techniques for deep learning.
result Successfully attacked one set of classifiers without impacting others.
Study efficient sequential evaluation of large language models using historical data.
problem Sequentially evaluate a new large language model (LLM) on a fixed question set.
method Construct a confidence sequence (CS) and design active querying rules to shrink CS width.
result Simple uniform sampling can sometimes outperform adaptive querying rules.
The problem of finding itemsets that are statistically significantly enriched in a class of transactions is complicated by the need to correct for multiple hypothesis testing. Pruning untestable hypotheses was recently proposed as a strategy for this task of significant itemset mining. It was shown to lead to greater s…
Reduces test set maintenance effort by 80-100%.
problem Lack of proper and up-to-date test sets in real-world scenarios.
method Simple technique to reduce labeling effort.
result Significant reduction in test set maintenance effort (80-100%).
The study of networks leads to a wide range of high dimensional inference problems. In many practical applications, one needs to draw inference from one or few large sparse networks. The present paper studies hypothesis testing of graphs in this high-dimensional regime, where the goal is to test between two populations…
New method tests independence in time series data.
problem Testing independence between time series data.
method Temporal dependence statistic with block permutation.
result Asymptotically valid and universally consistent test for independence.
Quriosity: Analyzing Human Questioning Behavior and Causal Inquiry through Curiosity-Driven Queriescs.CL
Quriosity analyzes curiosity-driven questions from diverse sources.
problem Understanding and analyzing human curiosity-driven questions.
method Collection of 13.5K questions from various sources, development of an iterative prompt improvement framework.
result 42% of questions are causal, revealing unique linguistic properties.
Automated vehicles need efficient testing, which this method addresses.
problem Current testing practices are insufficient for automated vehicles.
method Gaussian Process Classification to identify performance boundaries.
result It is feasible to test for challenging scenarios more efficiently.
New method tests mixtures of distributions with fewer samples than previously thought.
problem Testing if a distribution is a mixture of known distributions.
method Noise model where the noisy distribution is a mixture of the original and known noise.
result Sample complexity is the same as for non-mixture cases.
AdaStop improves statistical testing for Deep RL algorithm comparisons.
problem Statistical reproducibility issues in Deep RL.
method AdaStop, a new statistical test based on multiple group sequential tests.
result AdaStop ensures theoretically sound comparisons of Deep RL algorithms.
IRT improves algorithm evaluation across datasets.
problem Evaluating the performance of algorithm portfolios.
method Modified IRT framework for evaluating algorithm portfolios across datasets.
result Richer characteristics of algorithm performance are revealed.
Test assesses if a linear classifier is random or significant.
problem Determining if a linear classifier captures meaningful differences between classes.
method Proposes a homogeneity test related to linear separability, establishes upper bounds for p-values.
result Upper bounds for p-values are highly accurate for normally distributed samples.
MMD test detects adversarial attacks by addressing kernel limitations and non-independence issues.
problem MMD test's failure to detect adversarial attacks.
method Replaced Gaussian kernel with deep kernel, maximized test power, and used wild bootstrap for non-independence.
result MMD test is aware of adversarial attacks.
Improved QA system by re-ranking top-10 results using machine learning.
problem Enhance deployed QA systems without re-training.
method Learn similarity function using n-gram features; implement neural sentence embeddings.
result Mean reciprocal rank improves by 9.15%.
FAT ALBERT: Finding Answers in Large Texts using Semantic Similarity Attention Layer based on BERTcs.CL
Model improves BERT for answering multiple-choice questions in large texts.
problem Improving machine comprehension of large text corpora for question answering.
method Developed a model using BERT with a semantic similarity attention layer to extract key sentences.
result Outperforms leading models in MovieQA challenge with 87.79% test accuracy.
This paper presents the current state of a work in progress, whose objective is to better understand the effects of factors that significantly influence the performance of Latent Semantic Analysis (LSA). A difficult task, which consists in answering (French) biology Multiple Choice Questions, is used to test the semant…
KodeXv0.1 improves financial question answering over GPT-4.
problem Lack of specialized financial language models.
method Custom training on financial documents, RAG-aware 4bit LoRA tuning.
result KodeX-8Bv0.1 outperforms GPT-4 by up to 9.24% in financial tasks.
This paper investigates how policy conditioning affects reinforcement learning stability.
problem Improving stability and generalization of reinforcement learning agents.
method The authors study Jacobian conditioning behavior during policy optimization and propose a conditioning regularization algorithm.
result The proposed conditioning regularization algorithm enhances reinforcement learning agent generalization.
Identity testing for reversible Markov chains without symmetry assumption.
problem Identity testing of reversible Markov chains.
method Using distance notion from Daskalakis et al. [2018a], testing without symmetry assumption.
result It is possible to perform identity testing under weaker assumption of reversibility.
Malaria Likelihood Prediction By Effectively Surveying Households Using Deep Reinforcement Learningcs.LG
We build a deep reinforcement learning (RL) agent that can predict the likelihood of an individual testing positive for malaria by asking questions about their household. The RL agent learns to determine which survey question to ask next and when to stop to make a prediction about their likelihood of malaria based on t…
Deep Learning has managed to push boundaries in a wide variety of tasks. One area of interest is to tackle problems in reasoning and understanding, with an aim to emulate human intelligence. In this work, we describe a deep learning model that addresses the reasoning task of question-answering on categorical plots. We …
Efficient tests achieve best error rates in high-dimensional hypothesis testing.
problem Achieving optimal error rates in computationally efficient hypothesis testing.
method Linear spectral statistics and low-degree likelihood ratio analysis.
result An efficient test achieves the best possible error rates among all computationally efficient tests.