A new algorithm estimates item parameters in item response theory models.
problem Estimating item parameters in item response theory models.
method Computation of the stationary distribution of a Markov chain defined on an item-item graph.
result Our algorithm is consistent and enjoys favorable optimality properties.
Faster, more accurate IRT model for large datasets.
problem Speed and accuracy challenges in fitting IRT models to large datasets.
method Variational Bayesian inference for IRT models.
result Higher log likelihoods and improved missing data imputation.
New algorithm speeds up IRT model fitting for large datasets.
problem Inefficient algorithms for fitting Item Response Theory models to large datasets.
method Stochastic Variational Inference for IRT models.
result Higher log likelihoods and accuracy in imputing missing data.
New Bayesian method for sparse multidimensional item response theory.
problem Sparse interpretable explanations for questionnaire data.
method Bayesian EM algorithm for sparse factor loadings.
result Reliable recovery of factor dimensionality and latent structure.
The paper presents a machine learning approach to multidimensional item response theory.
problem Modeling and predicting student performance from assessment data.
method Inspired by collaborative filtering, the paper defines a general class of models using penalized joint maximum likelihood (JML) for estimation and cross-validation for model selection.
result The high-dimensional model fit to large and sparse data does not lend itself well to traditional factor interpretation.
New estimators improve Rasch model item parameter estimation for sparse data.
problem Estimating item parameters in sparse Rasch model data.
method Random pairing maximum likelihood estimator (RP-MLE) and its bootstrapped variant (MRP-MLE).
result RP-MLE and MRP-MLE are minimax optimal and provide precise item parameter estimates.
FedIRT enables privacy-preserving psychometric estimation without centralizing data.
problem Privacy and data governance concerns in centralized IRT estimation.
method Federated Item Response Theory (FedIRT) and FedIRT-DP for differentially private estimation.
result FedIRT matches accuracy of centralized estimators while preserving privacy.
Proposes β4-IRT, a new IRT model with enhanced discrimination estimation.
problem Symmetry problem in β3-IRT causing incorrect discrimination values. method Gradient descent method, link functions, and new parameters for discrimination.
result Improved parameter recovery in experiments compared to β3-IRT. Study presents MMC model for better fitting multiple choice data.
problem Improving accuracy of latent trait estimates in IRT models.
method Fit autoencoders to MMC model, demonstrating better fit than nominal response model.
result MMC model outperforms traditional IRT models in fit.
New method for scalable learning of IRT models from large datasets.
problem Efficiently learning latent variables in IRT models from large numbers of examinees and items.
method Leveraging logistic regression and coresets for scalable IRT training.
result Scalable learning of IRT models from large data is achieved.
GPIRT uses Gaussian processes to estimate latent traits and IRFs from binary responses.
problem Nonparametric IRT models struggle to estimate flexible IRFs and latent traits simultaneously.
method GPIRT employs Gaussian process priors to relax IRF assumptions while estimating latent traits.
result GPIRT provides a flexible solution to IRT challenges, including active learning.
Ensemble unsupervised anomaly detection using IRT for hidden ground truth.
problem Challenges in constructing an ensemble from unsupervised anomaly detection methods.
method Use Item Response Theory to compute latent traits and construct an ensemble that downplays noisy methods.
result Demonstrated effectiveness of IRT ensemble on extensive data repository.
LaRT models LLMs' response accuracy and CoT length to evaluate reasoning ability and speed.
problem Valid evaluation of Large Language Models (LLMs) via response accuracy and chain-of-thought length.
method Introduces Latency-Response Theory (LaRT) to jointly model response accuracy and CoT length using latent ability and latent speed.
result LaRT yields higher estimation accuracy and shorter confidence intervals for latent traits compared to IRT.
Item Response Theory (IRT) aims to assess latent abilities of respondents based on the correctness of their answers in aptitude test items with different difficulty levels. In this paper, we propose the β3-IRT model, which models continuous responses and can generate a much enriched family of Item Characteristic Cur…
Deep learning based knowledge tracing model has been shown to outperform traditional knowledge tracing model without the need for human-engineered features, yet its parameters and representations have long been criticized for not being explainable. In this paper, we propose Deep-IRT which is a synthesis of the item res…
Framework for fast CAT calibration and administration using AutoML and IRT.
problem Calibrating and administering large-scale CAT tests with limited data.
method AutoIRT (AutoML + IRT) for calibration, BanditCAT for administration.
result Framework successfully launched new item types on DET practice test.
Paper uses NLP and IRT to score ESG factors from news articles.
problem Lack of precise ESG metrics in finance.
method Combines NLP and IRT models on ESG-related news data.
result Method offers more precise ESG metrics with temporal dynamics.
IRT improves algorithm evaluation across datasets.
problem Evaluating the performance of algorithm portfolios.
method Modified IRT framework for evaluating algorithm portfolios across datasets.
result Richer characteristics of algorithm performance are revealed.
Item response theory (IRT) models for categorical response data are widely used in the analysis of educational data, computerized adaptive testing, and psychological surveys. However, most IRT models rely on both the assumption that categories are strictly ordered and the assumption that this ordering is known a priori…
VTIRT speeds up IRT inference for dynamic learner proficiency.
problem Expensive and slow inference algorithms for dynamic IRT models.
method Variational Temporal IRT (VTIRT) for fast, accurate inference.
result Orders of magnitude speedup in inference runtime with accurate results.
A new framework evaluates large language models efficiently and accurately.
problem Evaluation of large language models is challenging due to stochasticity and heterogeneity of benchmarks.
method Interpretable and scalable framework based on Item Response Theory (IRT) and majorization-minimization principle.
result Our method achieves superior scalability and interpretability compared to existing approaches.
Researchers study how teachers' advising relationships influence their perceptions of satisfaction and students, not policy influence.
problem Understanding the relationship between teachers' advising relationships and their perceptions of satisfaction and students.
method Proposed a novel joint model of network and item responses (JNIRM) with correlated latent variables.
result Teachers' advising relationships contribute more to satisfaction and students than to influence over educational policies.
Item response theory (IRT) models are widely used in psychometrics and educational measurement, being deployed in many high stakes tests such as the GRE aptitude test. IRT has largely focused on estimation of a single latent trait (e.g. ability) that remains static through the collection of item responses. However, in …
In this article, we propose a novel probabilistic framework to improve the accuracy of a weighted majority voting algorithm. In order to assign higher weights to the classifiers which can correctly classify hard-to-classify instances, we introduce the Item Response Theory (IRT) framework to evaluate the samples' diffic…
We propose a new IRT model that directly factors test items without factor analysis.
problem Existing multidimensional IRT methods require factorization, which is posthoc and linear.
method We use a sparsity-promoting horseshoe prior to factorize items directly within the IRT model.
result Our model performs factorization directly and consistently selects the correct number of factors.
This paper introduces our solution to the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM). We used deep factorization machines, a wide and deep learning model of pairwise relationships between users, items, skills, and other entities considered. Our solution (AUC 0.815) hopefully managed to bea…
IRTCI uses IRT for categorical imputation, outperforming other methods in various datasets.
problem Missing data in datasets limits model testing and statistical inference.
method Item Response Theory for Categorical Imputation (IRTCI) based on IRT.
result IRTCI outperformed kNN, MICE, and Datawig in various datasets.
A new dataset tracks user interactions and click responses in online marketplaces.
problem Lack of exposure data in recommender systems datasets.
method Proposes a novel dataset including slates and click responses, allowing more accurate likelihood models.
result Models using exposure data show more natural likelihood, reducing bias towards previously exposed items.
ESRLCM clusters similar responses, more broadly than traditional models.
problem Clustering multivariate categorical data with common response patterns.
method Bayesian Equivalence Set Restricted Latent Class Model (ESRLCM).
result ESRLCM identifies clusters with similar item response probabilities.
Bayesian method improves adaptive testing item selection, ensuring full item exposure.
problem Adaptive testing selects items to estimate ability, but must also ensure diverse item exposure.
method Formulated as Bayesian model averaging, deriving optimal item sampling probabilities.
result Stochastic method achieves full item bank exposure without sacrificing accuracy.
The paper examines skill estimation and variance under model misspecification in IRT.
problem Underestimation and overestimation of skills when non-compensatory model is misspecified as compensatory.
method Theoretical approach to analyze underestimation and overestimation of skills and variance.
result Overestimation of skills occurs around the origin and asymptotic variance differs under model misspecification.
Model learns metrics and preferences from user comparisons.
problem Simultaneous metric and preference learning from user comparisons.
method Jointly learns a metric and latent ideal points for each user.
result Model captures individual preferences and learns metrics efficiently.
Archetypal analysis helps understand binary data sets.
problem Explaining binary questionnaire data.
method Using archetypal analysis for binary observations.
result The approach contributes to understanding binary data sets.
NASirt automates CNN architecture design for spectral data.
problem Designing optimal neural architectures for complex data.
method Neural Architecture Search (NAS) with Item Response Theory (IRT) for instance-level complexity.
result NASirt achieves high accuracy (97.40%) on spectral datasets.
Suppose that we wish to estimate a user's preference vector w from paired comparisons of the form "does user w prefer item p or item q?," where both the user and items are embedded in a low-dimensional Euclidean space with distances that reflect user and item similarities. Such observations arise in numerous se…
Computer simulations have become a popular tool of assessing complex skills such as problem-solving skills. Log files of computer-based items record the entire human-computer interactive processes for each respondent. The response processes are very diverse, noisy, and of nonstandard formats. Few generic methods have b…
Adaptive querying learns user psychometrics with AI personas.
problem Learning user psychometrics within query budgets.
method Persona-induced latent variable model with AI personas and large language model response distributions.
result Persona-based posteriors deliver accurate probabilistic predictions.
Knowledge tracing is a sequence prediction problem where the goal is to predict the outcomes of students over questions as they are interacting with a learning platform. By tracking the evolution of the knowledge of some student, one can optimize instruction. Existing methods are either based on temporal latent variabl…
An efficient algorithm selects the correct number of latent dimensions in multidimensional probit models.
problem Determining the correct number of latent dimensions in multidimensional probit graded response models.
method Adaptive Bayesian dimension selection framework using cumulative ordered spike-and-slab (COSS) prior and Albert--Chib latent response augmentation.
result The proposed method accurately recovers latent structures and avoids repeated model fitting.
Deep learning speeds up IFA estimation for large datasets.
problem Slow MML estimation for large-scale IFA models.
method Importance-weighted autoencoder (IWAE) for fast VI.
result IWAE yields accurate estimates faster than MH-RM.
New IRT method identifies useful datasets for ML classifier evaluation.
problem Lack of standard evaluation strategy for ML benchmarks.
method Applied Item Response Theory (IRT) to OpenML-CC18 benchmark.
result Not all datasets are useful for evaluating classifiers.
R2DE assesses new exam questions quickly and accurately.
problem Assessing newly generated exam questions efficiently and accurately.
method R2DE uses NLP to estimate difficulty and discrimination of questions based on their text.
result R2DE can estimate question difficulty and discrimination quickly and accurately.
Previous work on recommender systems mainly focus on fitting the ratings provided by users. However, the response patterns, i.e., some items are rated while others not, are generally ignored. We argue that failing to observe such response patterns can lead to biased parameter estimation and sub-optimal model performanc…
This paper explores the suitability of using automatically discovered topics from MOOC discussion forums for modelling students' academic abilities. The Rasch model from psychometrics is a popular generative probabilistic model that relates latent student skill, latent item difficulty, and observed student-item respons…
The recently proposed SPARse Factor Analysis (SPARFA) framework for personalized learning performs factor analysis on ordinal or binary-valued (e.g., correct/incorrect) graded learner responses to questions. The underlying factors are termed "concepts" (or knowledge components) and are used for learning analytics (LA),…
The Rasch model is widely used for item response analysis in applications ranging from recommender systems to psychology, education, and finance. While a number of estimators have been proposed for the Rasch model over the last decades, the available analytical performance guarantees are mostly asymptotic. This paper p…
TrueLearn uses Bayesian algorithms to match learners with educational resources.
problem Matching lifelong learners to open educational resources efficiently and effectively.
method TrueLearn uses a text ontology and Bayesian strategies to match learners with educational resources based on their background knowledge and material novelty.
result TrueLearn algorithms show promise in building an effective educational recommendation system.
Using the trends of estimated abilities in terms of item response theory for online testing, we can predict the success/failure status for the final examination to each student at early stages in courses. In prediction, we applied the newly developed nearest neighbor method for determining the similarity of learning sk…