Item Response Theory for Educators

What IRT Is

Item response theory is a family of probabilistic models that relate the probability of a correct response on a test item to two things: a latent characteristic of the learner (ability, $\theta$) and the characteristics of the item (typically difficulty, often discrimination, sometimes guessing). The same model fits both the response data of many learners and the calibration data of many items, and it produces a common interpretive scale on which ability and difficulty are directly comparable.

IRT is the methodological foundation of large-scale psychometric assessment. The GRE, GMAT, TOEFL, NAEP (the U.S. National Assessment of Educational Progress), the PISA international assessment, and a substantial portion of professional licensure testing are IRT-calibrated. Computerized adaptive testing (CAT), in which the next item shown is selected based on the running ability estimate, is essentially IRT-driven item selection.

This page covers IRT as a method that educators and curriculum designers can use, frame, and critique. It explains the parameterizations (1PL, 2PL, 3PL), what each parameter does, how calibration works in practice, and how TheoremPath uses IRT-style difficulty calibration for its assessment items. The boundary conditions and the relationship to BKT are made explicit.

The Core Model

The simplest IRT model is the one-parameter logistic (1PL), also known as the Rasch model. Each item has a single parameter, its difficulty $b_i$, and each learner has a single parameter, their ability $\theta_j$. The probability that learner $j$ answers item $i$ correctly is

$P(X_{ij} = 1 \mid \theta_j, b_i) = \frac{1}{1 + e^{-(\theta_j - b_i)}} = \sigma(\theta_j - b_i),$

where $\sigma$ is the logistic function. The interpretation is clean: the probability of a correct response is determined by the difference between learner ability and item difficulty, mapped through a logistic curve. A learner whose ability equals an item's difficulty has a 50% chance of answering correctly; an ability one logit higher gives roughly 73% probability; two logits higher gives 88%.

The two-parameter logistic (2PL) adds an item-specific discrimination parameter $a_i$:

$P(X_{ij} = 1 \mid \theta_j, a_i, b_i) = \sigma\bigl(a_i (\theta_j - b_i)\bigr).$

A high $a_i$ means the item discriminates sharply between abilities near $b_i$ (a steep logistic curve); a low $a_i$ means the item's response curve is nearly flat (it does not distinguish learners well at any ability level).

The three-parameter logistic (3PL) adds a guessing (or "pseudo-chance") parameter $c_i$:

$P(X_{ij} = 1 \mid \theta_j, a_i, b_i, c_i) = c_i + (1 - c_i) \sigma\bigl(a_i (\theta_j - b_i)\bigr).$

This is the standard form for multiple-choice items where a floor probability of correct response (typically near $1/k$ for $k$ options) is empirically observed.

A polytomous extension (the graded response model, the partial credit model) handles items with more than two response categories. The basic logic is the same.

Ability Estimation and Item Calibration

IRT is two separate inferential problems running on the same data:

Item calibration. Given a sample of learners and their responses, estimate the item parameters $(a_i, b_i, c_i)$ for each item. This is typically done by marginal maximum likelihood (treating $\theta$ as drawn from a prior distribution and integrating out) or by joint maximum likelihood (treating both $\theta$ and the item parameters as fixed effects to estimate). Software: R/mirt, R/ltm, the flexMIRT commercial package, the open-source pyirt and girth Python packages.

Ability estimation. Given calibrated items and a learner's responses, estimate $\theta_j$. This is typically done by maximum likelihood, weighted likelihood, or a Bayesian estimate (EAP, expected a posteriori, is standard).

The two problems can be handled iteratively (calibrate items on a reference sample, deploy them in production with $\theta$ estimated by MLE / EAP), or simultaneously (joint estimation in small applications). Production systems like the GRE separate the two: items are calibrated by the test sponsor on a controlled sample, and the calibrated parameters are treated as known for operational ability estimation.

The information function of an item summarises how much information the item contributes about $\theta$ at each ability level:

$I_i(\theta) = a_i^2 \cdot P_i(\theta) (1 - P_i(\theta)) \quad \text{(2PL form)}.$

Information is maximized at $\theta = b_i$, with magnitude proportional to $a_i^2$. This drives item-selection rules in adaptive testing: the next item is the one whose information at the current ability estimate is highest.

Empirical Status

IRT is a mature method with a substantial empirical record. The empirical case is largely about measurement: IRT-calibrated tests produce ability estimates that correlate strongly with external criteria (school grades, future test performance, work performance for occupational tests) at levels comparable to or better than classical-test-theory raw-score-based tests, with the additional benefit that the ability estimates live on a common scale and item difficulties live on the same scale.

Standard references for the empirical case:

• Lord, F. M. Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum, 1980. The standard textbook from one of the founders.
• van der Linden, W. J., and Hambleton, R. K., eds. Handbook of Modern Item Response Theory. Springer, 1997. Multi-author reference covering the family of models and their applications.
• Embretson, S. E., and Reise, S. P. Item Response Theory for Psychologists. Lawrence Erlbaum, 2000. Standard graduate-level introduction.
• De Boeck, P., and Wilson, M., eds. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, 2004. The bridge between IRT and modern statistical methodology.

The Rasch tradition (after Georg Rasch, the Danish mathematician whose 1960 monograph Probabilistic Models for Some Intelligence and Attainment Tests introduced the model) treats the 1PL form as a requirement for measurement, not an empirical simplification. The Anglo-American IRT tradition treats it as one empirical option among several. Both traditions agree that the structural assumption of unidimensional ability is what makes IRT work; they disagree on what to do when the assumption fails.

Mechanism: Why IRT Works (When It Does)

Three structural features carry the model's weight:

Ability and difficulty live on a common scale. The logit- scale ($\theta$, $b$ both real numbers, with the difference being what matters) lets the same scale describe a learner's expected performance at any difficulty. A score of 7 out of 10 on an easy test and 5 out of 10 on a hard test become directly comparable ability estimates. This is the major advance over classical test theory raw-score reporting.

Item information is local. An item of difficulty $b$ is most informative about ability levels near $b$. This makes adaptive testing efficient: after a few items, the system has a rough ability estimate and can select items targeted near that estimate, gaining substantially more information per item than a random or fixed-form selection rule would.

The probabilistic interpretation supports honest standard errors. Each ability estimate comes with a standard error that reflects how much information the responses provide about $\theta$. A learner who has answered five items near their ability gets a narrower confidence interval than one who has answered five items far from their ability. Reporting the standard error is part of the IRT discipline.

Boundary Conditions: Where IRT Fails

Five places the model is widely known to misbehave.

Unidimensional ability assumption. The standard 1PL/2PL/3PL models assume there is a single underlying ability $\theta$ that explains responses. Real assessments typically tap multiple abilities (verbal reasoning + working memory + domain knowledge, for example). Multidimensional IRT (MIRT) is the standard generalization but is much harder to calibrate; in practice unidimensional models are often used despite known multidimensionality, with the cost absorbed into the discrimination estimates.

Local independence. The model assumes that responses to different items are conditionally independent given $\theta$. Items that share a passage (reading-comprehension items), testlets, or items that depend on the same diagram violate this. Testlet response models exist but introduce extra complexity.

Ability is treated as a fixed trait, not a learning state. The IRT $\theta$ is the same value across the testing session. This is appropriate for psychometric assessment of a stable trait but is not the right model for an adaptive learning system where the goal is to track ability change. BKT and DKT are the standard tools for that role; IRT calibration of items is typically combined with a state-evolution learner model.

Calibration data demands. Stable item-parameter estimates typically require several hundred respondents per item, with respondents covering a substantial range of ability. Pilot testing on small samples gives unstable parameter estimates.

The Rasch / IRT scale-invariance debate. The Rasch tradition treats the 1PL as the only "fair" measurement model because it is the unique model under which item calibration and ability estimation are invariant under sample selection. Anglo-American IRT treats this as one consideration among several. The disagreement is partly philosophical (what counts as measurement?) and partly empirical (do real items satisfy the constant-discrimination assumption?). Educators encountering both traditions should know they exist and not be surprised by either group's vocabulary.

How TheoremPath Uses IRT-Style Calibration

TheoremPath's assessment items each carry an explicit difficulty parameter (an integer from 1 to 10, with documentation in CLAUDE.md's Quiz System section). The difficulty parameter is treated as an item characteristic in the IRT sense: it is calibrated from author judgment plus, where available, response data; it is used in the mastery-update logic to weight evidence; and it is the basis for the audit-difficulty.ts script that flags miscalibrated items.

The TheoremPath difficulty scale is intentionally coarser than a production psychometric scale (10 bands rather than a continuous $\theta$). Three reasons:

1. Author calibration is more reliable on a coarse scale. Asking content authors to mark items as "1-3 = foundation, 4-6 = intermediate, 7-10 = advanced / synthesis" produces more consistent labels than asking them to assign a continuous difficulty. The audit-difficulty.ts script flags items whose author label diverges substantially from response-data estimates.
2. Coarse scales are easier to communicate. A learner sees "advanced" or "core" rather than "$\theta = 1.4$ logits." The user-facing register of TheoremPath's mastery system is discrete bands, not continuous logits.
3. Production calibration data is limited. TheoremPath is not a high-stakes psychometric instrument and does not have the response volumes to calibrate continuous IRT parameters reliably. Coarser bands match the actual evidentiary basis.

The src/lib/adaptive/difficulty-calibration.ts module implements the calibration routines and the audit logic. The underlying philosophy is that IRT-style difficulty calibration is a useful discipline even when a full IRT model is not the right level of granularity for the application.

How Educators and Researchers Use IRT in Practice

Standardized testing. GRE, GMAT, TOEFL, NAEP, PISA, MCAT: all IRT-calibrated. The item bank is calibrated on field-test samples; the operational scoring uses the calibrated parameters to estimate $\theta$ on a common scale across different test forms. This is how it is possible for a Quantitative GRE administered today and one administered in 2018 to produce comparable scores even though no two test-takers see the same items.

Computerized adaptive testing (CAT). The CAT loop is: start with a midrange item, observe the response, update the ability estimate, choose the next item to maximize information at the current estimate, stop when the standard error falls below a threshold or a maximum item count is reached. The GRE General Test, Smarter Balanced state assessments, and a substantial portion of professional licensure use CAT.

Educational measurement research. IRT is the standard methodology for evaluating new assessment instruments. Validity arguments routinely include IRT-calibrated DIF (differential item functioning) analyses to identify items that perform differently across demographic groups, conditional on ability. This is one of the standard fairness audits in psychometrics.

Research-platform learner modelling. ASSISTments and similar research platforms typically combine IRT-style item calibration with BKT-style learner state evolution. The IRT side handles "how hard is this item" and the BKT side handles "where is this learner now." Production-quality adaptive learning systems use both.

Common Misapplications

Treating $\theta$ as a fixed trait when ability is changing. Standard IRT assumes ability is constant within the testing window. Using IRT to track learning over time, without an explicit state-evolution model, conflates measurement error with genuine ability change.

Calibrating items on too small a sample. Stable 2PL or 3PL calibration requires several hundred respondents per item. Reporting "difficulty" estimates from a pilot of 20-30 learners is a misapplication.

Ignoring local-independence violations. Items that share a reading passage, or items that depend on a common diagram, are not locally independent given $\theta$. Treating them as independent inflates the apparent reliability of the assessment.

Reporting raw scores alongside IRT scores without explanation. Raw scores and IRT scores are not interchangeable. A learner who gets 7 out of 10 on a hard form and 8 out of 10 on an easy form may have the same IRT $\theta$. Reporting both without explaining which is operative confuses learners.

Treating the discrimination parameter as a quality measure. A 2PL item with low $a$ may be working as designed (it asks about something orthogonal to the main ability dimension) or it may be broken (it has a typo, it's tapping a different construct). The parameter alone does not say which.

Related Methods

The pages on bayesian-knowledge-tracing-for-educators and fsrs-spaced-repetition-for-educators cover the related layers. The the-theorempath-pedagogy-thesis page covers how IRT, BKT, and FSRS fit together in the TheoremPath production stack.

What This Page Does Not Claim

This page does not claim IRT $\theta$ estimates are the right quantity to report to learners. The standard error of an IRT $\theta$ estimate from a 10-item assessment is substantial; using $\theta$ as a personalized claim about learner ability is a register error.

This page does not claim TheoremPath's coarse-band difficulty scale is psychometrically equivalent to a calibrated 2PL or 3PL parameter. It is not. It is a pragmatic reduction of the IRT discipline to what TheoremPath's calibration data can support.

This page does not claim IRT replaces classical test theory. Classical reliability indices (Cronbach's $\alpha$, item-total correlations) remain useful diagnostic tools and are not made obsolete by IRT. The two methodologies coexist in production assessment.

FAQ

Why use logits rather than probabilities?

The logit scale is unbounded (real-valued) and additive: the probability of correct depends on $\theta - b$, so a unit increase in ability has the same effect at $\theta = -2$ as at $\theta = +2$ in logit space. Probabilities saturate (0 to 1) and the same effect would look very different at the extremes. For calibration, reporting, and adaptive item selection, logits are the natural scale.

Is the Rasch model just IRT with $a = 1$?

In one sense yes (the 1PL is a Rasch model with the discrimination fixed at 1 across items). In a deeper sense the Rasch tradition makes a different claim: that constant discrimination is a requirement of measurement, not an empirical simplification. The disagreement between the Rasch and Anglo-American IRT traditions is on this philosophical point.

Can IRT work without large calibration samples?

Stable parameter estimates (especially for 2PL and 3PL) need several hundred respondents per item with respondents covering a substantial range of ability. Smaller samples can produce point estimates but with confidence intervals so wide that the estimates are not actionable. The 1PL Rasch model is somewhat more forgiving because it has fewer parameters.

What's a "good" item discrimination value?

In the 2PL framework, $a$ values around 1.0 are typical for educational assessment items; values above 2.0 are unusual and often indicate either an exceptional item or a calibration artifact. Values below 0.5 indicate the item discriminates poorly and may be tapping a different construct than the rest of the assessment.

How does IRT relate to BKT?

IRT models items with calibrated parameters; BKT models learners with evolving state. They answer different questions and, in production adaptive learning, are typically combined: IRT calibrates the item bank, BKT (or PFA, or DKT) tracks the learner's evolving knowledge state. The bayesian-knowledge-tracing-for-educators page covers the BKT side; the the-theorempath-pedagogy-thesis page covers how the two layers fit together.

How does this connect to TheoremPath?

TheoremPath's assessment items each carry a difficulty parameter calibrated by author judgment plus response data, audited by the audit-difficulty.ts script. The calibration framework lives in src/lib/adaptive/difficulty-calibration.ts. The TheoremPath implementation is intentionally coarser than a full IRT model (10 discrete bands rather than continuous $\theta$ and item parameters), for the calibration-data-volume reasons described above.

Internal links

• PedagogyPath: bayesian-knowledge-tracing-for-educators for the state-evolution companion; fsrs-spaced-repetition-for-educators for the review-scheduling layer; intelligent-tutoring-systems for the broader adaptive-learning context; the-theorempath-pedagogy-thesis for the canonical statement of how the components fit together.
• TheoremPath: the maximum-likelihood-estimation page for the IRT calibration machinery; the hypothesis-testing-for-ml page for the underlying statistical-decision framework.
• PhilosophyPath: the plato-vs-socrates source-discipline framework that PedagogyPath inherits.

Sources and further reading

Foundational:

• Rasch, G. Probabilistic Models for Some Intelligence and Attainment Tests. Danish Institute for Educational Research, 1960; reprinted MESA Press, 1993.
• Lord, F. M. Applications of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum, 1980. Standard textbook reference.
• Lord, F. M., and Novick, M. R. Statistical Theories of Mental Test Scores. Addison-Wesley, 1968. The unifying theoretical framework for IRT and classical test theory.

Modern textbooks:

• van der Linden, W. J., and Hambleton, R. K., eds. Handbook of Modern Item Response Theory. Springer, 1997.
• Embretson, S. E., and Reise, S. P. Item Response Theory for Psychologists. Lawrence Erlbaum, 2000.
• De Boeck, P., and Wilson, M., eds. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, 2004.
• Bond, T. G., and Fox, C. M. Applying the Rasch Model: Fundamental Measurement in the Human Sciences, 3rd ed. Routledge, 2015. The standard Rasch-tradition textbook.

Computerized adaptive testing:

• van der Linden, W. J., and Glas, C. A. W., eds. Computerized Adaptive Testing: Theory and Practice. Kluwer, 2000.
• Wainer, H. Computerized Adaptive Testing: A Primer, 2nd ed. Lawrence Erlbaum, 2000.

Differential item functioning:

• Holland, P. W., and Wainer, H., eds. Differential Item Functioning. Lawrence Erlbaum, 1993. The DIF standard reference.

Software:

• Chalmers, R. P. "mirt: A Multidimensional Item Response Theory Package for the R Environment." Journal of Statistical Software 48(6) (2012). The dominant open-source IRT package.

This page is part of PedagogyPath, sister site to TheoremPath in the path-network family. It documents one of the four pillars of the TheoremPath adaptive-learning machinery; the canonical statement of how the pillars fit together is at the-theorempath-pedagogy-thesis.