Computerized classification test: Difference between revisions

Revision as of 15:00, 22 October 2023

Revision as of 19:57, 15 January 2025

Line 10:

The starting point is not a topic of contention; research on CCT primarily investigates the application of different methods for the other three components. ''Note:'' The termination criterion and scoring procedure are separate in CAT, but the same in CCT because the test is terminated when a classification is made. Therefore, there are five components that must be specified to design a CAT.

An introduction to CCT is found in Thompson (2007)<ref>{{cite journal|last=Thompson, |first=N. A. (|year=2007). |title=A Practitioner's Guide for Variable-length Computerized Classification Testing. |journal=Practical Assessment Research & Evaluation, |volume=12(|issue=1). [|url=http://pareonline.net/getvn.asp?v=12&n=1]|archive-url=https://web.archive.org/web/20071021213959/http://pareonline.net/pdf/v12n1.pdf|archive-date=21 October 2007|url-status=dead}}</ref> and a book by Parshall, Spray, Kalohn and Davey (2006).<ref>{{cite book|last1=Parshall|first1=C. G.|last2=Spray|first2=J. A.|last3=Kalohn|first3=J. C.|last4=Davey|first4=T.|year=2006|title=Practical considerations in computer-based testing|___location=New York|publisher=Springer}}</ref> A bibliography of published CCT research is found below.

Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2006). Practical considerations in computer-based testing. New York: Springer.</ref> A bibliography of published CCT research is found below.

== How it works ==

Line 19 ⟶ 18:

Two approaches are available for the psychometric model of a CCT: [[classical test theory]] (CTT) and [[item response theory]] (IRT). Classical test theory assumes a state model because it is applied by determining item parameters for a sample of examinees determined to be in each category. For instance, several hundred "masters" and several hundred "non-masters" might be sampled to determine the difficulty and discrimination for each, but doing so requires that you be able to easily identify a distinct set of people that are in each group. IRT, on the other hand, assumes a trait model; the knowledge or ability measured by the test is a continuum. The classification groups will need to be more or less arbitrarily defined along the continuum, such as the use of a cutscore to demarcate masters and non-masters, but the specification of item parameters assumes a trait model.

There are advantages and disadvantages to each. CTT offers greater conceptual simplicity. More importantly, CTT requires fewer examinees in the sample for calibration of item parameters to be used eventually in the design of the CCT, making it useful for smaller testing programs. See Frick (1992)<ref>{{cite journal|last=Frick, |first=T. (|year=1992). |title=Computerized Adaptive Mastery Tests as Expert Systems. |journal=Journal of Educational Computing Research, |volume=8(|issue=2), |pages=187-213|doi=10.2190/J87V-6VWP-52G7-L4XX}}</ref> for a description of a CTT-based CCT. Most CCTs, however, utilize IRT. IRT offers greater specificity, but the most important reason may be that the design of a CCT (and a CAT) is expensive, and is therefore more likely done by a large testing program with extensive resources. Such a program would likely use IRT.

== Starting point ==

Line 25 ⟶ 24:

== Item selection ==

In a CCT, items are selected for administration throughout the test, unlike the traditional method of administering a fixed set of items to all examinees. While this is usually done by individual item, it can also be done in groups of items known as [[testlet]]s (Leucht & Nungester, 1996;<ref>{{cite journal|last1=Luecht, |first1=R. M., & |last2=Nungester, |first2=R. J. (|year=1998). |title=Some practical examples of computer-adaptive sequential testing. |journal=Journal of Educational Measurement, |volume=35, |pages=229-249|doi=10.1111/j.1745-3984.1998.tb00537.x}}</ref> Vos & Glas, 2000<ref>{{cite book|last1=Vos, |first1=H.J. & |last2=Glas, |first2=C.A.W. (|year=2000). |chapter=Testlet-based adaptive mastery testing. In |editor1-last=van der Linden, |editor1-first=W.J., and |editor2-last=Glas, |editor2-first=C.A.W. (Eds.) |title=Computerized Adaptive Testing: Theory and Practice|doi=10.1007/0-306-47531-6_15}}</ref>).

</ref>).

Methods of item selection fall into two categories: cutscore-based and estimate-based. Cutscore-based methods (also known as sequential selection) maximize the [[information]] provided by the item at the cutscore, or cutscores if there are more than one, regardless of the ability of the examinee. Estimate-based methods (also known as adaptive selection) maximize information at the current estimate of examinee ability, regardless of the ___location of the cutscore. Both work efficiently, but the efficiency depends in part on the termination criterion employed. Because the [[sequential probability ratio test]] only evaluates probabilities near the cutscore, cutscore-based item selection is more appropriate. Because the [[confidence interval]] termination criterion is centered around the examinees ability estimate, estimate-based item selection is more appropriate. This is because the test will make a classification when the confidence interval is small enough to be completely above or below the cutscore (see below). The confidence interval will be smaller when the standard error of measurement is smaller, and the standard error of measurement will be smaller when there is more information at the theta level of the examinee.

Line 49 ⟶ 47:

*Frick, T. W. (1992). Computerized adaptive mastery tests as expert systems. Journal of Educational Computing Research, 8, 187–213.

*Huang, C.-Y., Kalohn, J.C., Lin, C.-J., and Spray, J. (2000). Estimating Item Parameters from Classical Indices for Item Pool Development with a Computerized Classification Test. (Research Report 2000–4). Iowa City, IA: ACT, Inc.

*Jacobs-Cassuto, M.S. (2005). A Comparison of Adaptive Mastery Testing Using Testlets With the 3-Parameter Logistic Model. Unpublished doctoral dissertation, University of Minnesota, Minneapolis, MN.

With the 3-Parameter Logistic Model. Unpublished doctoral dissertation, University of Minnesota, Minneapolis, MN.

*Jiao, H., & Lau, A. C. (2003). The Effects of Model Misfit in Computerized Classification Test. Paper presented at the annual meeting of the National Council of Educational Measurement, Chicago, IL, April 2003.

*Jiao, H., Wang, S., & Lau, C. A. (2004). An Investigation of Two Combination Procedures of SPRT for Three-category Classification Decisions in Computerized Classification Test. Paper presented at the annual meeting of the American Educational Research Association, San Antonio, April 2004.