Custom-Built Assessment: Designing and Developing Adaptive Tests

What Is Adaptive Testing? - Organizations often seek innovative ways to use technology to improve assessment practices. One example of this phenomenon is the growing prevalence of internet or computer-based testing (ICBT) and, in particular, the use of adaptive testing. Computerized adaptive testing (CAT) is an innovative assessment format delivered on a computer or, increasingly, via computer over the internet.

In conventional paper-and-pencil (P&P) tests, all examinees receive the same items, most of which are designed to target a “medium” level of difficulty for the given testing population. This design works well for examinees of average ability but presents too many difficult items to low-ability examinees and too many easy items to those of high ability. In contrast, adaptive testing uses a statistical algorithm to select and present test items tailored to each individual examinee’s ability level.
 
Adaptive testing has emerged as a powerful assessment tool for organizations, leading to:

• reduced costs,
• shorter administration times,
• improved scoring precision, and
• simplified item development.

Our Experience with Adaptive Testing

HumRRO has more than 30 years of experience providing sound guidance on adaptive ICBT systems to a variety of clients in both the private and public sectors.
 
Military Testing. One of the world’s largest testing programs, the Armed Services Vocational Aptitude Battery (ASVAB) is the U.S. military’s qualifying test for enlisted service members. In the late 1970s, the U.S. Department of Defense (DoD) began converting the ASVAB from P&P to CAT. HumRRO employeeshelped program the resulting CAT-ASVAB and continue to provide programming, test development, psychometric, and R&D support to this test program. We recently wrote code that allows CAT-ASVAB to be administered on the internet.
 
Unproctored Internet-Based Adaptive Testing. HumRRO provided Procter & Gamble (P&G) with an adaptive testing engine that determines which items to present and updates applicants’ scores. Subsequently, we provided expert consulting services (e.g., item analysis, development, and review; item taxonomy creation; CAT algorithm modification) for the development of the first operational, adaptive unproctored internet-based test (UIT) of cognitive ability, now administered to P&G job applicants worldwide. We worked with P&G to develop an analytic framework that adheres to the highest scientific and legal standards for test development, validation, and maintenance. P&G’s ground-breaking effort now stands as the archetypal UIT selection system.
 
HumRRO is also working with IBM to develop an adaptive UIT selection system they can use to select applicants into one of three large job families worldwide. We will be providing IBM expert guidance throughout all stages of system development from item generation and calibration to test maintenance and verification. The HumRRO psychometrics team also will help IBM choose the most appropriate model for scoring and presenting CAT items, assist with quality assurance checks to ensure appropriate operation of the adaptive test, and evaluate potential measurement bias in test content.