IES Diagnostic Testing

Institute of Education Sciences Grant R305A100234,

An Adaptive Testing System for Diagnosing Sources of  Mathematics Difficulties

Susan Embretson, Principal Investigator

    The overall goal for this project is to build an online and on-demand adaptive assessment system to diagnose sources of students’ mathematics difficulties and deficits that are linked to state standards-based assessments.  A primary applied goal is to appropriately diagnose for which skills instruction will be most effective in remediating sources of mathematics difficulty.   Student performance on the state accountability test of mathematics is being interfaced with the adaptive diagnostic testing system to prescribe instruction.   
The diagnostic system for the assessments is being developed by combining standards-based knowledge categorizations of mathematics items with cognitive models of mathematical problem solving.  The assessment system will consist of seven components: (1) a diagnostic system module; (2) an item bank module ; (3) a diagnostic IRT model module; (4) an adaptive item selection module; (5) an interactive online test delivery module; (6) a score report module; and (7) a validity module.

    To build the system, a series of studies are being conducted that include both theoretical issues and assessment development surrounding the seven modules listed above.  The results of many of the studies have implications for research and development beyond the specific scope of this project.  
    
    For the diagnostic system module, both skill-based and cognitive complexity models were developed to explain item difficulty on the year-end achievement tests.  For the item bank module, an automatic item generator was developed and generated items were submitted to traditional item evaluation procedures.  For the diagnostic IRT module, the theoretical properties of alternative models were studied to select an appropriate model for the study.  For the adaptive item selection module, multistage testing accuracy was studied and implemented.  For the interactive test delivery module, year-end achievement testing results were interfaced with the multistage tests to assure the most accurate measurement.  For the score report module, automated individualized reports were prepared to indicate which skills were possessed by the participating classes as a whole and by individual students.  Finally, for the validity module, various aspects of validity for the skill diagnosis are being studied.