ATLAS Research Fellowship

Validation of a portion of the Learning Map using methods proposed by the fellow. The validation study may include:

• Validation based on new data collected
• A simulation study to evaluate different test designs that would allow for the collection of data for use in future validation studies

Artificial intelligence applications that can be used within an operational educational assessment program. Examples include but are not limited to:

• Item generation, alignment, scoring, etc.
• Supplement quality control and review processes while highlighting noteworthy factors (e.g., using AI to detect text with accessibility concerns for students with disabilities while highlighting text features that may reduce accessibility)
• Produce automated, real-time feedback to students, parents, and teachers about assessment performance
• Review ethical considerations for implementing artificial intelligence, including machine learning, with an assessment in an evidence-centered design framework
• Machine learning applications intended to support measurement in an operational educational assessment program, with an emphasis on confirmatory approaches that supplement and expand upon traditional psychometric measurement methods. Examples include but are not limited to: 
  • Provide additional context about machine learning output for assessments (e.g., predicting item difficulty and then identifying factors contributing to predictions of an incredibly difficult item)
  • Integrate machine learning algorithms into time and effort-intensive processes (e.g., using a machine learning algorithm as the second rater in interrater reliability analyses)

Evaluating the properties of longitudinal diagnostic classification models and their application to operational assessment programs. Examples could include, but are not limited to:

• Evaluation of the impact of model misfit on estimates of within-year growth for diagnostic models (e.g., using a transition diagnostic classification model). 
• Evaluating the sample size requirements for transition diagnostic classification models 
• Evaluating the impact of violating the assumption of item invariance for transition diagnostic classification models

Systematic literature reviews:

• Review and proof of concept simulation for an adaptive test that uses diagnostic classification modeling, including adaptation between short testlets.
• Review of the connections between cognitive models of learning and memory and Evidence-Centered Design as a method for test development. What current research findings on formal, organized learning models are relevant to developing assessments using principles of Evidence-Centered Design?
• Review of cognitive complexity models and commonly used taxonomies of cognitive complexity, specifically their relationship with three-dimensional science standards such as the Next Generation Science Standards. The scope might include comparisons among models, adaptations of models for novel applications to complex test constructs, and other research findings that may support domain definition and test development.
• Review and support for mapping cognition for early learning (i.e., preK-2) in ELA, mathematics, and/or science
• Fellow-proposed research ideas, including in areas of early childhood, education of students with disabilities, including students with extensive support needs (e.g., academic instruction, postsecondary opportunities, etc.), educator professional learning, NGSS, evidence-centered design, universal design for learning, diagnostic classification modeling, etc.

Other proposal ideas relevant to our mission and projects will be considered.