Validating Assessment Tools in Simulation

TL;DR: A wide range of assessment tools are available to assess learners on taught skills and knowledge, and there is stake-holder interest validating these assessment tools as discussed by the authors, which is critical for high-stakes certification, such as licensing opportunities and board examinations.

Abstract: Health care simulation is a growing field that combines innovative technologies and adult learning theory to reproducibly train medical professionals in clinical skills and practices. A wide range of assessment tools are available to assess learners on taught skills and knowledge, and there is stake-holder interest validating these assessment tools. Reliable quantitative assessment is critical for high-stakes certification, such as licensing opportunities and board examinations. There are many aspects to an evaluation in healthcare simulation that range from educating new learners and training current professionals, to a systematic review of programs to improve outcomes. Validation of these assessment tools is essential to ensure that they are valid and reliable. Validity refers to whether any measuring instrument measures what it is intended to measure. Additionally, reliability is part of the validity assessment and refers to the consistent or reproducible results of an assessment tool. The assessment tool should yield the same results for the same type of learner every time it is used. In practice, actual healthcare delivery requires knowledge of technical, analytical, and interpersonal skills. This merits assessment systems to be comprehensive, valid, and reliable enough to assess the necessary elements along with testing for critical knowledge and skills. Validating assessment tools for healthcare simulation education ensure that learners can demonstrate the integration of knowledge and skills in a realistic setting.The assessment process itself is influential for the process of curriculum development, as well as feedback and learning. Recent developments in psychometric theory and standard settings have been efficient in assessing professionalism, communication, procedural, and clinical skills. Ideally, simulation developers should reflect on the purpose of the simulation to determine if the focus will be on teaching or learning. If the focus is on teaching, then assessments should focus on performance criteria with exercises for a set of skill-based experiences – this assesses the teaching method's effectiveness in task training. Alternatively, if the focus of the simulation is to determine higher-order learning, then the assessment should be designed to measure multiple integrated abilities such as factual understanding, problem-solving, analysis, and synthesis. In general, multiple assessment methods are necessary to capture all relevant aspects of clinical competency. For higher-order cognitive assessment (knowledge, application, and synthesis of knowledge), context-based multiple-choice questions (MCQ), extended matching items, and short answer questions are appropriate. For the demonstration of skills mastery, a multi-station objective structured clinical examination (OSCE) is viable. Performance-based assessments such as Mini-Clinical Evaluation Exercise (mini-CEX) and Direct Observation of Procedural Skills (DOPS) are appropriate to have a positive effect on learner comprehension. Alternatively, for the advanced professional continuing learner, a clinical work sampling and portfolio or logbook may be used.In an assessment, the developers select an assessment instrument with known characteristics. A wide range of assessment tools is currently available for assessment of knowledge and application and performance assessment. The assessment materials are then created around learning objectives, and the developers directly control all aspects of delivery and assessment. The content should relate to the learning objectives and the test comprehensive enough that it produces reliable scores. This ensures that the performance is wholly attributable to the learner – and not an artifact of curriculum planning or execution. Additionally, different versions of the assessment that are comparable in difficulty will permit comparisons among examinees and against standards.Learner assessment is a wide-ranging decision-making process with implications beyond student achievement alone. It is also related to program evaluation and provides important information to determine program effectiveness. Valid and reliable assessments satisfy accreditation needs and contribute to student learning.