Assessment of borderline clinical competence using generalisability and decision studies in dental skills training

INTRODUCTION: Generalisability coefficients are widely used as a measure of reliability in educational sciences. However, coefficients do not inform about how much individual and or educational variance is needed for borderline competence. Probing depth measurements were used as an example to demonstrate the use of generalisability and decision studies in educational decision-making. METHOD: All six probing depths of seven elements of the first quadrant were assessed in a random crossed four-way elements (7), probing depth (6) observers (7-8) and repetitions design for both students (8) and faculty staff (7). Borderline competence was assessed in four decision study designs. RESULTS: In decision studies the generalisability coefficient varied from 0.31 to 0.88 for the first single and fourth repeated study design, respectively. The standard error of the measurement varied from 1.73 to 0.64 mm with a corresponding 95% confidence interval varying from 3 to 1 mm. CONCLUSION: Repeated measurement is not the most efficient way to improve educational decision-making. What is new to this approach is that we used the 95% confidence interval of the standard error of measurement as a measure for borderline competence expressed in the unity of the measurement instrument. Correctly performing the procedure of probing depth assessment is of clinical and educational importance in deciding whether further treatment is necessary and whether students fail or pass their exams.INTRODUCTION: Generalisability coefficients are widely used as a measure of reliability in educational sciences. However, coefficients do not inform about how much individual and or educational variance is needed for borderline competence. Probing depth measurements were used as an example to demonstrate the use of generalisability and decision studies in educational decision-making. METHOD: All six probing depths of seven elements of the first quadrant were assessed in a random crossed four-way elements (7), probing depth (6) observers (7-8) and repetitions design for both students (8) and faculty staff (7). Borderline competence was assessed in four decision study designs. RESULTS: In decision studies the generalisability coefficient varied from 0.31 to 0.88 for the first single and fourth repeated study design, respectively. The standard error of the measurement varied from 1.73 to 0.64 mm with a corresponding 95% confidence interval varying from 3 to 1 mm. CONCLUSION: Repeated measurement is not the most efficient way to improve educational decision-making. What is new to this approach is that we used the 95% confidence interval of the standard error of measurement as a measure for borderline competence expressed in the unity of the measurement instrument. Correctly performing the procedure of probing depth assessment is of clinical and educational importance in deciding whether further treatment is necessary and whether students fail or pass their exams.