Sarah Kelle, Hendrik Schossau, Richard T. Justenhoven & Aaron M. Smith
Through the digital revolution, not only the way we live changes a lot. This is true for online assessments, too, which are no longer restricted to desktop computers, but can also be completed on mobile devices. The goal of this study is to investigate the degree of mobile device usage for online assessments. Expecting the mobile device usage to increase over the years, the study aims to display equivalent test scores for a mobile-first cognitive ability test that was completed either on a mobile or non-mobile device. Data from recruitment processes was collected from 2017 to 2019. N = 837,485 candidates completed the ability test. Mobile usage was tracked over the years and the sample was split by different demographics. A t-test was applied to investigate differences in mean performance between different devices. The results reveal a rising number of test takers using mobile devices for online assessment. The t-test to examine the equivalence between mobile and non-mobile devices was significant, the effect size rather small. Hence, equivalence can be assumed for a practical use case. The results of the study demonstrate the need to be prepared for the increasing administrations of assessment on mobile devices. To meet this challenge and to ensure equivalent results between mobile and non-mobile devices, building online assessments mobile-first can be considered. This becomes especially relevant for cognitively loaded assessments like ability tests.