Expert-Rater Agreement and Inter-/Intrarater Reliability for the Furtado–Gallagher Computerized Observational Movement Pattern Assessment System
Abstract
Fundamental movement skills (FMS) play a crucial role in a child’s motor skill development, as their mastery is related to the acquisition and mastery of specialized sport skills and to a child’s general health through the prevention of weight gain and increased level of physical activity. Thus, the accurate and reliable measurement of FMS is of related importance. This article presents the results of two studies intended to assess expert-rater agreement and inter-/intrarater scoring reliability in the administration of a new observational FMS instrument, the Furtado– Gallagher Computerized Observational Movement Pattern Assessment System (FG-COMPASS). In Study 1, we found that four of the six scales that underwent modifications in a previous study yielded acceptable expert agreement, permitting their retention in the revised test, while the two remaining scales measuring side sliding and leaping yielded insufficient expert agreement, leading to their removal from the test. In Study 2, weighted kappa values reflected very good interrater agreement among three raters across two test administrations for the Locomotor and Manipulative subtests and the Total score (all skills combined). Future studies should (a) collect validity and reliability evidence for two additional skills planned for inclusion in the Locomotor subtest, (b) assess the impact of varying the number of trials given when assessing motor skill performances in life settings, and (c) test the feasibility of the web-based training tool and mobile version of the instrument., Fundamental movement skills (FMS) play a crucial role in a child’s motor skill development, as their mastery is related to the acquisition and mastery of specialized sport skills and to a child’s general health through the prevention of weight gain and increased level of physical activity. Thus, the accurate and reliable measurement of FMS is of related importance. This article presents the results of two studies intended to assess expert-rater agreement and inter-/intrarater scoring reliability in the administration of a new observational FMS instrument, the Furtado– Gallagher Computerized Observational Movement Pattern Assessment System (FG-COMPASS). In Study 1, we found that four of the six scales that underwent modifications in a previous study yielded acceptable expert agreement, permitting their retention in the revised test, while the two remaining scales measuring side sliding and leaping yielded insufficient expert agreement, leading to their removal from the test. In Study 2, weighted kappa values reflected very good interrater agreement among three raters across two test administrations for the Locomotor and Manipulative subtests and the Total score (all skills combined). Future studies should (a) collect validity and reliability evidence for two additional skills planned for inclusion in the Locomotor subtest, (b) assess the impact of varying the number of trials given when assessing motor skill performances in life settings, and (c) test the feasibility of the web-based training tool and mobile version of the instrument.
