Author: Nedjma Koval-Saifi & Jan Plass
Full Citation: Koval-Saifi, N., & Plass, J. (2018). Feed the Monster: Impact and technical evaluation. Washington, DC: World Vision and Foundation for Information Technology Education and Development. Retrieved from http://dl4d.org/portfolio-items/eduapp4syria/
Overview: Integrated Services, Indigenous Solutions (INTEGRATED), in partnership with Consortium for Research and Evaluation of Advanced Technologies in Education (CREATE) of New York University, conducted an impact evaluation using a longitudinal quasi-experimental design to estimate the impacts of the EduApp4Syria games on children’s literacy and psychosocial outcomes over time.
Methodology: In this design, we compared growth in literacy outcomes for two groups of children, each using one of the games, to a group of children in matched environments who did not have access to the games. All children in the study had little or no schooling and lived in the Azraq refugee camp in Jordan. This design relies on comparing two groups that experience the same historical trends and events over time. Simultaneously, INTEGRATED and CREATE conducted a technical evaluation, working closely with ACR GCD and DL4D to refine key game evaluation questions and identify usability improvements for the next release of the games.
Findings: Findings include:
Overall, the game resulted in positive learning outcomes across all age groups and genders with a relatively low dosage of 22 hours. Absolute gains were greater in learning of foundational literacy (letters and syllables), and less with reading fluency.
For children with previous exposure to basic literacy, the game, after letter and letter sound introduction, effectively reinforces previously acquired skills. Nevertheless, younger children showed higher rates of change across all subtasks.
However, the fast pace and lack of differentiation between letters, their voweled variations (letters with vowel diacritics), and syllables may have resulted in confusion among children with low levels of literacy. This was demonstrated by the increase in zero scores in Early Grade Reading Assessment (EGRA) subtasks among a subset of younger and/or less literate children.
The rate of smartphone penetration among Syrians participating in the study is very high, demonstrating promising implications for the widespread download and use of literacy apps such as Feed the Monster.
At a psychosocial level, the game appears to have supported the development of positive social outcomes, as demonstrated by both Strengths and Difficulties Questionnaire (SDQ) results and parental feedback. Children wanted to share the game and their learning with both siblings and peers, demonstrating impact on emotional states and social behaviors. Peer interaction increased because of game play, while gaming engendered high levels of motivation to attend the summer camp, which had implications for children’s motivations to learn.