Author: Nedjma Koval-Saifi & Jan Plass
Full Citation: Koval-Saifi, N., & Plass, J. (2018). Antura and the Letters: Impact and technical evaluation. Washington, DC: World Vision and Foundation for Information Technology Education and Development. Retrieved from http://dl4d.org/portfolio-items/eduapp4syria/
Overview: Integrated Services, Indigenous Solutions (INTEGRATED), in partnership with Consortium for Research and Evaluation of Advanced Technologies in Education (CREATE) of New York University, conducted an impact evaluation using a longitudinal quasi-experimental design to estimate the impacts of the EduApp4Syria games on children’s literacy and psychosocial outcomes over time.
Methodology: In this design, we compared growth in literacy outcomes for two groups of children, each using one of the games, to a group of children in matched environments who did not have access to the games. All children in the study had little or no schooling and lived in the Azraq refugee camp in Jordan. This design relies on comparing two groups that experience the same historical trends and events over time. Simultaneously, INTEGRATED and CREATE conducted a technical evaluation, working closely with ACR GCD and DL4D to refine key game evaluation questions and identify usability improvements for the next release of the games; provide feedback on the open beta versions built in Phase 2 of the competition; and inform improvements to be applied in Phase 3.
Findings: Findings include:
- Overall, the game effectively introduces children to the basics of Arabic literacy and has resulted in positive learning outcomes across all age groups and genders with a relatively low dosage of 27 hours. Absolute gains tended to concentrate in foundational Early Grade Reading Assessment (EGRA) subtasks (letter, syllable, and oral reading fluency [ORF]), gaining an additional two to three letters/syllables/words on average. Zero scores also decreased across all subtasks.
- Boys tended to have much higher gains and rates of change across subtasks than girls. While regression analysis did not demonstrate gender being a significant factor influencing subtask outcomes, it remains an area for further investigation to explain why boys outperformed girls in every subtask.
- The rate of smartphone penetration among Syrians participating in the study is very high, demonstrating promising implications for the widespread download and use of literacy games such as Antura and the Letters.
- Children were effectively rewarded by the game, through verbal praise and in-game rewards, and received feedback from the game. However, some children were unable to perform the tasks requested as fast or with as much precision as the older children. Younger children tended to be more frustrated with mini-games requesting more advanced fine motor skills, precision, and speed and tended to have difficulty in playing mini-games.
- At a psychosocial level, the game appears to have supported the development of positive social outcomes, as demonstrated by both an assessment employing the results of the Strengths and Difficulties Questionnaire (SDQ) and parental feedback. The sense of accomplishment and learning seems to have contributed to children feeling happy. The game enabled them to also feel a sense of ownership and attachment either through child profiles or through the customization of Antura. Peer interaction increased during game play, which had a positive impact on emotional states and social behaviors, while gaming engendered high levels of motivation to attend summer camp. This had implications for motivations to learn.