7 essential criteria for evaluating mobile educational applications

As an instructional designer, I have had the opportunity to not only design online and face-to-face training programs but also to evaluate eLearning courseware and suggest improvements in order to validate its relevance and effectiveness.

Graphic design, instructional strategies, assessment instruments, media integration, and interactivity levels are some of the aspects I often consider when evaluating eLearning.

Now, as I am making a transition from eLearning to mLearning design, I have started to rethink how to approach the creation phase and also, the evaluation of prototypes and mobile apps.

In this post, I´d like to share some criteria I have drafted by drawing upon some of the basic considerations to evaluate learning products and by applying them to mobile apps.

As a case study to exemplify each of the criteria, I have chosen the Nutrition Guru app for iPad, which was developed by Bottom-Line Performance and which is based on the pedagogical principles of spaced learning and repetition. Nutrition Guru is the native app version of Knowledge Guru, a game engine businesses use to teach fact-based knowledge such as product features, industry trends, and sales messages. Educators can also use the tool to help students with any academic subject that is difficult to memorize. Nutrition Guru is a sample for clients to preview how the game engine’s learning principles and features work.

Criterion 1: Content

When evaluating a learning product, I mainly focus on six aspects related to the content: synthesis, presentation, accuracy, relevance, its connection to the learning objectives and its adequacy for the target audience.

In the case of Knowledge Guru app, it cleverly encapsulates the learning content into layers (3 paths and 4-5 questions in each path) in order to facilitate information processing and retention through bite-sized chunks.

Besides, the app’s focus has a strong connection to learning purposes, which have been narrowly defined. Focus is a key concept for mLearning solutions.

Criterion 2: Personalization

The possibility of adjusting content and settings to meet specific needs of the learner does not only guarantee engagement throughout the learning process but also contributes to the acquisition of new knowledge and shows the learner how to apply it to real-life situations.

In the case of Nutrition Guru, even though the learner can enter his/her name and have an identity in a community of players, I think that personalization could be taken to the next level by allowing the learner to enter personal data (gender, age, height, weight, etc.) and only answer questions related to his/her own characteristics. This will also allow him/her to assume more responsibility in the learning process and therefore, play an even more active role. Another activity could be to answer direct questions about the learner´s habits rather than hypothetical situations. Or else, learners could take different actions in the scenarios given and those actions could take them closer or further away from the goals of the game.

Criterion 3: Meaningful Feedback

The type of feedback learners receive when performing certain actions is a crucial part of the design process as it can play a very important role in how the learners understand the subject, perceive their mistakes and gain new insights.

When using this app, learners are provided with specific feedback at the right moment. The feedback is meaningful, relevant and complete and I really like the designers’ approach in this regard. The feedback has proven to be a great way to provide self-assessment opportunities.

Criterion 4: High-order thinking skills

Skill development implies a process where the learner acquires a set of tools or learns how to use certain resources in order to solve a specific problem. To facilitate this progression towards a higher state of knowledge, it is necessary to design activities that can display elements that foster high-order thinking.

The Knowledge Guru game engine encourages the use of higher-order thinking skills, including evaluating, analyzing, understanding by means of a great narrative, scenarios, and feedback. Visual aids, like tables to display specific data, are very useful to promote additional opportunities for cognitive processing.


Criterion 5: Usability and technical performance

Jacob Nielsen defines usability by 5 quality components:

  • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
  • Efficiency: Once users have learned the design, how quickly can they perform tasks?
  • Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency?
  • Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  • Satisfaction: How pleasant is it to use the design?

In general terms, the app´s interface elements have been appropriately selected. User guidance and error management are some of the key strengths of this product to help the user accomplish tasks.

However, navigation could be improved by reconsidering some elements placement to match the way in which people hold and use touch-enabled devices.

Criterion 6: Interactivity and engagement

Interactivity has to do with the type of actions a learner is required to do and the type of response he/she gets as a consequence. The level of interactivity and the relevance of the actions allow for engagement and long-term information storage.

What I found really engaging about this app has to do with gathering information from the tables or from the feedback window in order to successfully answer a question. The real-life scenarios depicted are also quite compelling and demand a cognitive effort.  Furthermore, in the final section, which is called Guru Grab Bag, the learner has the opportunity to interact with information from previous sections as well as with new information.

In order to increase the level of interactivity, learners could also be asked to adjust some variables and evaluate different results in some questions.

Criterion 7: Integration of social interactions

In a previous article, I described some fundamental guidelines for mobile user experience (UX) design, one of them being the need for leveraging mobile users´ most common behaviors to promote learning. Learners use mobile devices to search, get informed and connect with others.

The fact that the learner can see other people´s scores somehow adds a sense of being part of a community and it also adds motivation as it appeals to the competitive nature in us. Perhaps, Knowledge Guru games could also provide opportunities for collaborative practice and idea-sharing in future versions.

I really enjoyed evaluating a mobile app for educational purposes, and I would love to hear your comments on these criteria that could help me to develop a more complete and effective evaluation rubric.

Privacy Preference Center

%d bloggers like this: