Abstract
A proliferation of retail online training materials exists for many industries, but often the person in charge of choosing the most appropriate online training materials is neither a training expert nor versed in best practices associated with online training. To this end, it is critical that uninformed decision makers have access to an easy-to-use evaluation tool which allows for the assessment of strengths and weaknesses among multiple online training programs. Additionally, this tool must take into account the context of the training situation to ensure the chosen program is not only instructionally sound but also meets contextually specific training needs. This article describes the creation, testing, and application of the Customizable Tool for Online Training Evaluation (CTOTE), an evaluation instrument developed to help decision makers: (1) assess multiple online training programs against known best practices and (2) consider context-specific training needs via a weighting process. The three-step development process is explained including item selection and revision, determination of content validity and reliability, and the use of a Delphi panel to inform contextualized weighting. The instrument is then tested across multiple online training programs with results compared to an established online training evaluation instrument to illustrate the impact of the contextualized weighting. Lastly, the application of the instrument in a specific industry setting (food service) is presented to demonstrate the effectiveness of the instrument in this setting and to establish the potential of the CTOTE in helping uninformed decision makers assess multiple online training programs and make effective context-specific purchasing decisions.
Similar content being viewed by others
References
Agariya, A. K., & Singh, D. (2012). e-Learning quality: Scale development and validation in Indian context. Knowledge Management & E-Learning: An International Journal (KM&EL), 4(4), 500–517. Retrieved from http://www.kmel-journal.org/ojs/index.php/online-publication/article/view/139/163
Alkhattabi, M., Neagu, D., & Cullen, A. (2010). Information quality framework for e-learning systems. Knowledge Management & E-Learning: An International Journal (KM&EL), 2(4), 340–362. Retrieved from http://www.kmel-journal.org/ojs/index.php/online-publication/article/view/21/62
Almanza, B. A., & Nesmith, M. S. (2004). Food safety certification regulations in the United States. Journal of Environmental Health, 66(9), 10–14.
ATD Research. (2015). 2014 state of the industry report. Alexandria, VA: Association for Talent Development.
Barker, N. K. (2004). The quality dilemma in online education. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 385–421). Retrieved from http://www.aupress.ca/index.php/books/120146
Barker, N. K. (2007). E-learning quality standards for consumer protection and consumer confidence: A Canadian case study in e-learning quality assurance. Journal of Educational Technology and Society, 10(2), 109–119. Retrieved from http://www.ifets.info/journals/10_2/10.pdf
Batalla-Busquets, J. M., & Pacheco-Bernal, C. (2013). On-the-job e-learning: Workers’ attitudes and perceptions. The International Review of Research in Open and Distributed Learning, 14(1), 40–64. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1304/2444
Becker, K., Fleming, J., & Keijsers, W. (2012). E-learning: Ageing workforce versus technology-savvy generation. Education + Training, 54(5), 385–400. 10.1108/00400911211244687.
Caudill, J. G., & Reeves, B. (2014). Strategic management of workplace e-learning. In S. Hai-Jew (Ed.), Remote workforce training: Effective technologies and strategies (pp. 17–27). Hershey, PA: IGI Global.
Egan, M. B., Raats, M. M., Grubb, S. M., Eves, A., Lumbers, M. L., Dean, M. S., & Adams, M. R. (2007). A review of food safety and food hygiene training studies in the commercial sector. Food Control, 18, 1180–1190. doi:10.1016/j.foodcont.2006.08.001.
Frash, R., Jr., Binkley, M., Nelson, D., & Almanza, B. (2006). Transfer of training efficacy in US food safety accreditation. Journal of Culinary Science & Technology, 4(2–3), 7–38.
Gillis, L. (2000). Quality standards for evaluating multimedia and online training. Whitby, ON: McGraw-Hill Ryerson.
Grollman, W. K., & Cannon, D. (2003). eLearning: A better chalkboard. Financial Executive, 19(8), 45–47.
Howton, J., Keiffer, E., Murphy, C. A., Sirsat, A. S., O’Bryan, C. A., Ricke, S. C., Neal, J. A. (2016). A comparison of online food safety training programs using the Customizable Tool for Online Training Evaluation. Food Control, 59, 82–87. doi:10.1016/j.foodcont.2015.04.034.
Illinois Online Network. (2010). ION’s quality online course initiative. Retrieved from http://www.ion.uillinois.edu/initiatives/qoci/
Istrate, O. (2013). Criteria for e-learning programmes evaluation. Journal of Advanced Distributed Learning Technology, 1(1), 27–35. Retrieved from http://jadlet.com/index.php/jadlet/article/view/12/18
Jan, P. T., Lu, H. P., & Chou, T. C. (2012). The adoption of e-learning: An institutional theory perspective. Turkish Online Journal of Educational Technology-TOJET, 11(3), 326–343. Retrieved from http://www.tojet.net/articles/v11i3/11331.pdf
MarylandOnline. (2011). Quality Matters rubric standards 2011–2013 edition with assigned point values. Retrieved from http://www.qmprogram.org/rubric
McCue, T. J. (2014, October). Online learning industry poised for $107 billion in 2017. Forbes. Retrieved from http://www.forbes.com/sites/tjmccue/2014/08/27/online-learning-industry-poised-for-107-billion-in-2015/
Mosharraf, M., & Taghiyareh, F. (2013). Qualitative development of e-learning environments through a learner relationship management methodology. Knowledge Management & E-Learning: An International Journal (KM&EL), 5(1), 56–65. Retrieved from http://www.kmel-journal.org/ojs/index.php/online-publication/article/view/211/171
Murphy, C. A., Keiffer, E. A., Neal, J. A., & Crandall, P. G. (2013). A customizable evaluation instrument to facilitate comparisons of existing online training programs. Knowledge Management & E-Learning: An International Journal, 5(3), 251–268. Retrieved from http://www.kmel-journal.org/ojs/index.php/online-publication/article/view/242/186
Neal, J. A., Dawson, M., & Madera, J. M. (2011). Identifying food safety concerns when communication barriers exist. Journal of Food Science Education, 10, 36–44. doi:10.1111/j.1541-4329.2011.00128.x.
Neal, J. A., Murphy, C. A., Crandall, P. G., O’Bryan, C. A., Keiffer, E., & Ricke, S. C. (2011). Development of an evaluation tool for online food safety training programs. Journal of Food Science Education, 10(1), 9–12. Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/j.1541-4329.2010.00112.x/epdf
Nunnaly, J. (1978). Psychometric theory. New York, NY: McGraw-Hill.
Pappas, C. (2015, January). The top e-learning statistics and facts for 2015 you need to know. eLearning Industry. Retrieved from http://elearningindustry.com/elearning-statistics-and-facts-for-2015
Pisik, G. B. (1997). Is this course instructional sound? A guide to evaluating online training courses. Educational Technology, 37(4), 50–59.
Pisik, G. B. (2004). Is this course instructionally sound? An updated guide to evaluating online training courses. Educational Technology, 44(1), 57–60. http://c.ymcdn.com/sites/aect.site-ym.com/resource/resmgr/Publication_PDFs/ET_Contents_Pages_2000-2015a.PDF
Santerre, C. R. (2005). X-train: Teaching professionals remotely. The Journal of Nutrition, 135(5), 1248–1252. Retrieved from http://jn.nutrition.org/content/135/5/1248.full
Seufert, S. (2002). eLearning business models: Strategies, success factors, and best practice examples. In C. Wankel & R. DeFillippi (Eds.), Rethinking management education for the twenty-first century (pp. 109–132). Greenwich, CT: Information Age Publishing.
Skillsoft. (2013). From boardroom to desktop: Skillsoft connects corporate strategy to the performance needs of individual learners with new Enterprise Learning Connection [Press Release]. Retrieved from http://www.skillsoft-usa.net/about/press_room/press_releases/may_02_06_elc.asp
Steele, J. (2015). Comparing computer-based employee training with traditional face-to-face methods. In D. Slykhuis & G. Marks (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2015 (pp. 504–507). Chesapeake, VA: Association for the Advancement of Computing in Education (AACE). Retrieved from http://www.editlib.org/p/150043/
Strother, J. B. (2002). An assessment of the effectiveness of e-learning in corporate training programs. The International Review of Research in Open and Distance Learning, 3(1). Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/83/160Ubell
Training. (2014, November/December). 2014 training industry report. Training, 51(8), 16–29. Retrieved from http://www.trainingmag.com/sites/default/files/magazines/2014_11/2014-Industry-Report.pdf
Turner, R. C., & Carlson, L. (2003). Indexes of item-objective congruence for multidimensional items. International Journal of Testing, 3(2), 163–171. doi:10.1207/S15327574IJT0302_5.
Van Rooij, S. W. (2011). Instructional design and project management: Complementary or divergent? Educational Technology Research and Development, 59(1), 139–158.
Zaied, A. N. H. (2012). Multi-criteria evaluation approach for e-learning technologies: Selection criteria using AHP. International Journal on E-Learning, 11(4), 465–485.
Zhang, D., & Nunamaker, J. F. (2003). Powering e-learning in the new millennium: An overview of e-learning and enabling technology. Information Systems Frontiers, 5(2), 207–218. doi:10.1023/A:1022609809036.
Acknowledgments
This research was supported in part by an USDA National Integrated Food Safety Initiative Grant to the authors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this entry
Cite this entry
Murphy, C.A., Keiffer, E.A., Neal, J.A., Howton, J. (2016). Customizable Tool for Online Training Evaluation. In: Spector, M., Lockee, B., Childress, M. (eds) Learning, Design, and Technology. Springer, Cham. https://doi.org/10.1007/978-3-319-17727-4_22-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-17727-4_22-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Online ISBN: 978-3-319-17727-4
eBook Packages: Springer Reference EducationReference Module Humanities and Social SciencesReference Module Education