Abstract
In the complete software development life cycle (SDLC), testing and maintenance stand slowest, and the reason being entire process is manual or manually maintained automation. Over time, companies have realized that it is very costly and time consuming. The biggest reason for manual maintenance of automation is its dependency on dynamic properties of the UI elements (xpath, class, ID) which directly depend on document object model (DOM). Cloud releases are very frequent, and changes in the UI properties are expected. In this paper, we are presenting a UI technology agnostic approach, which does not depend on the DOM and UI properties of the application rather behaves like a human and visually parses the screen. We use artificial intelligence to detect the UI elements, and even, just a mock-up of a screen can be parsed this way. This also enables test-driven development (TDD) of UI, among many other use cases. We used annotated UI image data for training our deep learning model, and the method has been validated with 94% accuracy on new UI elements. This approach does not require manual maintenance of the generated automates unless there is a functional change in the application. We propose UI technology agnostic, zero touch, self-healing UI automation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 779–788
Li J, Liang X, Shen S, Xu T, Feng J, Yan S (2018) Scale-aware fast R-CNN for pedestrian detection. IEEE Trans Multimedia 20(4):985–996
Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448
Li P, Huynh T, Reformat M, Miller J (2007) A practical approach to testing GUI systems. Empirical Softw Eng 12(4):331–357
Dallmeier V, Burger M, Orth T, Zeller A (2013) WebMate: generating test cases for web 2.0. In: International conference on software quality. Springer, Berlin, Heidelberg, pp 55–69
Börjesson E, Feldt R (2012) Automated system testing using visual gui testing tools: a comparative study in industry. In: 2012 IEEE fifth international conference on software testing, verification and validation. IEEE, pp 350–359
Memon AM (2002) GUI testing: pitfalls and process. Computer 8:87–88
Benedikt M, Freire J, Godefroid P (2002) VeriWeb: Automatically testing dynamic web sites. In: Proceedings of 11th international world wide web conference (WWW’2002)
Maheshwari S, Jain DC (2012) A comparative analysis of different types of models in software development life cycle. Int J Adv Res Comput Sci Softw Eng 2(5):285–290
Monier M, El-mahdy MM (2015) Evaluation of automated web testing tools. Int J Comput Appl Technol Res 4(5):405–408
Kazmi R, Afzal RM, Bajwa IS (2013) Teeter-totter in testing. In: 2013 Eighth international conference on digital information management (ICDIM). IEEE, pp 194–198
Memon AM (2007) An event-flow model of GUI-based applications for testing. Softw Test Verification Reliab 17(3):137–157
Sharma M, Angmo R (2014) Web based automation testing and tools. Int J Comput Sci Inf Technol 5(1):908–912
Jain A, Jain M, Dhankar S (2014) A Comparison of RANOREX and QTP Automated Testing Tools and their impact on Software Testing. IJEMS 1(1):8–12
Harrold MJ (2000) Testing: a roadmap. In: Proceedings of the conference on the future of software engineering. ACM, pp 61–72
Fuggetta A, Di Nitto E (2014) Software process. In: Proceedings of the on future of software engineering. ACM, pp 1–12
Chang TH, Yeh T, Miller RC (2010) GUI testing using computer vision. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1535–1544
Sjösten-Andersson E, Pareto L (2006) Costs and benefits of structure-aware capture/replay tools. In: SERPS’06, p 3
Nagarani P, VenkataRamanaChary R (2012) A tool based approach for automation of GUI applications. In: 2012 Third international conference on computing communication and networking technologies (ICCCNT). IEEE, pp 1–6
Yeh T, Chang TH, Miller RC (2009) Sikuli: using GUI screenshots for search and automation. In: Proceedings of the 22nd annual ACM symposium on user interface software and technology. ACM, pp 183–192
Amant RS, Lieberman H, Potter R, Zettlemoyer L (2001) Visual generalization in programming by example. In: Your wish is my command, pp 371–XIX
Jovic M, Adamoli A, Zaparanuks D, Hauswirth M (2010)Automating performance testing of interactive java applications. In: Proceedings of the 5th workshop on automation of software test. ACM, pp 8–15
Andrica S, Candea G (2011) WaRR: a tool for high-fidelity web application record and replay
Everingham M, Winn J (2011) The PASCAL visual object classes challenge 2012 (VOC2012) development kit. Rep, Pattern Analysis, Statistical Modelling and Computational Learning, Tech
Yin XC, Yin X, Huang K, Hao HW (2014) Robust text detection in natural scene images. IEEE Trans Pattern Anal Mach Intell 36(5):970–983
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M (2016) Tensorflow: a system for large-scale machine learning. In: OSDI, vol 16, pp 265–283
Agarwal S, Awan A, Roth D (2004) Learning to detect objects in images via a sparse, part-based representation. IEEE Trans Pattern Anal Mach Intell 26(11):1475–1490
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Singh, M.K., Fernandes, W.M., Rashid, M.S. (2021). Robust UI Automation Using Deep Learning and Optical Character Recognition (OCR). In: Gunjan, V.K., Zurada, J.M. (eds) Proceedings of International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications. Advances in Intelligent Systems and Computing, vol 1245. Springer, Singapore. https://doi.org/10.1007/978-981-15-7234-0_4
Download citation
DOI: https://doi.org/10.1007/978-981-15-7234-0_4
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-7233-3
Online ISBN: 978-981-15-7234-0
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)