Skip to main content
SpringerLink
Log in

Visual Language Theory: Towards a Human-Computer Interaction Perspective

  • Chapter
Visual Language Theory

Abstract

The main reason for using visual languages is that they are often far more convenient to the user than traditional textual languages. Therefore, visual languages intended for use by both computers and humans ought to be designed and analyzed not only from the perspective of computational resource requirements, but also from the perspective of languages that are cognitively usable and useful. Theoretical and practical research on visual languages needs to take into account the full context of a coupled human-computer system in which the visual language facilitates interactions between the computational and the cognitive parts. This implies that theoretical analyses ought to address issues of comprehension, reasoning, and interaction in the cognitive realm as well as issues of visual program parsing, execution, and feedback in the computational realm. The human aspect is crucial to visual languages, and therefore we advocate a correspondingly broadened scope of inquiry for visual language research. In this chapter we describe aspects of human use of visual languages that ought to be important considerations in visual language research and design, and summarize research from related fields such as software visualization and diagrammatic reasoning that addresses these issues. A framework consistent with the broadened scope of visual language research is proposed and used to categorize and discuss several formalizations and implemented systems. In the course of showing how a sample of current work fits into this framework, open issues and fruitful directions for future research are also identified.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. H. Abelson, G.J. Sussman, and J. Sussman.Structure and Interpretation of Computer Programs.MIT Press, Cambridge, 2nd edition, 1996.

    MATH  Google Scholar 

  2. M. Andries, G. Engels, and J. Rekers. How to represent a visual specification. Inthis volume.

    Google Scholar 

  3. A. Antonietti. Why does mental visualization facilitate problem solving? In R.H. Logie and M. Denis, editorsMental Images in Human Cognitionpages 211–227. Elsevier Science Publishers, Amsterdam, 1991.

    Chapter  Google Scholar 

  4. R. Arnheim.Visual Thinking.University of California Press, Berkeley, 1969.

    Google Scholar 

  5. A. Badre and J. Allen. Graphic language representation and programming behavior. In S. Salvendy and M.J. Smith, editorsDesigning and Using Human-Computer Interfaces and Knowledge Based Systemspages 59–65. Elsevier Science Publishers, Amsterdam, 1989.

    Google Scholar 

  6. J. Barwise and J. Etchemendy. Heterogeneous logic. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 211–234. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.

    Google Scholar 

  7. J. Bertin.Semiology of Graphics.1967. English translation by W.J. Berg, University of Wisconsin Press, Madison, 1983.

    Google Scholar 

  8. A.F. Blackwell. Metacognitive theories of visual programming: What do we think we are doing? InIEEE Workshop on Visual Languagespages 240–246. IEEE Computer Society Press, Los Alamitos, 1996.

    Google Scholar 

  9. P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Formalizing visual languages. InIEEE Workshop on Visual Languagespages 45–52. IEEE Computer Society Press, Los Alamitos, 1995.

    Google Scholar 

  10. P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Formalizing visual languages. In IEEE Workshop on Visual Languages, pages 45–52. IEEE Computer Society Press, Los Alamitos, 1995

    Google Scholar 

  11. P. Bottoni, M.F. Costabile, S. Levialdi, and P. Mussio. Specification of visual languages as means for interaction. Inthis volume.

    Google Scholar 

  12. F.P. Brooks. No silver bullet: Essence and accidents of software engineering.IEEE Computer20(4):10–19, 1987.

    Article  MathSciNet  Google Scholar 

  13. M.H. Brown and R. Sedgewick. Techniques for algorithm animation.IEEE Software2(1):28–38, 1985.

    Article  Google Scholar 

  14. M.D. Byrne, R. Catrambone, and J.T. Stasko. Do algorithm animations aid learning? Technical Report GIT-GVU-96–18, Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, USA, August 1996.

    Google Scholar 

  15. CACM. Special section on educational technologyCommunications of the ACM39(4), April, 1996.

    Google Scholar 

  16. S.-K. Chang Visual languages: A tutorial and survey.IEEE Software4:29–39, 1987.

    Article  Google Scholar 

  17. T.B. Dinesh and S. Üsküdarli. Input and output for specified visual languages. Inthis volume.

    Google Scholar 

  18. S. Douglas, C. Hundhausen, and D. McKeown. Toward empirically-based software visualization languages. InProc. IEEE Symposium on Visual Languagespages 342–349. IEEE Computer Society Press, Los Alamitos, 1995.

    Google Scholar 

  19. Y. Engelhardt, J. Bruin, T. Janssen, and R. Scha. The visual grammar of information graphics. In N.H. Narayanan and J. Damski, editorsProc. AID’96 Workshop on Visual Representation Reasoning and Interaction in DesignKey Center for Design Computing, University of Sydney, 1996.

    Google Scholar 

  20. M. Erwig and B. Meyer. Heterogeneous visual languages: Integrating visual and textual programming. InProc. IEEE Symposium on Visual Languagespages 318–325. IEEE Computer Society Press, Los Alamitos, 1995.

    Google Scholar 

  21. M.R. Frank and J.D. Foley. A pure reasoning engine for programming by example. Technical Report GIT-GVU-94–11, Graphics, Visualization and Usability Center, Georgia Institute of Technology, Atlanta, USA, April 1994.

    Google Scholar 

  22. E. Freeman, D. Gelernter, and S. Jagannathan. In search of a simple visual vocabulary. InProc. IEEE Symposium on Visual Languagespages 302–309. IEEE Computer Society Press, Los Alamitos, 1995.

    Google Scholar 

  23. G.W. Fumas. New graphical reasoning models for understanding graphical interfaces. InProc. Human Factors in Computing Systems Conference (CHI’91)pages 71–78. ACM Press, New York, 1991.

    Google Scholar 

  24. G.W. Fumas. Reasoning with diagrams only. In N.H. Narayanan, editorProc. AAAI Spring Symposium on Reasoning with Diagrammatic Representationspages 118–123, Technical Report SS-92–02, AAAI Press, Menlo Park, March 1992.

    Google Scholar 

  25. J.S. Gero and M. Yan. Shape emergence by symbolic reasoning.Environment and Planning B: Planning and Design21:191–218, 1994.

    Article  Google Scholar 

  26. J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editors.Diagrammatic Reasoning: Cognitive and Computational Perspectives.AAAI Press, Menlo Park, MIT Press, Cambridge, 1995.

    Google Scholar 

  27. E.P. Glinert. Nontextual programming environments. In S.-K. Chang, editorPrinciples of Visual Programming Systemspages 144230. Prentice-Hall, New York, 1990.

    Google Scholar 

  28. V. Goel.Sketches of Thought.MIT Press, Cambridge, 1995.

    Google Scholar 

  29. E.H. Gombrich.Art and Illusion: A Study in the Psychology of Pictorial Representations.Phaidon, London, 1968.

    Google Scholar 

  30. J.M. Gooday and A.G. Cohn. Visual language syntax and semantics: A spatial logic approach. In K. Marriott and B. Meyer, editorsInternational Workshop on Theory of Visual LanguagesGubbio, Italy, 1996. Monash University.

    Google Scholar 

  31. N. Goodman.Languages of Art: An Approach to a Theory of Symbols.Hackett Publishing Company, Indianapolis, 1976.

    Google Scholar 

  32. T.R.G. Green. Cognitive dimensions of notations. In R. Winder and A. Sutcliffe, editorsPeople and ComputersV, Cambridge University Press, 1990.

    Google Scholar 

  33. T.R.G. Green and M. Petre. When visual programs are harder to read than textual programs. In G.C. van der Veer, M.J. Tauber, S. Bagnarola, and M. Antavolits, editorsHuman-Computer Interaction: Tasks and Organization Proc. 6th European Conference on Cognitive Ergonomicspages 167–180, 1992.

    Google Scholar 

  34. T.R.G. Green and M. Petre. Usability analysis of visual programming environments: A cognitive dimensions framework.Journal of Visual Languages and Computing7(2):131–174, 1996.

    Article  Google Scholar 

  35. T.R.G. Green, M. Petre, and R.K.E. Bellamy. Comprehensibility of visual and textual programs: A test of superlativism against the match-mismatch conjecture. In J. Koenemann-Belliveau, T.G. Mo-her, and S.P. Robertson, editorsProc. Fourth Workshop on Empirical Studies of Programmers.Ablex Publishers, Greenwich, 1992.

    Google Scholar 

  36. M. Gross. The fat pencil, the cocktail napkin, and the slide library. In A. Harfmann and M. Fraser, editorsProc. ACADIA 94pages 103–113, 1994.

    Google Scholar 

  37. C.A. Gurr. On the isomorphism, or lack of it, of representations. Inthis volume.

    Google Scholar 

  38. V. Haarslev. A fully formalized theory for describing visual notations. Inthis volume.

    Google Scholar 

  39. E. Hammer. Logic and visual information. InStudies in Logic Language e.4 Computation.CSLI Press, Stanford University, Palo Alto, 1995.

    MATH  Google Scholar 

  40. M. Hegarty. Mental animation: Inferring motion from static displays of mechanical systems.Journal of Experimental Psychology: Learning Memory e4 Cognition18(5):1084–1102, 1992.

    Article  Google Scholar 

  41. M. Hegarty and M.A. Just. Constructing mental models of machines from text and diagrams.Journal of Memory and Language32:717742, 1993.

    Google Scholar 

  42. D. Hix and H.R. Hutson.Developing User Interfaces: Ensuring Usability Through Product E4 Process.John Wiley&Sons, Inc., New York, 1993.

    Google Scholar 

  43. R. Hübscher. Rewriting interaction. InProc. Human Factors in Computing Systems Conference (CHI’95)pages 105–106. ACM Press, New York, 1995.

    Google Scholar 

  44. R. Hübscher. Composing complex behavior from simple visual descriptions. InProc. IEEE Symposium on Visual Languagespages 88–94, IEEE Computer Society Press, Los Alamitos, 1996.

    Chapter  Google Scholar 

  45. [45] R. Hübscher. Visual constraint rules.Journal of Visual Languages and Computingforthcoming

    Google Scholar 

  46. J. Huttenlocher. Constructing spatial images: A strategy in reasoning.Psychological Review75(6):550–560, 1968.

    Article  Google Scholar 

  47. S.H. Joseph and T.P. Pridmore. Knowledge-directed interpretation of mechanical engineering drawings.IEEE Trans. on Pattern Analysis and Machine Intelligence14(9):928–940, 1992.

    Article  Google Scholar 

  48. K.M. Kahn and V.A. Saraswat. Complete visualizations of concurrent programs and their executions. InProc. IEEE Symposium on Visual Languagespages 7–14. IEEE Computer Society Press, Los Alamitos, 1990.

    Google Scholar 

  49. K.R. Koedinger. Emergent properties and structural constraints: Advantages of diagrammatic representations for reasoning and learning. In N.H. Narayanan, editorProc. AAAI Spring Symposium on Reasoning with Diagrammatic Representationspages 154–169, Technical Report SS-92–02, AAAI Press, Menlo Park, March 1992.

    Google Scholar 

  50. J. Larkin. Display based problem solving. In D. Klahr and K. Kotovsky, editorsComplex Information Processing.Lawrence Erlbaum Publishers, Hillsdale, 1989.

    Google Scholar 

  51. J.H. Larkin and H.A. Simon. Why a diagram is (sometimes) worth ten thousand words.Cognitive Science11:65–99, 1987.

    Article  Google Scholar 

  52. G.I. Lohse, K. Biolsi, N. Walker, and H.H. Rueler. A classification of visual representations.Communications of the ACM37(12):36–49, 1994.

    Article  Google Scholar 

  53. R.K. Lowe. Constructing a mental representation from an abstract technical diagram.Learning and Instruction3:157–179, 1993.

    Article  Google Scholar 

  54. R.K. Lowe. Selectivity in diagrams. Reading beyond the lines.Educational Psychology14:467–491, 1994.

    Article  Google Scholar 

  55. D.E. Mahling and D.L. Fisher. The cognitive engineering of visual languages. InProc. IEEE Symposium on Visual Languagespages 22–28. IEEE Computer Society Press, Los Alamitos, 1990.

    Google Scholar 

  56. K. Marriott and B. Meyer. The CCMG visual language hierarchy. Inthis volume.

    Google Scholar 

  57. K. Marriott, B. Meyer, and K. Wittenburg. A survey of visual language specification and recognition. Inthis volume.

    Google Scholar 

  58. P. McCorduck.Aaron’s Code.Freeman, San Francisco, 1991.

    Google Scholar 

  59. T. Menzies. Frameworks for assessing visual languages. Technical Report TR 95–35, Department of Software Development, Monash University, Australia, 1995.

    Google Scholar 

  60. B. Meyer. Pictures depicting pictures: On the specification of visual languages by visual grammars. InProc. IEEE Symposium on Visual Languagespages 41–47. IEEE Computer Society Press, Los Alamitos, 1992.

    Google Scholar 

  61. B. Myers. Visual programming, programming by example and program visualization: A taxonomy. InProc. Human Factors in Computing Systems Conference (CHI’86)pages 59–66. ACM Press, New York, 1986.

    Google Scholar 

  62. N.H. Narayanan, editor.Proc. AAAI Spring Symposium on Reasoning with Diagrammatic RepresentationsTechnical Report SS-92–02, AAAI Press, Menlo Park, March 1992.

    Google Scholar 

  63. N.H. Narayanan and M. Hegarty. On designing comprehensible interactive hypermedia manuals.International Journal of Human Computer Studiesforthcoming.

    Google Scholar 

  64. N.H. Narayanan, M. Suwa, and H. Motoda. How things appear to work: Predicting behaviors from device diagrams InProc. 12th National Conference on Artificial Intelligencepages 1161–1167. AAAI Press, Menlo Park, 1994.

    Google Scholar 

  65. N.H. Narayanan, M. Suwa, and H. Motoda. A study of diagrammatic reasoning from verbal and gestural protocols. InProc. 16th Annual Conference of the Cognitive Science Societypages 652–657. Lawrence Erlbaum Publishers, Hillsdale, 1994.

    Google Scholar 

  66. N.H. Narayanan, M. Suwa, and H. Motoda. Behavior hypothesis from schematic diagrams. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 501–534. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.

    Google Scholar 

  67. N.H. Narayanan, M. Suwa, and H. Motoda. Diagram-based problem solving: The case of an impossible problem. InProc. 17th Annual Conference of the Cognitive Science Societypages 206–211. Lawrence Erlbaum Publishers, Hillsdale, 1995.

    Google Scholar 

  68. A. Newell and H.A. Simon.Human Problem Solving.Prentice-Hall, Englewood Cliffs, 1972.

    Google Scholar 

  69. W.M. Newman and M.G. Lamming.Interactive System Design.Addison-Wesley, Wokingham, 1995.

    Google Scholar 

  70. J.V. Nickerson. Visual programming. Limits of graphic representation. InProc. IEEE Symposium on Visual Languagespages 178–179. IEEE Computer Society Press, Los Alamitos, 1994.

    Chapter  Google Scholar 

  71. D.A. Norman. Cognitive engineering. In D.A. Norman and S.W. Draper, editorsUser Centered System Designpages 31–65. Lawrence Erlbaum Publishers, Hillsdale, 1986.

    Google Scholar 

  72. D.A. Norman.The Psychology of Everyday Things.Basic Books, New York, 1988.

    Google Scholar 

  73. G.S. Novak and W.C. Bulko. Diagrams and text as computer input.Journal of Visual Languages and Computing4:161–175, 1993.

    Article  Google Scholar 

  74. M. Petre. Why looking isn’t always seeing: Readership skills and graphical programmingCommunications of the ACM38:33–44, 1995.

    Article  Google Scholar 

  75. M. Petre, A.F. Blackwell, and T.R.G. Green. Cognitive questions in software visualization. In J. Stasko, J. Domingue, B. Price, and M. Brown, editorsSoftware Visualization: Programming as a Multi-Media Experience.MIT Press, Cambridge, 1997.

    Google Scholar 

  76. B.A. Price, R.M. Baecker, and I.S. Small. A principled taxonomy of software visualization.Journal of Visual Languages and Computing4(3):211–266, 1993.

    Article  Google Scholar 

  77. J. Puigsegur, J. Agusti, and D. Robertson. A visual programming language. InProc. IEEE Symposium on Visual Languagespages 214–215. IEEE Computer Society Press, Los Alamitos, 1996.

    Chapter  Google Scholar 

  78. D.R. Raymond. Characterizing visual languages. InProc. IEEE Symposium on Visual Languagespages 176–182. IEEE Computer Society Press, Los Alamitos, 1991.

    Google Scholar 

  79. A. Repenning. Bending the rules: Steps toward semantically enriched graphical rewrite rules. InProc. IEEE Symposium on Visual Languagespages 226–233. IEEE Computer Society Press, Los Alamitos, 1995.

    Google Scholar 

  80. A. Repenning and T. Sumner. Agentsheets: A medium for creating domain-oriented visual languages.IEEE Computer28:17–25, 1995.

    Article  Google Scholar 

  81. M. Resnick. Beyond the centralized mindset.Journal of the Learning Sciences5(1):1–22, 1996.

    Article  Google Scholar 

  82. G.G. Robertson, S.K. Card, and J.D. Mackinlay. Information visualization using 3D interactive animation.Communications of the ACM36(4):57–71, 1993.

    Article  Google Scholar 

  83. F. Saint-Martin.Semiotics of Visual Language.Indiana University Press, Bloomington, 1990.

    Google Scholar 

  84. D.L. Schwartz and J.B. Black. Analog imagery in mental model reasoning: Depictive models.Cognitive Psychology30:254–219, 1996.

    Article  Google Scholar 

  85. T. Selker and L. Koved. Elements of visual language. InProc. IEEE Symposium on Visual Languagespages 38–44. IEEE Computer Society Press, Los Alamitos, 1988.

    Google Scholar 

  86. R.N. Shepard and L.A. Cooper.Mental Images and Their Transformations.MIT Press, Cambridge, 1986.

    Google Scholar 

  87. S.-J. Shin.The Logical Status of Diagrams.Cambridge University Press, Cambridge, 1994.

    MATH  Google Scholar 

  88. N.C. Shu. Visual programming languages: A perspective and a dimensional analysis. In S.-K. Chang, T. Ichikawa, and P.A. Ligomenides, editorsVisual Languagespages 11–34. Plenum Publishing Corporation, New York, 1986.

    Chapter  Google Scholar 

  89. A. Sinha and I. Vessey. Cognitive fit in recursion and iteration: An empirical study.IEEE Trans. on Software EngineeringSE-10(5):386–379, 1992.

    Google Scholar 

  90. D.C. Smith, A. Cypher, and J. Spohrer. KidSim: Programming agents without a programming language.Communications of the ACM37:54–68, 1994.

    Article  Google Scholar 

  91. R.B. Smith. The alternate reality kit: An animated environment for creating interactive simulations. InProc. IEEE Symposium on Visual Languagespages 99–106. IEEE Computer Society Press, Los Alamitos, 1986.

    Google Scholar 

  92. S. Steinman and K. Carver.Visual programming with Prograph CPX.Manning Publications/Prentice Hall, Englewood Cliffs, 1995.

    Google Scholar 

  93. K. Stenning, R. Cox, and J. Oberlander. Contrasting the cognitive effects of graphical and sentential logic teaching. Reasoning, representation and individual differences.Language and Cognitive Processes10(3/4):333–354, 1995.

    Article  Google Scholar 

  94. K. Stenning and J. Oberlander. A cognitive theory of graphical and linguistic reasoning: Logic and implementation.Cognitive Science19:97–140, 1995.

    Article  Google Scholar 

  95. S. Tessler, Y. Iwasaki, and K. Law. Qualitative structural analysis using diagrammatic reasoning. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 711–730. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.

    Google Scholar 

  96. E.R. Tufte.The Visual Display of Quantitative Information.Graphics Press, Cheshire, 1983.

    Google Scholar 

  97. E.R. Tufte.Envisioning Information.Graphics Press, Cheshire, 1990.

    Google Scholar 

  98. E.R. Tufte.Visual Explanations.Graphics Press, Cheshire, 1997.

    MATH  Google Scholar 

  99. B. Tversky. Cognitive origins of graphic productions. In F.T. Marchese, editorUnderstanding Images: Finding Meaning in Digital Imagerypages 29–53. Springer-Verlag, New York, 1995.

    Google Scholar 

  100. D. Wang, J.R. Lee, and H. Zeevat. Reasoning with diagrammatic representations. In J. Glasgow, N.H. Narayanan, and B. Chandrasekaran, editorsDiagrammatic Reasoning: Cognitive and Computational Perspectivespages 339–396. AAAI Press, Menlo Park and MIT Press, Cambridge, 1995.

    Google Scholar 

  101. D. Wang and H. Zeevat. A syntax directed approach to picture semantics. Inthis volume.

    Google Scholar 

  102. K. Wittenburg and L. Weitzman. Relational grammars: Theory and practice in a visual language interface for process modeling. Inthis volume.

    Google Scholar 

Download references

Authors
  1. N. Hari Narayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roland Hübscher
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, Monash University, Clayton, Victoria, 3168, Australia

    Kim Marriott

  2. Lehr und Forschungseinrichtung für Programmierung und Softwaretechnik Institut für Informatik, Ludwig Maximillians Universität, Oettingenstrasse 67, Munich, 80538, Germany

    Bernd Meyer

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer Science+Business Media New York

About this chapter

Cite this chapter

Narayanan, N.H., Hübscher, R. (1998). Visual Language Theory: Towards a Human-Computer Interaction Perspective. In: Marriott, K., Meyer, B. (eds) Visual Language Theory. Springer, New York, NY. https://doi.org/10.1007/978-1-4612-1676-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-1-4612-1676-6_3

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4612-7240-3

  • Online ISBN: 978-1-4612-1676-6

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation