Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

French sign language is the first means of communication of the French deaf community, representing around 120,000 people in France. A law passed in 2005 recognised it as a full language [1], which was the starting point for broader recognition from the public and its use in the public school system and in administration. Signed languages are analog, visual-gestural and multilinear (meaning that they allow the simultaneous transmission of several pieces of information) languages. Thus, they are distinct from vocal languages which are arbitrary, acoustic-vocal and monolinear. Up to now, due to this complexity, no satisfactory writing system has been created for sign language, and yet written sign language would offer deaf people the conditions for an unprecedented cultural enrichment.

Signed languages cannot be written with existing writing symbols, as they do not have the same roots and modality as their vocal counterparts. Studies have shown how a harmonious development of conceptualisation relies on a sign language-based education [2]. A writing system needs to be engineered to fit specific characteristics, including grammar, vocabulary and multilinearity among others. However, most endeavors in this direction have resulted in graphic codes for linguists [3, 4] rather than a practical writing system that could be used every day by the deaf community.

We aim at contributing to the creation of a handwriting system for sign language through a graphic approach. We base our work on a visualisation of movements through extensive techniques of investigation and experimentation. Our working hypothesis is that gestural signs produce readable graphic structures comparable to the characters used in writing [5]. We believe that in this particular case, the language and its writing could have much more in common than do vocal languages and their scripts. We have chosen to focus on gesture as a whole and not on one of the various parameters of sign language, so that our study departs from the traditional dissection of sign language [6, 7].

In this chapter we present photocalligraphy, a method aimed at exploring the graphic structures of movement in sign language, which relies on photography using long exposure times. Through the creation of the two corpora and their evaluation by the deaf community we noted that the sign language speakers modified their gestures to make their photographic representation clearer, thus adopting a writing-like behavior. After studying those modifications, their repetitions and patterns, we structured the consistent ones that we call graphic inscribing strategies.

We will begin by clarifying the context of our research: sign language and its current representation and capture, and then writing and movement visualisation. In the second section we will present our main contribution: our photocalligraphic tool and the resulting representations of sign language. We will then present both our corpora, the first one representing legibility (as defined in [8]) while the second deals with the sign modifications made by the speaker in order to increase this legibility. Before concluding we will focus on a particular concept that emerged through both corpora: graphic inscribing strategies.

Contextualisation and State of the Art

Sign language. It is important to keep in mind two major factors before investigating a writing system for French sign language. One is the means of communication. The visuo-gestural mode used in sign language differs from the voco-acoustic one in spoken language. Sign language uses gestures to produce signs, and vision to perceive them. The entire upper body acts as a medium. Taking only manual gestures, we have hand shapes, movements, orientation (of the palm) and location. The capacity to use several parts of the hand at the same time brings us to the second factor: multi-linearity. Sign languages can express a lot of information simultaneously, as opposed to spoken languages which are monolinear, making visualisation of sign language a challenge. Up until now none of the existing transcription systems has the qualities necessary to translate efficiently into written sign language.

Representing Sign Language. In order to keep a written record of French sign language, annotation systems such as HamNoSys [3] or SignWriting [4] have been created, the latter more visual than the first. Neither of these highly schematic systems can properly represent the richness of sign languages. They use more or less arbitrary representations to depict the sign, with a strong tendency towards geometrisation, making them closer to a notation than to a writing system. Most of the existing graphic transcription systems have been devised as scientific tools. The purpose is often to serve as an annotation system [9]. Those systems are successful mostly when used in a narrow, specialised context, and confirm the view that traditional linear writing is not suited for sign language [10].

For that reason, we needed to research a transcription system able to cover various levels of complexity that are communicated simultaneously. We focused on a graphic approach (as opposed to verbal description and subdivision) that takes into account multi-linear communication. By claiming that french sign language, in its gestural dimension, represents graphic structures comparable to writing, our research attempts to go beyond the traditional parametric organisation of sign language described by Stokoe [7, 11] and in France with Cuxac [12].

Captation and corpora. The existing corpora mostly rely on video recording of sign language and on motion capture. Their focus on sign language ranges from its vocabulary or its emergence (Creagest Project: [13]) to its grammatical structure [14], or even a comparison between signs or structures from different countries [15] (MARQSPAT Project [16]). The specific form of the corpus determines the corpus itself and the type of analysis it will allow [17].

Our method of capture differs from those used in existing corpora [13, 18] by moving away from annotating purpose to concentrate on the visualisation of movements. Inspired by cognitive science’s enaction paradigm [19], we also wonder about the feedback such a representation can make to the user. We focus especially on the trace the hands leave, and draw an analogy with the stroke of a pen which is regarded as the foundation for formalised writing according to G. Noordzij [20].

Writing. Latin languages are broken into phonemes and their writing systems are separated into small discrete sets of glyphs, like atoms, evolving from the pictograms of proto-sinaitic writing to more complex shapes [21] and to eventually represent sounds or, as we might say, shapes of sounds, pure conventional forms. Unfortunately, such simplicity and economy cannot apply to sign language, as sign language is closer to a continuous mode of communication, defined by its four manual parameters.

The Chinese writing system, on the other hand, appears to be closer to sign language with its phono-semantic compound nature. It developed, while maintaining consistency, by using simple rules of construction and graphic semantics. In Chinese calligraphy, the supple brush is meant to sense every modulation of the body and transfer the movement freely, allowing an infinity of variations in the stroke. This connection between the body, the tool and the sign gives the gesture authority over the sign and not the contrary [22].

Both models help us to reflect on the principle of the stroke of the pen (or brush) and the role of the body in the writing process or performance. It is the formal semantic and structural features of the Chinese writing system that came to be the main source of inspiration in our research, as it associates different graphic modes with the movements of the hand. Far from the conventional alphabetic forms of the glyphs in the transcription of vocal languages, there is the potential to write sign language as we speak it, with the same tools: the hands and the eyes.

Visualisation of Movement . By picturing various factors in movement simultaneously in legible photography, the work of the physiologist E. J. Marey is a milestone in many ways [23]. Next to Eadweard J. Muybridge’s sensational chrono-photography, Marey gave birth to the modern, scientific observation of the body in movement.

At a formal level, the work of Anton Giulio Bragaglia [24] crystallised our reflections. By using long exposure and welcoming the blur that results from fast movement, his technique, photodynamism, rejected Marey’s analytical methods and focused on capturing the sensation of movement rather than breaking it apart. More than a sequence, this visualisation depicted movement as an indivisible reality and form.

Unlike Picasso and Mili in 1949 [25], the process of photocalligraphy is not painting with light. The movement itself is clearly the origin of the traces in our process, extending the concept of the stroke by capturing the various dimensions of the hands.

Aside from this photocalligraphic representation, we acknowledge the various digital methods of rendering movement, and their advantage (easier manipulation, modification and prototyping). While in this work we focus on analog capture, with the blur of movement characterising our renderings, we are also exploring the digital dimension of movement: its various representations and the manipulation of such a graphic digital object.

Photocalligraphic Capture

We devised a photographic process allowing us to visualise hand movement in space specifically in the context of sign language. We call this technique of visualisation photocalligraphy, referring to the formalism of our images. The photocalligraphy method focuses on isolated signs and gestures and turns them into a graphic imprint. This way of representing the gestural dimension of sign language envisions the hand as a graphic tool, similar to a living brush.

In order to isolate the hands and the face, the sign language speaker wears a black garment with long sleeves and stands in front of a uniform black background. A camera frames the upper body plus the space necessary to perform large signs, thus capturing the meaningful signing space and defining our frame of reference. We use a digital camera (Nikon Dsign languageR D90) to shoot long exposure images (duration around 2 s., depending on the sign), in order to capture the entire duration of the sign. This records the continuous trace of the hands in movement without the need for post-editing. Avoiding post-editing enables the sign language speaker to see instantly the graphic potential of their movements. It proved in our experience and in feedback from the sign language speakers that the system was very close to a process of writing. An exposure of 2 s was found to be the optimal value. Because the speaker cannot hear the noise made by the camera shot, we indicated it by opening and closing a hand (start and end of the exposure). This is important, to synchronise the duration of the shot and to convey the time available to perform the sign. Sessions are recorded on video so as to save discussion and keep track of the evolution of the sign language speakers’ behaviour.

In our set-up, a screen faces the sign language speaker and displays the image just taken. This visual feedback enables the sign language speaker to see what they have produced, bringing the experience even closer to writing. This process also instigates the exploratory aspect of our work, as the sign language speaker often reacts to his creation and tries to give the next gesture a particular scriptural direction. Without being an actual writing tool, photocalligraphy using this particular set-up demonstrates the concept that French sign language gestuality includes a scriptural dimension.

Visualising Sign Language Gestuality

We focused our research on movement, the most graphically dense parameter of sign language. Yet, we have realised that the object of our study is not this parameter alone. Indeed, changes in configurations and orientation both have an impact on the rendering (see Fig. 12.1), and the positions of the hands over the body are implied in the location of the movement. Facial expression can also be represented if recorded.

Fig. 12.1
figure 1

Photocalligraphies of the signs [TO SUCCEED] and [CLEVER]

Full size image

In the end, what we capture focuses more on gesture as a whole than on movement as defined in the context of sign language. The resulting object is a picture, a projection of the 4D (space, time and depth) space of sign language on the 2D space of photography. Being 2D, we had to compensate for the loss of two other dimensions (time and depth). Long exposure was intended to compensate for time while the freedom given to sign language speakers to choose how to face the camera partially compensated for depth. Yet, in our case, dynamics and depth are still the main issue in sign language representation. In the next sections we will see how the sign language speakers used our system to overcome these limitations.

We don’t deny the major part exploration plays in our work and in the sign language speakers’ experience of our set-up. While spoken language and writing traditionally use completely different modes (voco-accoustic and gestuo-visual), in sign language there is the theoretical potential to use a common channel for writing and speech. Eventually, such an experimental approach pushes the boundary of how we define writing and puts the writer in the situation of recalling his gestural language as the act of writing: an analogical graphic transcription of an oral sign. We aim to show the impact of associating this graphic inscription with a sign from the sign language vocabulary in a future publication.

Signer/Writer Dilemma. The graphic dimension that we perceive in oral speech justifies the dual nature of our two corpora: collecting images that record the execution of a gesture or oral communication, and a scriptural performance. Such a procedure confronts the sign language speaker, who is an expert in their language, with a situation where they have to develop a critical sense of their scripting capability. We name this double ability: signer/writer. The dilemma for the signer/writer is to inscribe a mark that will respect the natural shape of the sign, and yet also result in the greatest legibility in the final picture (Fig. 12.2).

Fig. 12.2
figure 2

Photocalligraphy of the sign [Abstract]

Full size image

Corpora and Evaluations

For our corpora we chose a representative sample set of 100 signs in French sign language, aiming to represent the different forms in the vocabulary of French sign language [26]. Signs were selected based on their graphic parameters (dimensions, dynamic, symmetries, rotation and shapes) as well as gesture parameters (one or two hands, mouth movements, repetitions, change in hand configuration, contact with the body, position, spreading of the movement). Once we chose our set of signs, we made a list using illustrations from a dictionary that is considered as a reference in French sign language: the IVT (International Visual Theater) dictionary [27]. During sessions with sign language speakers, we presented them with the pictures from the IVT dictionary as reference to avoid influencing them with our concepts of the signs.

We produced two photographic corpora. Using the first one we were able to study a broad range of photocalligraphies and then test their legibility with sign language speakers. The second corpus focused on the variation and alterations in photocalligraphies among sign language speakers. In both cases we worked with native speakers of sign language. In this way we were able to test both whether our research direction was meaningful and whether it was acceptable to the sign language community. Both corpora will be available in the near future.

First Corpus: Angle and Legibility

For this corpus, we set up 12 viewpoints spread over a 150° arc in order to photograph the sign language speakers (a man and a woman for this corpus) from different angles. This was to capture the dimension of depth and to explore there was an optimal angle, defined by each sign’s various parameters. As it was our first large scale experiment, the main objective for the first corpus was to test the legibility of the graphic records. Despite some limitations in this first set-up, we already found that the sign language speakers were intensely involved, ascribing great importance to their realisations and making good use of the visual feedback to improve them.

Legibility. The next step was to test the ability of those graphic records to convey the original meaning of the sign. For that, we conducted an online evaluation that we describe in [5]. Eighty sign language speakers of various levels of skill participated, resulting in an average 63 % comprehension. By comprehension we mean that the subjects were able to recognise the sign depicted in a photocalligraphic. These results confirmed our research direction but above all exposed the progress yet to be made.

Angle. In the end, the angle proved to be a non-variable. Rotation did help legibility by giving the photocalligraphies a feel of 3D when viewed successively as a short animation. No definite rule appeared for an optimal angle other than simply that which the sign language speaker would have chosen by instinct.

Variations. We saw a huge difference in legibility between the two sign language speakers in some signs, as can be seen for instance with the sign [chain] in the Fig. 12.3, with 100 % recognition for the realisation on the left and 60 % for the one on the right. As our system does not instantly create an instant visual representation of sign language for every sign, the sign language speakers themselves took to distorting some signs, making them different from the prototype but improving their graphic representation.

Fig. 12.3
figure 3

Two different realisations of the sign [Chain]

Full size image

Those variations implied the existence of rules for improving the legibility of our photocalligraphies; our understanding was that these same rules might apply in a possible gesture based sign language writing system. This new direction prompted us to devise a second corpus aimed specifically at the study of these alterations.

Second Corpus: Alterations

In order to study variations in the performance of signs, we worked this time with eight native sign language speakers, men and women: some, from deaf families, had learnt french sign language since they were born while others had learnt it in high school or even when they reached adulthood. For this corpus the setup (Fig. 12.4) was simpler. We only captured the image from one angle, trusting the sign language speaker to choose the best angle. As in the first set-up, visual feedback was given and the sign language speaker could create a different version of a sign if they wished, by modifying the angle, speed or dynamic. Our aim here was not so much to achieve the best graphic imprint but to study the processes themselves and their evolution.

Fig. 12.4
figure 4

The setup for the second corpus

Full size image

Each speaker performed 25 signs out of the whole sample set. The resulting corpus comprises a series of 200 images covering the 100 signs, and over 25 h of annotated video.

A session took place as follows:

  • Explanation of the project, presentation of the different working steps;

  • First capture of the 25 signs in video as a reference;

  • Experimentation with the photocalligraphic set-up;

  • Second capture of the same 25 signs with the photocalligraphic set-up;

  • Selection of the best pictures for each sign taken during the session;

  • Discussion of the working session.

The protocol was organised to allow the sign language speaker to master the set-up with minimum intervention from us. We did not express any subjective judgment on the quality of the images produced, even when asked to by the sign language speaker. When they were uncertain, we advised them to think of what they would want to see in the image. Then, we could assist with technical advice on how to realise their vision. When we felt that the speaker had developed a particular process of modification for the photocalligraphy, we asked them to describe it.

At the end of the session, the pictures were displayed again and we discussed with them the question of legibility and the potential offered by the set-up. We also watched another series of pictures produced by a different sign language speaker and ask the subject to identify the meaning, to pick out the most legible ones sign by sign, and to explain their choice.

We noticed that similar strategies (observed earlier) were used spontaneously by most participants without any direction from us. This would imply that these techniques are a generic response to the writing/performance process rather than arising from the individual alone.

Alterations used for a specific visual purpose were identified and an underlying structure emerged. Because of that structure and the recurrence of these purposeful modifications of the signs, we decided to call those alterations strategies of graphic inscription. By this, we mean all the techniques of production of the sign used in order to make its graphical representation more legible and closer to the mental visualisation the person has of the sign.

Graphic Inscribing Strategies

Sign language speakers can see the result of their trials and variations in the visual feedback. If a variation is considered effective it is integrated, and can reveal a specific process. When this process proves itself to be common to a variety of signs and speakers, it becomes a graphic inscribing strategy as seen in Fig. 12.5. Those strategies are connected to the set of parameters that those various signs share, implying the existence of possible generic rules.

Fig. 12.5
figure 5

Evolution in the realisation of the sign [Forest]

Full size image

As sign language speakers build up an understanding of the set-up and skill in using it, they are able to improve their production by recalling acquired strategies, showing that a learning process has occurred. The first underlying structure we found in the inscribing strategies related to the two missing dimensions of the projection: time and depth.

Time Related Strategies

With an exposure longer than 1/4 of a second, a moving object produces a motion blur. The stiller an object remains, the sharper and brighter it will appear. In contrast, movement will make it blurry and under exposed. This gives the speaker freedom to shape the dynamic of the photocalligraphy by accentuating different parts of the sign.

One of such strategy is to break down some of the sign into key positions. Those key positions are either a strong variation in direction or a modification of the hand’s configuration. The emphasis of key parts eases the analysis of the sign as a whole as it sharpens the most revealing components.

Some strategies were used to define the flow of the sign: where it begins, where it ends. This is in fact a piece of information our photocalligraphies do not record, and feedback from our first corpus indicated that its absence reduces legibility and makes it harder to recognise the sign. Most sign language speakers dealt with this issue by making the end configuration of a sign brighter in order to hint at a direction of the movement. Finally, when there was any kind of repetitive motion in a sign, sign language speakers usually chose to remove it in order to avoid graphical overlays of hands or movement trail.

Space Related Strategies

By default, the sign language speaker puts themself in front of the camera during the shot. This promotes a face-to-face position similar to the natural communication stance in sign language. In the case of movements on the axis of the camera, the loss of information related to depth impairs the legibility of the sign. A line becomes a dot and the entire movement is flattened into a blurry form. Here, the sign language speaker can choose to turn slightly sideways in order to present the movement from a better angle.

Moreover, some movements are too slight, and this creates overlays. In this case the movement can be exaggerated to reduce overlays.

The speaker can rotate not only the body at the beginning of the sign (thus impacting the whole sign) but also the hands during the sign. This way, they can choose the best angle for their current specific hand configuration, to maximise legibility and recognition while not altering the trail too much.

Conclusion

In this chapter we presented our photocalligraphic set-up as well as the graphic inscription strategies that emerged from both our corpora. We hope to offer a valid approach to creating a script that takes graphic design into account. We feel that this is a multidisciplinary field of study where the importance of exploratory graphic design is under-represented.

The list of strategies we have are only those that arose from our sessions. The next step will be to broaden the field and search for more of those strategies, which will help us to better understand the structure of this first set. We will also carefully associate these strategies with the parameters of the sign with which they were used in our sample set. Then we will test for generalisation by searching signs with similar parameters in our sample set, check whether the strategies are applicable to those, and observe their effects.

Perspectives . The next logical step will be to measure the impact of those strategies on legibility. Once we have assembled enough of them, we will again evaluate them and compare signs with and without the graphic inscribing strategies. This will also be the occasion to make this evaluation using higher resolution images. Because of the set-up, the quality of pictures from the first evaluation was low. We hope that this improvement in quality, together with the use of graphic inscribing strategies, will have a positive impact on legibility.

The photocalligraphic inscription system shares certain characteristics with writing tools, hopefully implying that the rules developed for one medium will also apply to the other. We are interested in learning from the strategies developed through our visualisation technique and applying this knowledge to a writing system. The strategies would be translated into rules of composition, harmony, balance, etc.

We will have to overcome some problems if we want to propose relevant answers to the challenge of graphic visualisation of french sign language. For example, how do we choose which are the meaningful parameters in a spacially performed gestural sign in order to translate it into a graphic sign that is static and flat? We can also question the status of those images. Are they a representation of the gesture, or a representation of the language? How do they affect the cognitive model of language? It also begs the question of articulation between signs. This would imply taking into account the segmentation and grammar of the language itself.