Beginning to Multiply (with) Dynamic Digits: Fingers as Physical–Digital Hybrids

Abstract

The development of touchscreen technology is providing alternative ways for learners to conceptualise, visualise, experiment with and communicate about mathematical ideas and relationships. While the multi-touch affordances of touchscreens enable children to produce and transform ‘screen objects’ with their fingers (by means of varied forms of pressure and propulsion), they also invoke an intricate interrelationship between the user’s fingers and the surface of the device itself. Drawing on a half-hour video recording of two primary school children using the TouchTimes iPad app (about multiplication) for the first time, we examine how mutually interactive the children’s fingers were, both with each other and with this particular touchscreen technology, not least their combining in ways which challenge the seemingly clear distinction between digital and physical tools, when viewed as discrete and disjoint entities. We also explore fingers being used as objects in themselves, while examining ways of doing multiplication digitally with fingers, with a particular focus on the singular role of fingers as physical intermediaries. Our aim is to consider possible ways to develop well-educated fingers in relation to engaging with mathematics.

Put your Finger into every Bottle, to feel whether it be full, which is the surest Way, for feeling hath no fellow. (Swift 1745, p. 32)

Until modern times, apart from the esoteric knowledge of the priests, philosophers, and astronomers, the greater part of human thought and imagination flowed through the hands. (Mumford 1967, p. 238)

the computer’s smooth screen forms a rock face: what holds are there to grab onto? (Serres 1999/2011, pp. 11–12)

Every contact leaves a trace. (Locard’s exchange principle)

The advent of touchscreen devices and their ease of use has brought about opportunities that can be of benefit to young children in particular, who are able to interact with this technology in more intuitive, user-friendly ways through simple tactile actions such as tapping, swiping, pressing, swirling, gliding and pinching. The increasing availability of iPads in elementary classrooms is providing access to new resources and new means which show promise in supporting mathematics learning (see, for instance, Sedaghatjou and Campbell 2017). Consequently, there is also a growing body of research in mathematics education related to understanding how the affordances of touchscreen devices, such as iPads, can facilitate such learning and how, more specifically, this type of technology affects engagement with counting, numbers and arithmetic (Sinclair and Pimm 2015a; Baccaglini-Frank 2018).

While we also have a number of video recordings of individuals working on the TouchTimes multiplication app (about which, see below for more detail), we have opted here to focus on a pair of young children working ‘together’, in part so as to have access to a more intricate involvement of hands and the feeling in/of the fingers, in order to explore to what extent it is possible to feel multiplication. One of our central foci has to do with what hands and, in particular, fingers do, especially when more than one hand is involved (as opposed, for example, to engaging with a computer via a mouse¹).We are particularly interested in the fluid, rhythmic actions that arose, not least involving repetition, as well as which elements/features of the app the pair attended to initially, without guidance, and which ones were attended to subsequently. And, additionally, there were some unexpected aspects – of rhythm, music and dance – that arose.

The notion of chirality,² of handedness, and its potential connection to asymmetric actions in this manual mathematical setting, is something we will employ. And, lastly, there is a potential connection in this work to Donald Winnicott’s (1953, 1971) notion of a ‘third area’, the place of ‘transitional objects’, as inner and outer reality (related to the distinction between ‘me’ and ‘not-me’) both continue to emerge and develop. In the future, we plan to explore to what extent fingers in this context can be productively seen as mathematically transitional objects.³

Fingering Things out

“Don’t do it! I’m just fingering it out!” (said a four-year-old girl – see Phillips 1996, p. 82).

Considerable attention has been given to the role of finger usage in the development of number sense in children. Emerging neuroscientific research suggests that there is a functional and beneficial relationship between early numerical representations involving and invoking fingers and the development of later mental representations of number (e.g. Domahs et al. 2008). And, in their synthesis of neurocognitive and mathematics education research, Solyu et al. (2018) suggest that, “features of numerical cognition may be grounded in the finger sensorimotor system” (p. 127), in ways that extend into adulthood.

Santi and Baccaglini-Frank (2015) summarise that, “[microworlds] activate the use of gestures, natural language, symbolic language, touching, tapping, dragging, artifacts, etc. that enhance the student’s sensory motor experience in space and time” (pp. 226–227). Research findings highlighting the importance of finger-based interactions for numerical development have also influenced the development of new learning technologies. Specific examples include TouchCounts (TC)⁵ (Jackiw and Sinclair 2014) and TouchTimes (TT) (Jackiw and Sinclair 2019), its recent expansion which we will engage with in detail below. Both of these are educational applications that make use of multi-touch technology affordances, in order to provide alternative ways for children to encounter, investigate, process and engage with certain mathematical ideas. In both of these interactive apps, children can summon and seemingly engage ‘directly’ through finger touch with ‘objects’ that are made to appear on an iPad screen, and, by using their fingers and dynamic actions, it is possible to interact with mathematical phenomena in an open-ended and exploratory way.

The design of TouchCounts is “intended to offer an expressive environment in which learners could create and relate mathematical objects directly with their fingers and hands” (Sinclair and Heyd-Metzuyanim 2014, p. 84). Research related to the use of TC in particular has shown that there may well be a strong connection between the way that children use their fingers and the way they come to think about numbers (Sinclair and Sedaghatjou 2013), that the linked interaction between fingers and eyes – between the tangible and the visible – enables new ways of thinking about number (Sinclair and de Freitas 2014) and that the development of ‘gestural subitising’ may be assisted through the task of placing several or all available fingers simultaneously on the screen (Sinclair and Pimm 2015b). Tangibility was found to be of particular significance in Sinclair and Pimm’s examination of the mutually interactive visible, audible and tangible effects of TC on young children coming to terms with counting and early arithmetic. However, the input actions used by the children were noted to be “different from those [gestures] discussed in the mathematics education literature in two ways: they involve contact with a screen and they perform an action” (Sinclair and Pimm 2014, p. 210). For a little more on this, see the sub-section after next.

Calder and Campbell (2016) have pointed out that, “the visual and dynamic elements of engaging mathematical thinking through digital technologies reposition both the types of knowledge and understanding required and the ways in which learning emerges, which simultaneously shape the learning experience in a range of interrelated ways” (p. 50). After using TC in a classroom-based intervention with first-grade students, Ferrara and Savioli (2018) described seeing the children’s “mathematical doing not only in terms of creatively touching numbers, but also in terms of imagining and feeling quantities” (p. 243). This is an interesting blend of finger use, mathematics and technology that can also include aesthetic and affective elements.

TouchTimes

TouchTimes (TT), a multi-touch iPad application, is designed to improve children’s flexible and relational understanding of multiplication.⁶ Currently, it has two sub-applications (termed ‘Grasplify’ and ‘Zaplify’), of which the former invokes a distinct role for each hand (related to multiplicand and multiplier) in multiplication, while the latter emphasises the commonality of both hands, irrespectively of order (related to commutativity). Given that both sub-applications involve direct use of fingers on the screen, we conceive of TT’s primary operation as multaplying.

In this article, we focus exclusively on the sub-application Grasplify, which, when first opened, displays an oblong screen divided in half by a vertical line (Fig. 1a). Whichever side of the iPad screen is touched first by the user’s fingers (or knuckles), a different-coloured disc (termed a ‘pip’) appears beneath each finger (as if summoned to the surface from below) and the numeral corresponding to the number of visible pips held on the screen at that instant is displayed at the top of the screen on the same side as the pip-creating fingers (Fig. 1b; left side of the screen touched first, so the numeral appears on top left). This numeral adjusts instantly when fingers are added or removed from the pip side, whether temporally in sequence or simultaneously. In an intriguing way, it is as if the pips are counting the contact fingers, because it is the iPad that ‘counts’.

Fig. 1

a Initial screen of TT; b creating pips; c creating pods; d finished expression

In order to preserve each pip (that will otherwise vanish), the corresponding fingers must maintain continuous screen contact, acting as an anchor. So, for pips initially to appear and then to continue to exist, the user’s fingers must linger on the screen. The individual pip-creating fingers, however, can slide around on the screen surface while maintaining screen contact, and the associated pips will move with them. If the first contact is on the left side (LS) of the vertical dividing line,⁷ then the pips will initially appear on the left (as in Fig. 1b). However, TT is designed to be symmetric in this respect, and pips can be created on the right side (RS) of the line provided the first placement of finger(s) is to the right of the vertical line (and the multiplicand will then appear on the top right and the numerical expression will go from right to left).

When a user subsequently taps a fingertip (or -tips) from her or his second hand on the other side of the line, enclosed bundles of pips (called ‘pods’) appear beneath each contact finger of that latter hand (Fig. 1c). Here, each finger beckons a bundle – an instance of the one summoning the many. The number of pips is the multiplicand. The number of pods is the multiplier, and each pod contains a duplicate of the pip configuration (matching both the relative screen locations of the pips created by the first-hand placement and their corresponding colours). So, in general, pods are pied.

When pods show up, a second numeral also appears, separated from the first by the multiplication sign (‘×’) which sits above the vertical line. However, there is a short time delay before the third numeral, representing the product appears. And, as mentioned above, note that when the pips are created on the right-hand side of the screen, the numerical expression appears and develops from right to left (e.g. ‘12 = 3 × 4’). Significantly, unlike the pips and the pods, the mathematical symbols that appear on the screen cannot be ‘manipulated’ directly – but they can most definitely be indirectly changed.

Unlike with pip-fingers, when a contact pod-finger is removed from above its pod, that pod nonetheless remains on the screen, but becomes slightly smaller, allowing more of them to be seen at once (Fig. 1d). This is a non-trivial asymmetry between pips and pods (and is the first instance of significant chirality, where the left side of the screen cannot be superimposed onto the right, something human chiral hands reflect). After pods are created, TT encircles all of them into a single entity, by surrounding them with a pale white background (like a faint handkerchief), and (after a small delay) displays the corresponding mathematical expression (e.g. ‘3 × 4 = 12’) at the top of the screen, with the product also marked in white.

If a pip-conjuring finger is lifted, the content of each of the pods adjusts accordingly: that is, each pod will then contain one fewer pip (the vanished pip from each pod being of the same colour as each other and the directly disappearing, original pip). The existence of all pods is maintained, provided there is continual screen contact by at least one pip-finger, but if all pip-fingers are removed from the iPad screen, both pips and pods disappear (reverting to Fig. 1a). Additionally, individual pods can be dragged to the trash (at the bottom pod-side corner of the screen) to be deleted without resetting the screen contents as a whole.

Embodying a multiplicative rather than an additive model, TT enables children to create and co-ordinate “visual images of composite unit structures in multiplicative situations” (p. 306), which Downton and Sullivan (2017) argue is the core of multiplication and fundamental to developing multiplicative reasoning directly, as “young children’s (8-year-olds) concept of multiplication is based on the meaning they give to the composite units they construct” (p. 306). Using different fingers (and hands) dynamically with TT, children can (almost simultaneously) create units of one and units of more than one, generating a multiplicative model that includes the co-ordination of two quantities, in a manner similar to Fig. 2.

Fig. 2

A multiplicative model (Boulet 1998, p. 13)

On Gesture and Action

Unlike TouchCounts, there is no audio channel in TouchTimes, so the primary axes of attention are the tactile and the visual. But, as the Serres quotation from the beginning of our article draws attention to, there is no actual variation, no differentiated texture, in the feel from different parts of the screen. But that does not mean that the tactile experience of the user is uniform and undifferentiated: on the contrary, we believe it is influenced by what the eyes see. In other words, eyes are needed for fingers to feel. The primary trace these fingers leave is simply epitheleals on the screen. But of far more interest to us – a potentially Locardian research question – is what contact trace does the screen cede to the fingers (and hence, perhaps, to the brain)?

Albeit about a different context (namely ‘how the body stands and walks’), Serres (1999/2011) also observed, “second reversal: sight touches and touch sees” (p. 25), which connects with Nemirovsky et al.’s (2013) later notion of ‘perceptuomotor integration’ and is a core phenomenon in Sinclair and de Freitas’ (2014) article on ‘tangible gestures’ (for more on this seeming oxymoron, see immediately below), exploring different ways in which the hand and the eye co-ordinate (not least by their drawing on Gilles Deleuze’s analysis of a Francis Bacon painting). The hand is thus an eyewitness.

In contrast with this, Novack and Goldin-Meadow (2017) insist on a category distinction between ‘gesture’ and ‘action’ in general, claiming, “Gesture is a special kind of action, one that represents the world rather than directly impacting the world. For example, producing a twisting gesture in the air near, but not on, a jar will not open the jar; only performing the twisting action on the jar itself will do that” (p. 653). They see gestural functions as quite distinct from what they term ‘instrumental’ actions: “Our hypothesis is that the effects gesture has on thinking and learning grow not only out of the fact that gesture is itself an action, but also out of the fact that gesture is abstracted away from action” (p. 653).⁸

A further instance, one that echoes the above terminology, comes from conducted music. With or without a baton, the conductor’s hands gesture repeatedly in the air in order to beat time, while all the instrumentalists’ hands touch (the strings, cover the holes or press the keys of the woodwinds, …). Ironically, the lowest point of the conductor’s downbeat is called the tactus (as in ‘tactile’). It means touch, but also the sense of touch. Yet, because it is a gesture, it does not actually touch anything (except the air).⁹ But the virtual intent of the conducting gesture is aspiring to touch time.¹⁰

A Touch of Method, a Further Trace of Theory

The episodes described in this article (presented in chronological order) took place in an elementary school in the Lower Mainland of British Columbia, Canada, and formed part of an exploratory conversation between an interviewer and a pair of third-grade students (who had had no previous experience either with TT or with TC). This interaction was part of an iterative design experiment aimed at refining the TT prototype and developing accompanying tasks suitable for use with students in grades two or three. That day, three pairs of children, one pair at a time, had explored the app with the interviewer in a small area separate from their classroom.

This occurred prior to the pair of girls (whom we name Jacy and Kyra) whose TT explorations (approximately thirty minutes long) constitute the empirical focus of this article. We created a transcript of the video from which we have selected short episodes to illustrate the complex interplay between the tangible actions used by the girls in engagement with TT, some of which we see as relevant to the aesthetic and affective dimensions of mathematics learning. We also noticed the potential origins of a couple of pertinent gestures from their actions. The episodes we examine were chosen because they exemplify the mutually interactive relationships among the TT app itself, the pair of users, their two sets of hands and multiple fingers, and mathematics.

The intellectual framing of this article primarily draws upon theories of embodiment and the relationship between physical movement and mathematical meaning-making (see, for instance, Radford 2009; Nemirovsky et al. 2013). Adopting a non-dualist orientation to learning and thinking, we engage with the relational, ontological perspective of inclusive materialism (de Freitas and Sinclair 2014). In this view, there is an agential relationship between the app (TouchTimes, in this case) and the user, and through tactile interaction they mutually constitute each other. In particular, the app both enables and prevents certain actions, taking on an animate role through communicative contact during interaction with the user (and, in this sense, both ‘manipulate’ one another, even though the screen does not have hands).

We attend especially to the relational entanglements among the learners, the app and the (manual) mathematics. Sensori-motor actions are not subordinate to thinking:

Thinking, hence, does not occur solely in the head but in and through language, body and tools […] gestures, as a type of bodily action, [… are] genuine constituents of thinking” (Radford 2009, p. 113; italics in original).

Given this position, alongside the contact-rich design of TT, we are interested in what physically occurs during learner engagement with TT. Therefore, we examine the complex actional and interactional relationships among a pair of students (and their fingers, in particular) and this digital device, while they engage in a tactile manner with the mathematical concept of multiplication.

Finally, in an arithmetic context, Brian Rotman (1987, p. 27) refers to the active human subject as ‘the one-who-counts’. As the one-who-counts, I count with my fingers, but my fingers can also be the ‘thing-to-be-counted’, so I can be counting both on my fingers and with my fingers. In relation to a touchscreen device, there is additionally ‘what-is-counted-on’ (counting on a screen with my fingers) as opposed to intransitive counting (which is purely verbal, the ordinal reciting of number names). With TT, as with TC, the app itself is the-one-‘who’-counts. However, with TT, the owner(s) of the hand(s)/finger(s) that create the pods is (are) the-one-who-multiplies: here, the ‘multiplier’ (as the word structure suggests) is actually a person – a person whose hand is guided by the multiplier’s eye.

Fingering More Things out

We begin by describing Jacy and Kyra’s preliminary interaction with TT on a single iPad, starting when the interviewer suggested, “Why don’t you play a little bit – just touch whatever you want”. Each girl started by tapping a single finger up and down on the screen (Jacy on the RS,¹¹ Kyra on the LS) in what appeared to be random but discrete motions (Jacy’s index finger of her left hand, Kyra’s of her right hand). Within five seconds from the outset, Jacy’s finger was pressed down continuously on the RS while she watched Kyra repeatedly tap on the LS, creating one-pods. When Kyra reached over the vertical line with her right hand (RH) to tap on the RS, she momentarily created two-pods for the first time (Fig. 3a; Jacy wearing the white top with the heart, Kyra the blue jacket with flowers), which prompted Jacy’s response, “What! How did you do that?” While trying to figure out how Kyra had created the two-pods, ten seconds later Jacy placed her thumb down on the screen, and held it there continuously, in addition to her index finger (Fig. 3b), thereby creating and maintaining the presence of two pips, which opened up new possibilities (and likely, visually, made the difference between pips and one-pods more evident, which may not have been noticed before). Whenever two-pods (or more-pods) are created, it is the first instance of multiplication in this app.

Fig. 3

a Kyra’s accidental two-pods; b Jacy’s thumb and index finger held down; c Jacy using two hands to create pips

Kyra touched the screen next, creating a two-pod and, at this point, Jacy added the index finger of her other hand to the RS and then proceeded to use fingers (plural) from both hands (Fig. 3c). Jacy’s ability to use double pip-hands on the RS was enabled by Kyra’s actions on the LS and, like ripples in a pond, each girl’s actions appeared to influence the other. Less than ten seconds after Jacy first did it, Kyra began tapping multiple fingers on the screen, as if playing keys on a piano (for more on this image, see below) and then quickly brought her other hand into action as well. At various times, they each used fingers from both hands rapidly and varyingly (which, inadvertently, altered who was creating pips and who pods). Shortly thereafter, Jacy asked Kyra, “How do you get those mini ones?” (as mentioned above, a pod shrinks when the finger above it is lifted), which marked the transition from random tapping of finger(s) on the screen to a more focused exploration of TT.

In passing, within the interplay between the girls’ fingers/hands and the iPad screen, there was noticeable shifting between forms of contact and their intensity: discrete movements that included simple tapping (Fig. 3a); a sharper, more intense poking motion (which we will later call staccato touch); alternating, repetitive rhythmic contact (Fig. 4a), which we later distinguish between trill and tremolo; sequential tapping with several fingers in turn reminiscent of playing notes on a musical keyboard (Fig. 4b). In addition, there are continuous movements at varying speeds (sometimes with several fingers at once), including pressing (Fig. 4c), pushing, pulling, tracing, swirling and smearing. Arguably, the girls’ discrete movements seem to be intended to prompt the appearance of the virtual ‘objects’ that dwell ‘beneath’ the screen, while their continuous movements seem to be motivated by a desire to relocate the configurations already visible on the screen’s surface.

Fig. 4

a Alternating, rhythmic thumb-tapping; b musical keyboard playing; c pressing and swirling

Kyra began to engage in a continuous tracing motion, seemingly in response to the two circular pips beneath her fingers, while Jacy’s statement (“Wait! Get as much as you can. Get as many as you can.”) indicates that she had started to attend to quantity with the addition of pips and/or pods onto the screen, a direct result of discrete action. At this stage, there was no indication that they were attending to the varying numerals or arithmetic expressions at the top of the screen.

Using both hands, Jacy created eight pips and then began moving them around the screen in a swirling motion when Kyra said, “I’m changing the shapes”. What she was referring to is unclear, although Kyra seemed to believe that her forms of fingering (both the rapid tapping and the ‘piano playing’) were causing the changes she had observed. However, it was actually Jacy’s pip-swirling motions that were causing the movement of the pips within the Kyra-pods.

Jacy suddenly removed both hands from the RS and, because she was creating the pips at this point, it caused the iPad screen to go blank, prompting her comment, “I made them go away!” The pair then proceeded to experiment by placing their hands on the screen and then abruptly lifting them up, causing the disappearance of all visible pips and pods. Kyra noted, “When you let go, I think it goes away”, but when she next lifted her hands from the pods, nothing happened. Jacy noted, “Yours don’t go away, but hang on […]” and abruptly removed her hands from the screen, causing everything to vanish.

This prompted Kyra to say, “Let me try this side and you try that side”. She had some emerging sense that it mattered which side of the screen was first made contact with (i.e. that the sides are asymmetric in their functionality) and she physically took hold of the iPad and turned the screen around in the hope of accessing what had previously been the RS. When she created the pips, Kyra said, “Ha-ha, now I control it”, the ‘it’ seeming to be making everything disappear by removing the pip-hand(s) from the screen. While Kyra’s hands were off the screen, Jacy placed her fingers down again, thereby becoming the pip-creator as before. When Kyra again placed and then removed her hands from the screen, nothing disappeared and she said, “Why do you always get it?” (perhaps wondering why, according to Psalm 31, “My times are in thy hand”).

Kyra went back and forth on both sides of the screen a few times, but because Jacy was still making contact rhythmically with one hand on the screen, she maintained control of summoning the pips. Kyra did not yet realise that the order of contact, first or second, with the two sides was what determined the creation and control of pips and pods, not the LS or RS itself. Although Kyra was intentionally trying to determine how to control what appeared and stayed on the screen, she had difficulty doing so because Jacy’s rhythmic hand placement and removal disrupted Kyra’s attempts.

On Chirality and Multiplication

‘Take care of the pips and the pods will take care of themselves.’¹²

There is an important pair of distinctions related to the manual aspects of engaging with TT. One involves which hand beckons what (the pips or the pods) on which side, in relation to the notional¹³ vertical dividing line which visibly separates the screen into two quasi-equal parts, while the other involves which hand touches (whichever side of) the screen first or second: in other words, left/right (spatial) and first/second (ordinal). This both reflects and even accentuates the asymmetry inherent in this multiplying: in other words, despite its commutativity, multiplication is not even-handed.

In passing, there is a significant issue that arises from this app in that the notation generated from it appears in the reverse order from the conventional, anglophone, North American interpretation of written multiplicative expressions. Because the pips need to be created first, they comprise the multiplicand; the number of contact touches on the other side reflects the multiplier and each one generates a pod beneath it. The first number is clearly a cardinal whereas the second one is a different type of number (once, twice, thrice, four-times, five-times, …), which David Fowler (1987) categorised as adverbs:

These two particular kinds of number can act on each other to give multiplication, as in Socrates’ “dis duoin” (twice two) or “treis tris” (three three-times). (p. 14)

Needless to say, it is the latter Socratic instance that TT consistently deploys. The fact that this is at odds with (arbitrary) notational conventions North American teachers are familiar with and mostly adhere to results in some significant discussions when teachers work with TT themselves (see Bakos and Sinclair 2019a).

This asymmetry is something which is unhelpfully smoothed over on the one hand by commutativity (which has to do solely with the result not the process) and on the other (in some languages) by not distinguishing in the mathematics register between ‘multiplicand’ and ‘multiplier’ (e.g. Turkish, Czech, Arabic, …). Even in English, the two numbers are sometimes simply referred to as ‘multiplicative factors’. Commutativity (like many aspects of algebra) creates the illusion of no difference and, hence, of indifference.¹⁴

The temporal extent of the contact between the pip-hand and the screen directly relates to time of existence of both pips and pods (the phrase ‘keeping in touch’ comes to mind). The pip-finger(s) must maintain continuous contact in order for pips (and for subsequent pods) to remain visible on the screen surface: when they are lifted, even briefly, things disappear the way speech does. Or, as Jan Zwicky (2018) observes, “one’s own actions […]: gestures that vanish in the air like music” (p. 50). The fingers of the second-contact hand, however, can be lifted without effect and the pods will remain on the screen, much like writing on paper (as opposed to speech). Pods have a longer self-existence – again, like written words: they are only erased when the pips that determine (and constitute) them are. But, always, the first hand of contact is also the potential eraser for the entire screen.

Both girls did not actually need to lift their hands at the same time; and yet they both almost always did – the beginning of a ritual gesture, perhaps, though there is also an actual (albeit virtual) effect of so doing. The first time it happened, Jacy said, “I made them go away!” and, almost as an echo, Kyra said, “They died away”. Later, Kyra instructed Jacy, “Now let go”. And both girls laughed when everything vanished. At this point, there were pips and pods located on both sides of the screen (Fig. 5), causing the centrality and differentiability of the iPad screen to fade. Although it is possible for one user to do this, it is easier when there are at least three hands involved.

Fig. 5

Both pips and pods on both sides of the screen

Both Jacy and Kyra, whenever they decided to reset the screen to start afresh, would lift their hands jointly and extensively, like medical personnel rapidly removing physical contact for a defibrillator to countershock a patient’s heart: “Clear!” (Fig. 6). This repeated manner of un-touching the screen was on its way to becoming a stylised – even ritualistic – gesture (see Coles and Sinclair 2019) and gently emphasises the co-variational aspect of multiplication.

Fig. 6

“Clear!”

A Dance to the Music of Times

As previously mentioned, TT has no audible elements. However, besides talking with Kyra, Jacy provided her own soundtrack related directly to what her fingers were doing, often rhythmically and precisely, at times as if the screen were an instrument she was playing: duh-duh, duh-duh, duh-duh; dah, dah, dah, dah, lah-dah-da; der, da-der, da-der, lah-dah-da; and many more, with varying pitches as well. This rhythmic vocalisation continued throughout the half-hour involvement.

Approximately five minutes into their exploration, Jacy engaged Kyra in another experiment by telling her, “Wait, just press a lot. Press a lot. Press your whole hand” (Fig. 7a). Kyra responded by pressing all her right-hand fingers simultaneously onto the LS, while Jacy created five-pods with her index finger, one after another (singing in rhythm), and the iPad screen flashed 5 × 1, 5 × 2, 5 × 3, 5 × 4, 5 × 5, 5 × 6. When Kyra abruptly removed her hand, she reset the screen which caused both girls to laugh.

Fig. 7

a “Press your whole hand”; b Jacy manipulating Kyra’s hand; c pinky and thumb up

At this point, Jacy said, “They just turned into threes”, even though the previous pods were five-pods not three-pods, using the past tense, although she was about to turn them into three-pods. She then proceeded to direct the pip-creation by physically lifting her partner’s pinky¹⁵ and thumb after Kyra had placed all five fingers on the screen rather than just the three Jacy wanted. This controlled placement and removal of her fingers seemed to confuse Kyra, who was holding three pips instead of five. Trying again, Jacy instructed, “Wait. Put those [indicating Kyra’s entire RH] and then take up your pinky and thumb”, and created a single pod on the RS, after Kyra replaced her three fingers on the LS:

Press your pinky and thumb away. Wait. Put your pinky and thumb down [while physically pressing Kyra’s pinky and thumb down (Figure 7b)] and now take them away. [Jacy laughs.] You changed it.

Jacy was treating Kyra’s fingers as objects to manipulate. As Kyra’s pip-creating pinky and thumb alternated rhythmically on and off the screen (akin to a tremolo), the pod composition held by Jacy also alternated between containing five pips and three. This instance, which involved the co-ordination of two pairs of hands, highlights the complex interplay between the girls and TT while illuminating the intertwined nature of discrete repetitive-tapping and continuous mutual holding. With each placement and removal of Kyra’s pinky and thumb, the colour of the affected pips changed, giving the appearance of the pod swinging back and forth (Fig. 7c). It was when Jacy created a second pod, that Kyra pointed with her free hand at the pods and, laughing, declared, “They’re dancing!” While swirling the dancing pods around the screen, Jacy continued to sing.

It was during this brief episode that Jacy began to experiment with how the appearance and disappearance of pips affected the relative configuration and colours of the pips in the pods. The motion and changing colours of the pips within these ‘dancing’ pods seemed to draw the pair’s attention to the relation between the pip-creating and pod-creating touches, which Bakos and Sinclair (2019b) have deemed significant in terms of multiplicative thinking. Although the appearance and disappearance of pips on the LS was a direct action, the corresponding effects on the pips within the pod bundles on the other side of the screen involved action at a distance.

While Kyra continued to alternate her pinky and thumb rhythmically on and off the LS, Jacy created and arranged additional pods. When Jacy began tapping repetitively on the LS, Kyra stopped tapping, leaving her hand momentarily motionless. Jacy said (of the fictional ‘dancer’), “It looks like he’s switching his legs”. The screen images below (Fig. 8) show what we believe the image was that prompted Jacy’s comment. When she tapped with her index finger on the RS, the bottom right pip appeared and disappeared rhythmically within each of the pods, in a motion reminiscent of childhood flip books.¹⁶ The pip configuration shape within the pods could be viewed as a stickman and the single flashing pip gave the appearance of movement, hence Jacy’s comment about ‘legs’.

Fig. 8

“It looks like he’s switching his legs”

Shortly thereafter, Jacy exclaimed, “Wait! Let me do this one now”, as she physically removed Kyra’s hand from the iPad to take control of the pip-creation on the LS. While Jacy rhythmically tapped her pinky and thumb on and off the screen to her personalised soundtrack, Kyra created a flurry of pods (Fig. 9a) on the RS, alternately using her left index finger and right middle finger in sharp, poking motions. Jacy urged Kyra to, “Make more. Make a lot of dancing lines” (Fig. 9b). The motion of the dancing lines seemed to draw Jacy’s attention to the shape of the pods.

Fig. 9

a A flurry of pods; b “Make a lot of dancing lines”

Lastly, an area we have not focused much attention on here is the energy and enthusiasm Jacy and Kyra presented throughout this session: the strong interactive talk, the singing/playing/ dancing with their fingers, the rhythmic speed with which they interacted with the TT app. As Nicholas Brady put it:

With Rapture of delight [thou] dost see

Thy Favourite Art

Make up a Part

Of infinite Felicity.¹⁷

For these two girls, at times, multiplication via TT clearly became their rapturous Favourite Art.

On Sonic Motion

There are different ways of touching the elements of the TT app and the ways of touching do matter. ‘Dancing’, in particular, involves a particular form of motion (see also Chorney et al. 2019), one that is interesting not only for how it expresses multiplication as a one-to-many relation (one touch changes many pods), but also for the particular kind of gesture involved. Arzarello et al. (2002) have offered a taxonomy of ways/functions of/for ‘dragging’ a mouse with Cabri-géomètre.¹⁸ We are certainly not yet in a position to do so here with respect to forms of hand or finger movements when engaging with TT. However, we have found that the terminology used for keyboard-playing may well provide a helpful, distinguishing set of terms for the finger actions involved (if not the functions): in particular, legato versus staccato touch (smooth succession of pressing keys without overlap or gaps versus rapid and distinctive individual impact/contact) and trill or tremolo (for pairs or more of fingers rapidly going back and forth on the same keys, whether adjacent or further apart).¹⁹

Though ‘touchscreen’ is a particularly accurate term, the iPad screen in TT is not a mirror (nor is it a window). Unlike touching a mirror (itself an inherently chiral act), physical interactions with touchscreen apps like TouchCounts and TouchTimes have corresponding visible reactions displayed (almost) simultaneously on the screen. We found that asymmetric hand actions were employed by the girls when engaging with TT, which accentuated the asymmetric nature of multiplication. In addition, the absence of an aural axis from the app freed Jacy up to create a soundtrack, that both reflected and was reflected in her fingering, and both reflected and was reflective of the rhythm provided by the touch of her fingers.

For almost twenty years (1966–1984), Joseph Cooper presented a BBC TV classical music quiz programme entitled Face the Music. One of the regular challenges that was part of this programme involved ‘the dummy keyboard’, where a pianist played but no sound emerged, and the contestants were to try and identify the piece of music being performed. Cooper said: ‘You hear the rattle it makes, and see my movements, but you don’t hear any music”.²⁰ Likewise, with TT there is no sound (other than that of physical tapping the screen). So Jacy sang to provide her own soundtrack, treating the screen as if it were a dummy keyboard while producing the music herself (see Fig. 10).

Fig. 10

Playing TouchTimes with both hands

Fingering Yet More Things out

Seven minutes into the girls’ exploration, the app crashed, allowing the interviewer an opportunity to engage the girls more explicitly with certain multiplication tasks using TT: “Can I ask you a few things to do?” In this section, we detail two of these tasks, the first of which involves singling out the multiplier to see its effect, while the latter explores different possibilities (changing either the multiplicand or the multiplier, or both) to create a pre-proposed product.

Counting by Fours and Fives

Kyra was asked by the interviewer to “put four fingers down”. Which she did, after clarifying that she could do so on either side, she placed four fingers simultaneously on the screen to create four pips at once, while Jacy subsequently created a single pod without prompting (Fig. 11a). When the interviewer pointed to the RS and asked, “What do you see over here?”, Jacy replied, “Oh! Four times one equals four”. The interviewer’s intent had been to draw the girls’ attention to the number and colours of the pips in the four-pod, but instead Jacy responded with this multiplicative reading of the screen (reflected in the arithmetic expression at the top).

Fig. 11

a One four-pod; b several five-pods

Then, hoping to elicit the unitising of five pips into a single pod, the interviewer asked what they could do so that Jacy would be making a five-pod on the RS. Kyra immediately added her thumb to the four pip-creating fingers she already had placed on the LS. However, Jacy did not acknowledge that there were now five pips in her single pod, as she was focused on the placement of her thumb on the screen, which created a second five-pod beneath it, overlapping the first one. She separated them by pushing the pod away with her thumb, which she then used iteratively to conjure up additional five-pods (Fig. 11b) at places within reach, while continuing to ‘hold down’ (albeit unnecessarily) the first pod with her right index finger.

The interviewer asked, “Are you making a five now?” While still adding more pods, Jacy absently observed the numerical expression and said, “Fifty-five, sixty, …”, and then, with excitement, exclaimed “Wait! It’s counting up by fives! Sixty-five, seventy, seventy-five, eighty, eighty-five, ninety, ninety-five, …”, each number said in conjunction with creating the next pod and watching the expression change. This instance illuminates the back-and-forth nature of Jacy’s action to create additional pods, and the impact of her action on the mathematical expression displayed on the iPad screen. It was these changes that Jacy noticed, leading to her realisation that the product (affected by the number of pods), was increasing by five each time.

Making Twelve

In the following episode, which occurred after some seventeen minutes, the girls were asked how ‘4’, in the 2 × 2 = 4 on display at the top of the screen (Fig. 12a), could be turned into ‘12’. There were two phenomena of interest that occurred: (a) the first involves making a number a noun, a noun with an indefinite article; (b) the second arose from an instance of Kyra counting by means of touching some of her fingers with other fingers, but what she was actually counting by this means was unclear.

Fig. 12

a Pointing to the product; b fingers counting on fingers; c “four, eight, twelve”

(a) Jacy placed four RH fingers sequentially (thumb, index, middle, ring) on the LS, while using her LH index finger on the RS to create a single four-pod, and stated, “I created a four”. Kyra echoed, “You created a four”. The use of the number noun (“a four”) indicates a more reified way of speaking, although it is unclear whether this ‘mathematical object’ arose from the number of fingers or of pips. Jacy then reset TT, placed four fingers simultaneously on the LS, creating four pips and subsequently each girl created a pod, with Jacy maintaining screen contact both with her four pip-creating fingers on the LS and her single pod-creating finger on the RS. “Can you just create a four?”, asked Jacy, just before Kyra added a third pod on the RS and Jacy’s knuckle made contact inadvertently with the RS screen as she held down her pod, making an additional one.

Jacy again removed her fingers and reset the app. She instructed Kyra to put four on the screen, which Kyra did by simultaneously placing three fingers from her left hand (index, middle, ring) on the screen, quickly followed by her pinky. Using her index finger on the RS, four-pods were sequentially created by Jacy, while the equations flashed across the top of the screen: 4 × 1 = 4, 4 × 2 = 8, 4 × 3 = 12, 4 × 4 = 16. (Jacy did not appear to be paying attention to these, 4 × 5 = 20, 4 × 6 = 24, 4 × 7 = 28, until she asked, “What number are we going to?”) When the interviewer responded, “Twelve”, Kyra immediately said, “Four, eight, twelve”. Not seeming to hear this, Jacy’s hand remained motionless and she appeared unsure how to proceed.

(b) The interviewer then suggested, “You can put them in the trash if you want”. Kyra could then be seen touching her pip-maintaining fingers on the LS with her RH index finger (Fig. 12b) in what appeared to be a count, four, eight, twelve (middle, ring, pinky) and then a second time (pinky, ring, middle). Meanwhile, Jacy dragged pods to the trash, while commenting on the expressions at the top of the screen, “Twenty, sixteen, and then it will be twelve”. Kyra then stated, “Yes, four times three is twelve.” Kyra again confirmed, “Yeah, look […] four, eight, twelve” while she touched and counted three of her LH fingers with her RH index finger (Fig. 12c).

In response to the description in the previous four paragraphs, enumerating the content of the pods by touching three of her four pip-fingers one by one while saying “four, eight, twelve” was both intriguing and somewhat confusing. Kyra could have ‘touched’ the pods themselves one by one as a means of counting them, but that screen contact would have generated more pods (as Jacy had done). Kyra likely wanted the screen contents to remain stable. Perhaps touching her own fingers above the screen while counting felt safer than a pseudo-touching (that could be called gesture-counting) of the screen objects that can appear or disappear. So, when the focus became the expression at the top of the screen, it appears that other physical components and variants came into play.

More generally, counting with a finger sometimes involves objects-being-counted contact (especially if an object is also moved once it has been touched/counted) and sometimes not (the objects can be pointed at, but not actually touched). Here, because each screen touch would create another pod, actual contact with the pods would change what was being counted. Therefore, instead of counting the pods, Kyra was possibly calculating the product by means of counting the fingers that were creating (and holding) the pod-contents. However, she may not have been counting her fingers (with another finger). Rather, she may have been attuned to the connection between the pips and the pods and the number of pips in each pod provided by the app.

Have We Actually Fingered Anything out Yet? A Conclusion of Sorts

In this article, we have attempted to illustrate the complexity of the interrelationship between the actions and interactions of a pair of students (each possessing a pair of hands and multiple fingers) and the iPad touchscreen application TT, all while balancing the affordances of the tangible nature of touchscreen technology with the mathematical offerings provided by this particular app.

In the analyses above, we have tried to show that digital and physical tools are not always distinct and separate entities, suggesting rather that they can combine in ways that are of a more hybrid nature. Although fingers are objects in themselves, we have shown how fingers also became physical intermediaries among the girls, the iPad and multiplication. The app itself was occasionally used by the girls as if it were a manipulative for play or for use in achieving their goals. All of which highlights the intricate idea of touch which involves the user(s) fingers, the touchscreen application and aspects of the mathematics. We are, however, still open to further discussion on, and subsequent exploration of, whether or not ‘timesing’ – multiplying – itself can be touched.

An iPad is clearly a physical object. One can pick it up and move it around (as Kyra did). And so are human bodies: both students, at times, moved themselves around the iPad (both their hands and their full bodies) whilst engaging with TT. But is the TouchTimes app itself ‘physical’ in any sense? Does it make sense to call it a ‘digital manipulative’ or is that actually a catachresis (like ‘tactile gesture’ perhaps), a combination of words that effectively destroys the essence of the noun (see Pimm 1988, for more on this)?

This is perhaps where Winnicott’s notion of ‘third area’ comes into play. On occasion, the two girls were using the app as if it were a manipulative, handling it (literally) in order to achieve some goals, as well as simply playing with it. But they could not touch the app directly, nor the arithmetic: all they could touch in relation to it was the screen (recall the Serres quotation from the very outset of this article). This third area, both mine and not-mine, potentially provides an interesting means to pass beyond the seemingly perpetual desire to distinguish concrete from abstract (and possibly tactile from digital) dichotomously within mathematics education. And this is perhaps a core reason why students seem perennially directed to stop using their hands in order to access more formal (abstract) mathematics. However, TT provides a clear instance of the significance of the interaction between the hands and the eyes on the one hand, and the hands/eyes and the device on the other, when engaging in and thinking about multiplication.