From Evaluation to Crits and Conversation

Abstract

Soon after the launch of the iPhone the British artist and printmaker David Hockney began sending his friends pictures he had made using painting and drawing apps. One of these friends was the writer and art critic Martin Gayford and in 2009 he received an iPhone drawing of a sunrise over the East Yorkshire town of Bridlington (Thames & Hudson 2007). The file on Hockney’s iPhone was identical to the one that he sent to Gayford which prompted him to ask—where was the original? Hockney sent many images to his friends and they in turn could share them until London was awash with “original Hockneys”. While Hockney’s limited edition etchings and lithographs attract high prices, this new method of production and dissemination challenged the notion of a limited edition “print run” because digital files can be reproduced with no diminution of quality at virtually no cost. These kinds of reflection led to a project called “Digital Originals” where we conducted several studies of practicing artists and developed an app called Repentir.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Soon after the launch of the iPhone the British artist and printmaker David Hockney began sending his friends pictures he had made using painting and drawing apps. One of these friends was the writer and art critic Martin Gayford, he received an iPhone drawing of a sunrise over the East Yorkshire town of Bridlington (Gayford 2007). The file on Hockney’s iPhone was identical to the one that he sent to Gayford which prompted him to ask—where was the original? Hockney sent many images to his friends and they in turn could share them until London was awash with “original Hockneys”. While Hockney‘s limited edition etchings and lithographs attract high prices, this new method of production and dissemination challenged the notion of a limited edition “print run” because digital files can be reproduced with no diminution of quality at virtually no cost. These kinds of reflection led to a project called “Digital Originals” where we conducted several studies of practicing artists and developed an app called Repentir. The app was given to a variety of people in technology shows, art shows, and private viewings. Responses were mixed but this chapter argues that a straightforward evaluation of such an app would not tell us very much. Computing technologies are now a part of almost every aspect of human activity, evaluation methods which developed when computers were confined to the workplace are no longer enough. This chapter reflects the move from usability evaluation to arts based techniques like polyphonic evaluation, “crits” and finally conversation.

2 The Varieties of Evaluation

When HCI was focused on the workplace evaluation was relatively straightforward. Metrics like ease of use, ease of learning, time on task and accuracy were observable, measurable and comparable. All nice and scientific. But as computing technology seeped from the workplace into the home and then our pockets the goals of the technology became less clear. Workplace computing is almost by definition task and problem focused and the goal of design is to discover optimal solutions to clearly defined problems. Increasingly computing technology is being applied to “wicked” problems where complex vested interests struggle over limited resources and there is no technological quick fix. When technology is conceived as a “silver bullet” it can result in “solutionism” a phrase coined by Dobbins (2009) and popularized by Morozov (2013) as quick fixes for complex psychological, social or environmental problems. It can also take the form of solutions to problems that do not really exist.

In HBO’s satirical comedy Silicon Valley the protagonists develop “SeeFood” a “Shazam for food”, where you take a picture of what you’re about to eat and it tells you what it is. The show exaggerates current trends in Silicon Valley and academia but it does not exaggerate much and SeeFood is a perfect example of solutionism. Clearly there would be many usability issues with an app like SeeFood, would it be accurate? How long would it take? Would it be easy to use? But these questions would not answer larger questions like—is it stupid?

Kaye and Sengers (2007) provide a historical overview of the five phases of evaluation in HCI: evaluation by (1) engineers, (2) computer scientists, (3) cognitive psychologists, (4) HCI professionals and (5) evaluation for experience. There is now a body of work that has adopted theory and methods drawn from the humanities, cultural studies and critical theory to address, aesthetics, user experience and politics e.g. (Bardzell 2009, 2010; Blythe and Robinson 2008; Blythe et al. 2006, 2008, 2010; Boehner et al. 2005). Despite the apparent plethora of methods available, evaluation remains problematic (Lindsay and Jackson 2012). Kusunoki and Sarcevic (2012) found that much evaluation work still draws on traditions such as usability testing, heuristic evaluation and cognitive walkthrough. Vermeen et al.’s statistical overview of UX methods (Vermeeren et al. 2010) reveal many methods to be minor adaptations on standard usability tools which do not account for UX factors. Alternatives to standard evaluation metrics include: considering the ways that participants articulate their values [i.e. “Value Centered Design” (Friedman 2006)]; operationalizing values to provide a focus for evaluation (Cockton 2008) and; having designers articulate a particular use quality, so that they can evaluate against it (Greenberg and Buxton 2008). These approaches specify values as the basis for success or otherwise, irrespective of the nature of the values being evaluated.

“SeeFood” might be evaluated in terms of the values of the team developing it. Silicon Valley regularly mocks the rhetoric of the tech industry by having all of their start ups claim that they are “making the world a better place by…” doing whatever it is that they happen to be doing. So the values behind SeeFood could be portrayed as idealistic and utopian. At another moment in the series one character explain that their business model is not really about making apps at all but rather selling stock. Making money might then be a criteria for evaluation and this would rest on popularity. But other approaches take more critical views.

Interdisciplinary criticism calls for a mix of evaluation methods including critical analysis drawing on traditions from the Arts and Humanities (Blythe et al. 2006). Bill Gaver describes “polyphonic evaluation” which draws on multiple perspectives and voices, to argue that conflicting responses can be valuable (Gaver 2007). Bardzell's (2009) “interaction criticism” proposed a mapping of critical theory and aesthetics for interaction design. Greenberg and Buxton (2008) considered usability evaluation as harmful. They argued that it is always possible to find some metric by which a new design might be judged superior to another one. And focusing on usability allows researchers to disregard usefulness. They advocated art school traditions like the “crit” where experienced and new designers respond to work in progressive stages of iteration. The approach suggests that prototypes might be considered as sketches. The drift of all of these approaches is away from binary judgments towards nuanced discussion.

3 Mixed Reviews

Repentir is a mobile app that allows users to take photographs of an oil painting and then rub the image to reveal previous versions of the work right back to the pencil drawings and the blank canvas beneath (Hook et al. 2013). The app was developed with the British artist Nathan Walsh, who took photographs of his urban realist oil painting, portraying the street in New York “23 Skidoo” (Fig. 1).

Fig. 1

The Associate Director of the gallery demonstrating the app with 23 Skidoo in New York

Repentir was presented at Walsh’s solo exhibition at a prestigious New York gallery. The gallery’s Associate Director, who negotiated sales, said gallery visitors reacted with “wows”, and thought the app was amazing. One of the gallery’s co-owners was less impressed; he said that this kind of thing had been done in the sixties when artists documented works-in-progress and re-presented them in sequence, on slides or film.

There was another show featuring large scale copies of the work at a small gallery in the English city of York where prints of 23 Skidoo were put on sale to raise funds for a charity that supports new artists (Fig. 2). The app worked just as well with the large-scale facsimiles as the original oil painting. Again, the response was overwhelmingly positive; visitors reacted with “wows” and enthused that the app was “magical”.

Fig. 2

Interactive prints exhibition in York

The gallery owner, declared in a webzine video that the app “revolutionized the way you look at work, it undresses the painting”. After the show, he estimated that the number of prints sold was triple what he would have expected to sell normally.

Another Walsh painting TransAmerica was shown with the Repentir app at a CHI conference in Paris. Hiroshi Ishii, a pioneer of Tangible User Interfaces approached the second author and gently informed him that his group had done something similar a few years back, making X-ray imagery of hidden traces under the surface of a painting.

The artist’s friends and collectors also viewed Repentir in Walsh’s studio. While the fellow realist artist David Finnigan was delighted with the app, collectors were less impressed, spending little time with it; one said it added little to the painting. While 23 Skidoo was still in progress, Martin Gayford visited Walsh’s studio. Gayford is best known for his work on David Hockney and for his book describing the experience of sitting for the painter Lucien Freud. Gayford reflections on Hockney's iPhone drawings had inspired the project and the researchers were thrilled at the serendipitous opportunity to meet him. For Gayford the app was “interesting” but, as will be discussed in later sections, his responses were nuanced and not easily reducible to approval or disapproval metrics.

Reviews of the Repentir prototype were then mixed. As a group of researchers endeavoring to understand Repentir and how its design and reception might inform the practices of others, what were we to make of this?

4 Process

Over eighteen months the research team met up regularly with Walsh. Initial visits involved biographical interviews and observations of the painter’s practice. Walsh’s work is sometimes described as “photo-realist” although he prefers “urban realist”. Walsh uses photographs for reference but he does not use projection or tracing techniques, a point that he has emphasized in interviews:

People have said … ‘OK, that’s a big photograph … or - it’s somehow been constructed on a computer. There’s a digital print or some sort of underhand process there… It’s just hard work and graft … It’s a process and there’s been a big progression over the last 5 or 10 years.

Walsh works six days a week from a small windowless studio.

During initial visits he was working on the painting in Fig. 3. The first author observed that early stages of the work produced beautiful effects, sometimes lost during the painting process leading him to suggest experimenting with forms of documentation. For example, a “slow print” might play the painting back in real time over the time it took to make the painting (Briggs and Blythe 2012). Walsh was enthusiastic: “I like the idea of shifting things away from a print that just duplicates what we’ve already got … [or] a poorer version of the actual painting.” Following discussion a camera was set up in the artist’s studio and Walsh would take a photograph of his work at the end of each working day.

Fig. 3

Nathan Walsh work in progress and painting

5 Developing Repentir

The slow print idea was developed further with the second author suggesting an app: “where you click on an area … and it shows you images where painting had been happening” (Jon). Walsh was again enthusiastic suggesting the name “Pentimento, which is under-drawing”. As this was trade marked he later suggested “Repentir”, another art history term indicating evidence of painters’ corrective work (Fig. 4).

Fig. 4

Repentir TransAmerica

Walsh proceeded to document his next painting “TransAmerica” in eighty images used to develop an app. Users of Repentir take a photograph of an area of Walsh’s painting which they can then explore in detail, by rubbing or scrolling to expose the under-workings.

6 Comment and Crits

Initial responses to Repentir were for the most part positive. The camera icon was self-explanatory, the screen featured a slider, which most users dragged to reveal previous stages; while the rub function seldom required explanation. Walsh was very enthusiastic about the app as he relayed to artist Finnigan: “I thought it was great fun … I like the rub function because it just seems…you’re touching something and you’re somehow closer to the process of making the picture”. Finnigan, was also positive: “It’s an aspect of seeing a picture visually that, it’s quite unique.” Mel, one of the New York gallery co-owners was likewise enthusiastic, and reported positive responses from visitors: “My client the other day, his immediate reaction was, ‘Wow, look at all the detail.’” When asked about the general reaction in New York the Associate Director, Ella relayed reactions like: ‘Wow! Oh my God! Wow! How do you do that?’ She suggested, “for a museum, this is amazing, it’s a great educational tool”. Martin Gayford was also positive if still guarded in his response: “I think looking underneath the surface it works well, it’s quite an interesting proposition.” Overall then, there were positive reactions from artistic, commercial and critical perspectives.

There were however negative comments about both Walsh’s work and the app. As previously noted, Leon the other co-owner of the gallery was not impressed saying that this kind of thing had been done before: “Raphaella Spence, about ten years ago … the publicity people for Chrysler put a camera in her studio and every five minutes…it snapped a picture. And they made that into a film.” He noted that artists often sent him photographs of works in progress which he used to help generate sales.

Although Walsh’s collectors did not entirely dismiss the app they were not impressed and did not engage with it for long: “It’s a good idea but it doesn’t work as well as I would expect it to,” said one. These collectors were responding to an early iteratino of the app that involved waiting for image downloads, and reduced image quality due to the painting being so large—a serious barrier to engagement. Artist Finnigan, while positive, also complained about image quality: “I keep taking blurred pictures.” Ease of use was impaired by the long processing time and image quality though these were addressed before the New York show. These problems would perhaps be the main focus of a traditional HCI evaluation. But for us they are amongst the least interesting findings with more general discussions more valuable than binary judgments.

As prices of Walsh’s work had risen in line with moves to more prestigious galleries, collectors of his paintings now operated at a level of wealth that Walsh found hard to comprehend, something with which he was not entirely comfortable: “You think we’re playthings for the jet set? It’s kind of true, isn’t it?” For Walsh, part of the appeal of Repentir was making his work accessible without simply making photographic prints. The fact that the app would work on a postcard as well as the original surprised Mel, the gallery co-owner who joked, “You just fucked up my whole marketing campaign!” But for Walsh the app made the work “inclusive”.

The art market for Walsh’s work is small and specialised. The exposure of labor was crucial for both the galleryists and the artist. Walsh’s process is “very labor intensive. To even make four paintings a year, it requires that level of commitment”. For Walsh and Mel (the gallery co-owner) the app provided value as “evidence of the effort”, or as fellow-artist Finnigan put it, the “veracity of the effort”. For the artists the quantity of labor was directly related to quality, prices and marketing. In response to some dismissive comments which were posted on the Huffington Post board Walsh posted: “I make paintings based on photographic information …. What I don’t do is project, Photoshop, grid, trace, or stitch together the information I gather.” Walsh does not decry other methods but the painstaking nature of his craft is an important part of the narrative that informs its value.

Gallery co-owner Mel, suggested that the app was also of practical value: “I don’t know how people think these paintings get done… You know it’s so complex… So it really, really helped justify the prices!” Marx noted in Capital: “the correctness of the law discovered by Hegel… that merely quantitative differences beyond a certain point pass into qualitative changes” (Marx 1906). Quality increases as more time is spent on a product; ultimately then quality is a quantitative value. While it is no accident that the art which artists produce is known as their “work”, the notion of work is disrupted when technology is, or is suspected to be involved. Digital technology is met with suspicion in this space because, as Gayford points out, it “fundamentally disrupts the art world”. As in the recording industry, previous models of monetization through reproduction have been upturned and new mechanisms must be sought.

Leon, the gallery co-owner made it clear to the first author that he didn’t have long for an interview. The first author then decided to risk offence by asking about money and what kinds of people visited the gallery.

• Leon: That’s a bad question, everybody in the world comes through and some people buy. Now there’s not a lot of this to go round, there’s people that respect disciplines, imagery…

• Mark: The people that are buying must be fairly well off though?

• Leon: Yes, of course. But not nearly as well off as the people that spend $50 million dollars on a Basquiat.

Leon claimed that serious collectors did not care how an image was made or if it would increase in value:

The only time anybody buys a work in this gallery is because they like what it looks like, they can afford it, and they’re going to buy it. If I get any indication that it’s investment or the decorator told them to have it, they can’t buy from me.

Although Leon stressed that art collectors were ordinary and the act of collection nothing to do with money, both Walsh and Finnigan noted that they were producing art for the very wealthy. This made the artists uncomfortable, and also the researcher. It might be possible to construct an argument for the app’s economic value from sales, but we would argue that the app was of greater value as a prop for reflection.

7 Conversation

When Gayford visited Walsh’s studio he played with the app for a few minutes but remained to engage in a wide ranging discussion for around 2 hours. Much of this discussion concerned the use of digital technology in the production of art. Both Walsh and Gayford insisted there was nothing wrong with using technological aids in painting. Gayford pointed out that if Vermeer did use a camera obscura this makes his achievement no less astonishing. And yet, anxiety about technological intervention was threaded throughout the discussions. Walsh: “It’s just something that I battle against, I have to kind of say, ‘Yes, you could project a photograph [but] I’m trying to create a … completely different space which is independent of the photography”. However, both Walsh and Finnigan in another conversation named another realist painter who they strongly suspected of printing photographs onto canvases then painting on top of them to make them look like paintings. One buyer had asked this artist for “work in progress” shots and, when the request was refused, had asked for their money back. Much of the anxiety around technology then concerned deception. Any technique was acceptable so long as it was not “underhand”. The suspicion of technology was ultimately a question of legitimacy.

Gayford noted that it is the universal aspects of digital images that people don’t like: “It doesn’t have that difference in texture …. An oil painting is rather good in the digital age because it is still completely impossible to reproduce the actual object, so you’ve still got unique value in one thing”. For Gayford the value of seeing an original work is practical: there is no better cadmium blue than cadmium blue paint: “you can’t print that color…. So nothing that you see on a screen or a piece of paper is going to correspond to the cadmium blue which is the paint on a canvas”. The notion of an original in this sense is material. However high resolution a screen may become there will always be limitations to digital reproduction and meanwhile value in the tangible original (Fig. 5).

Fig. 5

Martin Gayford in Walsh’s studio

Materiality was also important in the printmaking Gayford had observed. He described Lucien Freud: “looking at them [prints] very hard, he was saying, ‘No, there’s a tiny mark there […] What he said to me was buyers of artists’ prints are really, really fussy people.” This demand for material perfection perhaps indicates that the immateriality of digital media fundamentally limits its appeal to collectors. Walsh was deeply concerned with creating nuanced texture across the painting surface—with some areas much smoother than others: “Within a photograph, there’s no hierarchy of different marks, it’s just one sort of surface”. Mark-making was part of what defined the painting as beyond photographic reproduction. While Repentir presented a flat image on glass, the action, of rubbing through the layers, echoing the gestures of the artist, perhaps added richer experience and texture than if encountering a static reproduction, despite the surface of the device always being uniform.

Many discussions around the app were concerned with other potential uses. At the CHI conference some delegates thought the designed system too simple and pointed out that AR offered far greater functionality, suggesting access to interviews, preparatory drawings and so on. As Walsh noted “Techs want it to be all singing, all dancing.” One CHI attendee suggested that it might be used with Old Masters where preparatory sketches were in existence. Gayford suggested “digital conservation, where… you could look at what’s underneath the varnish…”. There is much controversy over art restoration and many claim that some damages the work. Until recently the colors in Michelangelo’s Sistine chapel were much darker and while some claim this is how the artist would have conceived it, others argue it is the invention of restorers. While rubbing down to the canvas with the Repentir app Gayford joked: “this is what art restorers claim they don’t do”. A more serious point was that alternative versions of color within an app could show what different types of restoration or varnish removal might look like.

But Gayford also related the app to wider art historical and philosophical concerns: “I think you’ve hit on an interesting area. I think this history of pictures is actually deeply involved with time because until the invention of movies, which is pretty recent, the picture, from the moment of drawing bison on cave walls, is a way of freezing time. The image you got was a sort of compact version of reality in which you could dwell on things. You looked at a frozen image for a long time, instead of a constantly changing world. And you have new techniques and you can play around with it more.” Gayford pointed out that the way an image was displayed drastically affected the amount of time it was possible to look at it. Before reproduction, a painting in a gallery could be looked at for no more than a few hours. But crowded scenes by painters like Breugel repay multiple views, before reproduction this was only possible in private collections. At the same time, he argued, paintings compressed time: the repeated observations of an artist over hundreds of hours are compressed into the single glance of a viewer. In a sense, Repentir allowed an unpacking of those repeated, hidden observations, returning time to the moment frozen in the final image.

Gayford made a comparison to Picasso’s performance paintings, made by painting on glass in front of a film camera: “the drawing at the end isn’t particularly interesting, but Picasso quickly worked out that it’s the way you got there which is kind of interesting.” The value in exposing process with Repentir was partly commercial, in that it exposed labor and skill, but for Gayford there was also artistic value in that sometimes process can be more interesting than the final static image.

8 So What?

Responses to Repentir were mixed. For the artist and his community of practice it was a very interesting development, valuable in demonstrating their often-disputed work practices. For the New York galleryists the app was primarily of utilitarian value in demonstrating the labor and time involved to justify high prices. For technology developers, like Ishii, this instantiation of AR was nothing new. And while the app helped to justify prices and make sales, within an art market in service of the “one percent” this was an anathema to the politics and values of both the artists and researchers. The app tripled sales of reproductions in the York gallery, and raised money for the charity, but this was a miniscule fraction of what the original painting sold for. Meanwhile for the critic Gayford, the app was an interesting work in progress.

What then constitutes success or a failure? From the gallery’s point of view we might argue Repentir was a successful sales gimmick. From the collector’s point of view it added little; perhaps detracting from the magic of the final image. From the technologist’s point of view we might note that the idea itself and the technological implementation are not very original. From the critic’s perspective we might claim it was a successful resource for historical and critical reflection and discussion.

9 Parallax Views

As Research through Design becomes more common practitioners have begun to ask fundamental questions about its epistemology. Zimmerman and Forlizzi argue for the importance of rigor in research design and methodology (Zimmerman and Forlizzi 2008). Gaver argues for openness and points out that designs are not repeatable in the way that scientific findings are (Gaver 2012). The reasons for making prototypes are also debatable. Academic researchers are not engaged in market research or developing prototypes and evaluating responses to judge whether mass production would be a good idea or not. Both Gaver and Zimmerman and Forlizzi agree that the point of a research prototype is not market potential but rather contribution to knowledge.

Though most of us would concede that we are not going to solve the world’s problems with an app our discourse is firmly based in binary judgements around success or failure. This is rarely the most interesting aspect of Research through Design. Repentir was a success for some and not for others. So what? The reason that such an argument is unsatisfactory is perhaps because evaluation itself is an inappropriate perspective. Over the last ten years the funding criteria for university research emphasizes “impact” stated explicitly as a contribution to the economy, society or culture. Repentir arguably supported sales in the existing art market, so it can be judged a success. But many artists and critics find the existing art market morally reprehensible; Repentir supported this so it is a failure. In this sense Repentir was a success and a failure for the same reasons.

For the philosopher and cultural critic Slavoj Žižek there is no contradiction in the act of interpreting an artifact in opposing or paradoxical ways. Žižek frequently draws on the notion of the Parallax view. Parallax refers to the phenomena where objects appear differently depending on the perspective from which they are viewed. Although we cannot see ourselves looking, our gaze is an integral part of every observation. Žižek illustrates the point with reference to the Moebius strip and the curved space that bends in on itself:

We do not have two perspectives, we have a perspective and what eludes it, and the other perspective fills in this void of what we could not see from the first perspective. [27:29]

Repentir then can be seen as a success and a failure at the same time. The reality is not found in the agreement between two perspectives. Rather one perspective supplies that which the other cannot. This is not then postmodern relativism, where all perspectives are equally valid, rather it is an insistence on the reflexive inclusion of the observer in any observation. Evaluation strategies that seek simple answers for success and failure metrics are inappropriate for Research through Design.

Attempts to widen HCI’s approach to evaluation have encouraged multiple and competing accounts that have shifted the emphasis from binary judgments to nuanced assessment and analysis. But it should be noted that it is not just HCI that has a problem with evaluation. As G. K. Chesterton remarked of the modernist project itself, the position can be characterized as: “Let us not decide what is good, but let it be considered good not to decide it” (Chesterton 1928). He extends this logic in this succinct formulation: “let us not settle what is good, but let us settle whether we are getting more of it” (ibid.).

It is not enough to ask if we did what set out to do, we must also ask if we should have even tried it in the first place.