Keywords

1 Introduction

Since ancient times, the idea of robots (or “automata”) performing entertainment functions (and not just replacing human operators in laborious tasks) had led to the design and creation of many ingenious machines, autonomous robots, and manipulators [1]. The creation of “robots” with the specific task of drawing can be traced back to the end of the XVIII century, when the first drawing automatons appeared [2, 3].

However, these were isolated achievements (more would come in the XIX century), as it would be necessary to wait for the birth of modern robots in the broad sense (in the 1950s) to see the beginning of a series of projects and realizations of “artist” robots (specifically, robots that draw or paint), in the sense that we have today [4].

In recent years, thanks to advances in robotics technology, there have been many examples of robots that paint. This work will present a brief historical overview of robots that draw and paint, highlighting their functional and technical characteristics. It will also emphasize the trend in the development of these systems.

Starting with the drawing automata by Jaquet-Droz and Maillardet, we show the evolution of drawing machines in the XIX and XX centuries. Then, we examine the first painting machines, which appeared in the 1950s, as well as the first robotic drawing prototypes, which become increasingly autonomous and creative with the advent of cybernetics and computer graphics. Furthermore, we depict drawing and painting robots of the XXI century with enhanced manipulation, control, and computing capabilities. Finally, recent trends and future developments in this field are discussed, such as the introduction of collaborative robotics and artificial intelligence (AI).

2 The Drawing Automata by Jaquet-Droz and Maillardet

At the end of the XVIII century, the Swiss watchmaker Pierre Jaquet-Droz (1721–1790), his son Henri-Louis, and Jean-Frédéric Leschot created three automatons: the musician, the draughtsman, and the writer [5]. These automatons are still in use and can be seen at Neuchâtel’s Musée d’Art et d’Histoire in Switzerland. They were created as advertisement and entertainment toys to increase watch sales among Europe’s nobility. In particular, the draughtsman (Fig. 1a) is a small child who can draw four different images: a portrait of Louis XV, a royal pair (believed to be Marie Antoinette and Louis XVI), a dog, and a scenario of Cupid driving a chariot carried by a butterfly. The draughtsman works using a cam system that codes hand movements in two dimensions, plus one to elevate the pencil. The automaton also moves in his chair and blows dust off the pencil on a regular basis.

The most complex of the three automata is the writer. This machine can write any custom text up to 40 letters long using a system identical to the one used by the draughtsman for each letter. The text is coded using a wheel that selects characters one by one. The writer uses a goose feather inked from time to time, including a wrist shake to prevent ink from spilling.

Another Swiss watchmaker, Henri Maillardet (1745–1830), was a master of automatons who worked in London [6]. He designed an automaton who wrote in French and English, with cursive and ornate typography, drew a boat with a candle, a Chinese pagoda, and cupid (Fig. 1b). It is believed that Maillardet’s automaton has the largest “memory” of any such machine ever constructed: four drawings and three poems (two in French and one in English). The memory is stored in the cams, which are brass disks. Three steel fingers follow the uneven edges of the cams as the clockwork motor turns them. The cam motions are translated into side-to-side, front-to-back, and up-and-down movements of the doll’s writing hand via a complicated system of levers and rods that make the markings on paper. Maillardet’s automation toured England as a performer at fairs and trade exhibits, and it is currently part of the collections at The Franklin Institute in Philadelphia.

Fig. 1.
figure 1

Drawing automata: (a) the draughtsman by Jaquet-Droz; (b) Maillardet’s automaton; (c) John Nevil Maskelyne’s Zoe automaton; (d) the mechanical artist by Phillip Vielmetter; (e) replica of Gakutensoku by Makoto Nishimura.

Full size image

3 Drawing Automata in the XIX and Early XX Centuries

Further examples of drawing automata can be found in the XIX and early XX centuries. John Nevil Maskelyne’s “Zoe” automaton can be considered as a prototype of an early anthropomorphic mechanical drawing machine working as a master-slave manipulator [7]. The automaton was built in 1877 and shown at the magic show “Maskelyne and Cooke” at the Egyptian Hall in London, where he would write and draw pictures as suggested by the audience. This sketching automaton is composed of a small figure placed on a pedestal on the stage with a frame to hold sheets of drawing paper in front of the figure (Fig. 1c). The marvelous effect was produced by a complicated mechanism, which is set in motion by a man below the stage, and the apparatus below the stage is connected up to the figure by means of a small rod, which is passed up to the stage but hidden from the audience.

Another example of drawing automata of the early XX century is the machanical clown artist (Fig. 1d), a clever toy produced in Germany from approximately 1885 until about 1905 by the firm Phillip Vielmetter Mechanische Werkstatten of Berlin [8]. In this machine, a small pinion is driven by a worm gear at the end of a manually operated crankshaft within the base on which the mechanical artist is seated. The pinion, which is located on the horizontal plane, has a couple of pins on which one of the sets of replaceable cams that come with the toy is mounted. The motion of the first or second cam causes the line to be vertical or horizontal. Conversely, the joint action of the cams delivers diagonal or curved lines.

Gakutensoku (Japanese for “learning from the laws of nature”), the first robot to be built in Japan, was created in Osaka in 1928 [9] (Fig. 1e). The robot was designed and manufactured by the biologist and botanist Makoto Nishimura (1883–1956). Gakutensoku was first exhibited in Kyoto as part of the formal celebration of the Showa Emperor’s ascension to the throne. The robot visited several expos and wowed onlookers with its incredible calligraphy abilities before going missing while touring in Germany in the 1930s.

4 The Revolution of the 1950’s: Machines as Extension of the Artist

In the middle of the XX century, the Swiss sculptor Jean Tinguely (1925–1991) was one of the first artists to use painting machines to create complex and random patterns [10]. With his generative works called Métamatics, machines that produce artworks (Fig. 2a), he investigated the relationship between art and machine: his exhibitions presented artworks that are at the same time machines that produce in turn their own artworks.

Another example of drawing machine in the same period is given by the creation of Raymond N. Auger, who developed his automatic painting machine between 1955 and 1959 [11]. The actions of his machine, like those of a pianola, are dictated by a heavily perforated roll of thickened paper running through its “brain” (Fig. 2b). An armature is attached that opens and closes, grabs the brushes, dips them in four paint pots filled with black, red, blue, and yellow paint, revolves on a rotating stand and finally paints on the canvas.

Furthermore, in 1957 Akira Kanayama (1924–2006) started developing a painting machine, which was a four-wheeled device that the artist could control from remote to create paintings [12] (Fig. 2c). His machines were made by attaching a can of quick-drying paint to an automatic toy car that produced paintings mimicking Jackson Pollock’s drips artworks. The machine could operate even when the artist was not in the room.

Jean Tinguely’s machines gave inspiration to the cybernetic machines of Desmond Paul Henry (1921–2004), one of the few early British computer art and graphics pioneers of the 1960s [13]. During this time, he built three mechanical drawing machines (in 1960, 1963, and 1967) based on analogue bomb-sight computer components. These machines could not be pre-programmed and the resulting drawings, such as abstract, curvilinear, and repetitive line sketches, were each unique (Fig. 2d).

The beginning of computer graphics in the 1960s gave rise to the programming of more autonomous machines, and so appeared the first cybernetic sculptures and robot art [14]. In this context, Nicholas Schöffer (1912–1992) is seen as the founder of the so-called Cybernetic Art, with his CYSP I (1956), an acronym that joins the first two letters of the words “cybernetic” and “spatiodynamic” [15].

In the first half of the 1960s, Frieder Nake (born in 1938) [16], together with Georg Nees (1926–2016) and A. Michael Noll (born 1939), started to use the emerging computing technology as a creative tool for obtaining computer-generated patterns and used the upcoming pen plotters to create artistic graphics. For his contributions, Nake is considered one of the pioneers of computer art (or algorithmic art) and generative graphics.

Fig. 2.
figure 2

Drawing machines in the 1950s: (a) painting machine by Jean Tinguely; (b) drawing machine by Raymond N. Auger; (c) wheeled painting machine by Akira Kanayama; (d) cybernetic machine by Desmond Paul Henry; (e) AARON by Harold Cohen.

Full size image

In 1968, the British artist Harold Cohen (1928–2016) created AARON (Fig. 2e) at the University of California, San Diego: a plotter designed to produce artistic images, which is regarded as the most important painting machine in contemporary art [17]. AARON was developed in over 30 years of research, during which it produced physical artworks in the form of drawings and, later, paintings. AARON cannot learn new styles or imagery on its own; each new capability must be hand-coded by Cohen.

Examples of painting machines at the end of the XX century are given by the artistic installations by Rebecca Horn (born in 1944). In the late 1980s she developed a painting machine capable of spraying black ink (“black rain”) on a wall, by means of glass funnels, three books, a metal structure, as well as motors, and electronic devices to control the painting [18]. A further example of robotic painting in the late years of the XX century is given by the drawing machines by Angela Bulloch (born in 1966), first presented in 1991. Her machines draw horizontal, vertical and elliptical lines on the gallery wall according to some external stimulus, noises made by visitors to the exhibition, or the rhythm of their sitting on and standing from a bench placed facing the work [19].

5 Painting Robots in the XXI Century

5.1 Drawing Machines in the Early XXI Century

Drawing and painting robots proliferate in the XXI century, employing not only plotter-like machines, but also autonomous robots, industrial manipulators, and cable-driven devices. In 2001, Leonel Moura (born in 1948) developed the ArtSbot project (Art Swarm Robots), being inspired by ants to create unique artistic paintings, impossible to anticipate. In his works, several small autonomous robots, called Mbots, move in a haphazard way inside an arena, each equipped with color detection sensors, obstacle avoidance sensors, a microcontroller and actuators used for locomotion and painting [20] (Fig. 3a). His Robotic Action Painter (RAP) was exhibited as a permanent installation at the American Museum of Natural History in New York.

In the early 2000s, an unusual drawing project called MEART was developed. It consists of a pneumatically actuated robotic arm to create drawings (Fig. 3b), located at the Australian Centre for the Moving Image, and controlled by a living network of neurons from rat cortex grown on a multielectrode array (MEA) and placed at the Georgia Institute of Technology, Atlanta [21]. Internet was used to overcome physical distance between its components. Given its unique nature the MEART project was described as “neurobotic”, a cross between neurology and robotics, envisioning a future in which humans create “thinking entities” with the potential to become intelligent and unpredictable beings.

Additional examples of painting robots in the early 2000s were built using cable-driven devices. One notable example is Hektor, a portable spray painting machine created by Jürg Lehni and Uli Franke at the École Cantonale d’Art de Lausanne in 2002 [22]. The only components of this cable-driven mechanism are two motors, toothed belts, and a spray can holder (Fig. 3c). By the means of geometric triangulation and gravity, a custom software running inside Adobe Illustrator moves the spray can along predefined drawing paths and remotely activates the can nozzle.

A drawing machine with a similar working principle (a cable-driven robot with two degrees of freedom inspired by Hektor) is the Polargraph by the Scottish programmer, designer and maker Sandy Noble [23]. The machine decodes a bitmap and uses a polar coordinates system to generate a map of the file, recording pixel position, size, and brightness. Akin to a plotter, the hardware requests each pixel in turn and renders it on the page using its own shading and movement algorithms.

Fig. 3.
figure 3

Drawing and painting robots in the early XXI century: (a) mobile robots by Leonel Moura; (b) the MEART project; (c) the cable-driven robot Hektor; (d) the Robotlab project; (e) the drawing robot by Tresset and Fol Leymarie; (f) e-David by Deussen et al.

Full size image

Furthermore, an original painting machine was developed by Sam van Doorn, using a modified pinball machine, the Styn Flipper [24]. By playing this one-of-a-kind machine, unpredictable artworks are created based on the unusual patterns created by the motion of the balls, which are ink-covered spheres that move across a temporary poster placed on the game surface.

5.2 The Use of Robotic Manipulators to Make Art

With the development of robotic technology and the significant market availability of programmable manipulators and industrial robots [25], several examples of artistically skilled robots have emerged. These robots, often based on mechanical arms commonly used in factories, appear since the beginning of the XXI century.

The artist group Robotlab, founded at the beginning of 2000 by Matthias Gommel, Martina Richter (aka Martina Haitz) and Jan Zappe, developed a robot capable of painting portraits like a human painter. This was initially shown at the Center for Art and Media Karlsruhe (ZKM) in Germany [26]. The robot was also used for writing down the Bible on rolls of paper, as well as for months-long, uninterrupted large-size drawings (Fig. 3d). Machines that are normally located in special industrial spaces are integrated into the context of art through Robotlab’s projects, invading various fields such as music, dance, and science.

Other examples are provided by the work of Calinon et al., who created a humanoid robot in 2005 that drew human portraits using a robotic arm with four degrees of freedom (DOF) and a quill pen [27], and by the Chinese painting robot capable of drawing bamboo leaves with a brush created by Yao et al. [28].

The Artist Robot was installed in the Futuroscope Park in France in 2006 and drew pictures of visitors like a human artist [29]. In 2008, Aguilar et al. presented an articulated painting arm capable of converting images into painted artwork [30]. During painting, the artist controlling the system can influence the outcome by controlling both the hardware (such as the palette and brush type) and the algorithmic parameters. Furthermore, Kudoh et al. investigated and developed a painting robot with multi-fingered hands and stereo vision in 2009, which was capable of obtaining a 3D model of an object, composing a picture, and painting it on canvas [31]. Lu et al. developed a robotic manipulator that uses visual feedback to determine stroke position and orientation the same year [32].

Sun et al. also investigated a calligraphy robot, which consists of a 6-DOF robot arm capable of accurately reproducing Chinese Calligraphic characters [33]. Furthermore, Kim et al. built a robotic manipulator in 2013, proposing a system capable of detecting and extracting brightness in images and drawing them by repeatedly reproducing overlapped lines [34].

Further examples of artistic painting machines, mainly in the form of plotters, are those developed by the artists Ben Grosser, who developed an interactive painting machine (also capable of listening and producing music) [35], and Holger Bär, who started in 1987 to program the first plotter-style painting machine [36]. Furthermore, Doug Marx and Luke Kelly’s Vangobot is a specialized plotter capable of creating colorful paintings with multiple brushes and by smoothly blending colors [37]. However, no feedback mechanisms from the canvas is implemented for that robot.

Tresset and Fol Leymarie created a robotic installation capable of drawing sketches of people [38], which consists of a robotic arms bolted on an old school desk and holding a black ball-point pen (Fig. 3e). The machine can only draw obsessively guided by a visual input from a camera focused on the subject or looking at the drawing in progress. Tresset’s work explores recurring themes such as embodiment, passing time, childhood, conformism, obsessiveness, nervousness, the need for storytelling, and mark-making. He is best known for his performative installations that feature stylized actors making marks with robotic agents, as well as his exploration of the drawing practice with computational systems and robots.

In terms of brush painting, one of the most impressive examples of an artistic robot capable of reproducing non-photorealistic images is provided by e-David (Fig. 3f), a machine developed by Deussen et al. at the University of Konstanz in Germany [39], and still under development [40]. The purpose of e-David is to create acrylic paintings that look similar to those of human artists. This requires to be able to handle brushes and paints like humans do. For this reason, an industrial robot arm with six DOFs is adopted instead of a simple pen plotter, and a visual feedback loop controls the painting process using a standard digital camera. Strokes are not determined at the beginning, but are computed, iteratively, based on the difference between the given target function and the strokes already painted on the canvas.

Fig. 4.
figure 4

Other examples of drawing and painting robots in XXI century: (a) Busker robot by Gallina, Scalera et al.; (b) Baxter robot programmed by Daniel Berio; (c) light painting by Chris Noelle; (d) robot painting on large surfaces by Song et al.; (e) Dream Painter by Varvara and Mar; (f) PIX18 by Hod Lipson; (e) Cloud Painter by Pindar Van Arman.

Full size image

5.3 Recent Artistic Robots and the Role of Artificial Intelligence

The first example of painting robot using the watercolor technique is Busker Robot, developed at University of Trieste and Udine (Italy) by Gallina, Scalera et al. starting from 2016 [41, 42]. The system is based on a collaborative robot with 6 DOFs (Fig. 4a) and a series of algorithms for non-photorealistic rendering and image processing [43]. More specifically, starting with a digital image, the robotic system is capable of processing both the backgrounds and details of the subject to be painted, and extracting a series of paths that the robot can recreate on the canvas. The artist in charge of the system can change both the algorithm settings and the hardware variables (e.g., brush type, color dilution, etc.) throughout the process; this generates unexpected effects that are the result of both the software and the properties of watercolor. The same robotic collaborative system was also used for experimenting other artistic techniques, such as airbrush [44], palette knife [45], as well as etching and sponge painting [46].

As far as collaborative robotics is concerned, a notable example of robotic drawing using a collaborative manipulator is given by the work of Daniel Berio et al., who developed in 2016 an approach to generate rapid and fluid drawing movements on a compliant Baxter robot [47] (Fig. 4b).

Further special applications of robotic painting developed in recent years include the robotic light painting, performed both using a manipulator [48, 49] (Fig. 4c) or a quadrotor [50], to create light images generated in a long exposure photograph. Furthermore, the authors in [51] developed a robotic system for heliography (painting with the sun) using a thermal-vision guided robot. A notable example of artistic painting with robots also include the interactive multi-robot painting described in [52], where a team of mobile robots equipped with different color paints create pictorial compositions by leaving trails of color as they move throughout a canvas. Furthermore, painting on large non-planar surfaces has been also investigated by Song et al. using a 7-DOFs impedance-controlled manipulator mounted on a 3-DOF mobile platform (Fig. 4d) with impressive results [53].

Advanced recent robotic painting include not only systems based on image processing and non-photorealistic rendering techniques for image generation and robot path planning, as in [43, 54, 55], but also on a deeper interaction with the human artist. Examples are given by the robot avatar for portrait drawing by artists through remote manipulation via 5G network shown in [56], the inclusive robotic system to paint with eyes through eye tracking described in [57, 58], and the collaborative drawing framework with an industrial robot presented in [59]. Furthermore, another example of human-robot interaction is provided by Dream Painter (Fig. 4e), a robotic installation by Mar Canet Sola and Varvara Guljajeva for creating AI-based drawings starting from the voice of a user interacting with the robot [60].

In recent years, AI has been playing a significant role in the field of artistically-skilled robots, revolutionizing the intersection of technology and creativity. AI-driven algorithms are empowering robot artists to generate captivating and innovative artworks, blurring the boundaries between human creativity and machine intelligence [61]. In this context, Hod Lipson’s PIX18 (Fig. 4f), developed at Columbia University, is an example of painting machine based on deep learning algorithms that conceives and creates art on its very own, with minimal human intervention [62]. The PIX18 project has gone through three generations of systems; the first was an articulated arm and the latter were repurposed gantry robots. The software was developed from scratch and is continuously learning and evolving.

Another example of AI-driven painting machine is given by the robot Cloud Painter by Pindar Van Arman [63], who uses over two dozen competing AI algorithms to produce creative artworks (Fig. 4g). In particular, Generative Adversarial Networks (GANs) are used to imagine portraits, Convolutional Neural Networks to apply a style to imagined portraits, and Visual Feedback Loops to paint and analyze one brushstroke at a time. More recently, reinforcement learning is used for improving the control of artistic robot applications in [64], whereas StyleGans neural networks trained on Vincent van Gogh’s artistic heritage are used in [65] to paint artistic images in authentic colors.

Artificial intelligence is also applied for robotic painting in the Ai-Da robot, the first ultra-realistic humanoid robot created in 2019 capable of drawing and painting using cameras, AI algorithms, and two robotic arms [66]. Ai-Da can be seen as an example of how a human-like appearance can become popular, despite the results produced by this robot are simple and discontinuous strokes, each ending with a dot due to the incapability of the machine to control the pressure on the canvas and to the compliance of her arms.

The current and future role of AI in robotic painting raises questions about its capacity to create genuine art, as it relies solely on available data and the algorithms dictating its actions. This prompts a debate on whether AI-driven creations can truly embody artistic expression or if they remain fundamentally bound by their programming.

6 Conclusions

In this paper we presented a brief historical overview of robots that draw and paint, highlighting their functional and technical characteristics. We started with a description of the drawing automata by Jaquet-Droz and Maillardet: amazing examples of automatons capable of writing and reproducing various figures using mechanisms and cams. Then, we shown the evolution of drawing machines in the XIX and XX centuries, like the “Zoe” automaton, the mechanical clown artist, and the Gakutensoku Japanese robot. The painting machines that appeared in the 1950s have been analyzed, from Jean Tinguely’s kinetic machines to Harold Cohen’s AARON, as well as the first robotic drawing prototypes, which became increasingly autonomous and creative with the advent of cybernetics and computer graphics. Furthermore, we depicted drawing and painting robots of the XXI century with enhanced manipulation, control, image processing, and computing capabilities. Notable examples have been illustrated, as for instance the MEART project, the Robotlab group, the installation by Tresset and Fol Leymarie, e-David, and Busker robot. Finally, recent trends and future developments in this field have been discussed, such as the introduction of collaborative robotics, human-machine interaction, and artificial intelligence.