Keywords

1 Design Experiments in Nonrepresentational VR

Although virtual reality environments provide endless opportunities for novel perceptual experiences, many applications in this area restrict themselves to representational iconography and goal-directed activities (e.g., battles/fights, races, games, training simulations etc.). The design of such representational environments is based on the development of models (objects or processes) that simulate to a degree the real world. Nonrepresentational iconography refers to the depiction of objects and spaces that do not represent real objects. Opportunistic interaction refers to the embedding of the user in situations that present opportunities to act and explore a virtual environment rather than being subservient to specific goals imposed by the system. In nonrepresentational and opportunistic VR the lack of correspondence between perceptual and real-world experiences can be a mixed blessing. On the one hand, infinite design opportunities emerge as the analogs of real-world constraints can be ignored. On the other hand, this can lead to incomprehensible or alienating experiences. At the same time nonrepresentational design as a process is more complex due to the sparsity of tools and methods that can aid the designer in such systems.

We describe our experience with the development of two nonrepresentational mobile VR systems, Quoternion [8] and Atypical [9]. Quoternion is a minimal game in which the user navigates a non-objective space seeking to drive a set of spheres in a designated area in the presence of another distinct set of moving spherical objects that can collide with and deflect the user-controlled spheres. The environment imposes no time limits and there is no scoring, thus allowing the user to freely explore a constantly varying non-objective visual backdrop, while listening to philosophical quotes from a computer-generated voice and improvisatory jazz samples (see [7] for a short video of the work). Atypical is a variation on the same visual theme as Quoternion differing in that the user position remains fixed and there is no game played. Instead the user can probe the environment by firing spherical objects at desired directions. If these objects collide with other moving objects a random audio, video or textual effect is created. Furthermore, in Atypical the environment constantly provides interaction opportunities through the generation of novel biomorphic objects consisting of flower textures projected in constantly changing geometries and various other moving shapes. The difference in user position between these two systems reflects our desire for Quoternion to be used by a user standing up and able to freely rotate, while Atypical can be used while sitting. Since there are no real-life analogs to base one’s creative efforts, nonrepresentational VR design has to rework the basics of form, color, motion, rhythm and sound to this new medium. In this respect, it faces similar problems to those negotiated by early 20th century abstract artists such as Kandinsky [5] or Mondrian [6]. In addition, we believe that the absence of real-life grounding of the experience motivates the user to actively construct meaning based on his personal experiences similar to what is the case in non-objective art [4]. Based on these observations, we strove for a completely stochastic and geometrical visual backdrop for our environments. This consists of a varying set of spheres and cubes that are continuously deformed and rotated with different speeds. In addition each such surface is rendered using two rotating texture coordinate systems over which continuously deforming star and circle shapes with varying colors are mapped. There is no correlation between the transformations applied to the objects forming the visual backdrop. Consequently, the user feels that there is no coordinated motion of the backdrop objects, something that we have found empirically to reduce the appearance of vection. Furthermore, the space is populated with rigid moving spheres with stable color patterns. These spheres offer interaction opportunities to the user. All moving objects use periodic functions in their transformation thus allowing the imposition of a desired rhythm in their individual behavior. Rhythm is also imposed by the use of a constant rhythmic part (a drum loop in the case of Quoternion a piano waltz in the case of Atypical) embellished with jazz samples for Quoternion or random piano or percussive samples notes for Atypical that are triggered by user actions in the environment. In both cases we impose no restrictions on the ways with which the user interacts with the environment. In Quoternion the game is presented as an opportunity to enrich the exploratory experience rather than a task, while in Atypical the user can choose whether to probe the environment or not. Consequently, she has total freedom to explore each environment at her leisure. Furthermore, in Quoternion the user decides whether to move or stop by pressing a controller button. The direction of motion is always forward from the current orientation of the user’s head. Therefore the user’s motion is always deliberate and predictable providing a consistent kinesthetic experience. Figure 1 gives an idea of the overall visual environment in these environments. Our preliminary trials with ten users indicate that they find this interaction pleasing. We have observed no adverse reactions (e.g., nausea) so far for uses of the system averaging 20 min.

Fig. 1.
figure 1

Example of nonrepresentational visual backdrop and moving objects in Quoternion.

Full size image

On a more practical level we envision the use of Quoternion and Atypical as parallel supporting activities that can maintain engagement in a boring or unpleasant main task and enrich the overall experience. For example, we have started testing Quoternion and Atypical while exercising on a treadmill or indoor bicycle with encouraging results. We will try to test both systems in more somber environments related to patients undertaking chemotherapy or hemodialysis treatments.

2 Symmetrical Texture Generation in Real-Time

Symmetrical patterns have been a constant source of inspiration for various visual art forms by evoking feelings of order, harmony or balance [1]. Furthermore, the existence of detailed mathematical descriptions for generating all possible symmetries on the plane has opened up the possibility of efficient, algorithmic generation of such patterns. The kinds of iconography that can be generated from symmetry is vast and can be made even bigger by combining symmetrical forms with color or motion animation or by introducing noise in such patterns in a principled way. In this case, the important question that concerns us is how we can efficiently help and motivate digital artists to create and experiment with symmetrical designs that are visually interesting and aesthetically pleasing. Our efforts in dealing with this question focus on the creation of digital environments for creative visual exploration of symmetrical patterns and their interplay with geometry and animation. We want to create tools that can be easily deployed by artists and designers, therefore we target mobile devices as our main development platform. To this end we have created and deployed Symmetricon [10], a free app for generating, animating and displaying, in real-time, plane symmetry patterns. Symmetricon is able to create 30 types of plane symmetries [3] from either the camera stream, or images and videos stored in a mobile device. There exist a variety of mobile apps that focus on the generation of symmetrical patterns (e.g. Kaleidoscope Pro, Picasso, etc.). Most of them are drawing apps that allow the user to create the primary cell and then generate mainly kaleidoscope tessellations. To the best of our knowledge Symmetricon is the only existing app that can generate animated, symmetrical tessellations from real-time camera stream and apply them as textures to various surfaces.

The symmetrical texture generation process in Symmetricon receives as input the frames captured from the camera stream of a mobile phone. These frames comprise the primary cells for the subsequent operations. Symmetricon then generates the final repeating tile which is either a rectangle or a hexagon depending on the symmetry group specified by the user. This tile is a bitmap image which is passed as a texture unit to the fragment shader of an OpenGL environment. This is straightforward in the case of rectangular tiles. In the case of hexagon tiles, given that textures are rectangular in OpenGL, the hexagon tile is turned into a rectangle by adding the top left triangle to the bottom right side of the hexagon and so on. The rest of the operations consisting of all the motions (translations, rotations, reflections and glides) applied to the repeating cell in order to create the tessellation are being performed in the fragment shader in parallel thus speeding the tessellating process and achieving real-time performance. In parallel with the generation of the tessellation the fragment shader computes the animation pattern specified by the user in the GPU as well. This can either be a horizontal/vertical translation of adjacent rows/columns of the tessellation, a rotation of a 4 × 4 tile block around its geometrical center or non-linear vortex and tunnel effects. The resulting tessellation is being pasted upon either a square surface, a sphere surface, a square grid (a square surface sampled with a grid of points) or a combination of the above depending on the user choice. The user can select a variety of animations thus creating various wave and geometrical deformation special effects. Symmetricon allows the user to select the types of symmetry groups he wants to use, the size of the tessellation along with the specific texturing surface (square, sphere or rugged plane) he wants to create and the type of animation he wants to employ on the pattern (none, horizontal, vertical, cyclical, vanishing, tunnel or pulsating effects). Based on these user specifications, the system creates a tessellation of the desired symmetrical pattern and embeds the resulting texture in a 3D scene. The user can then view the generated texture from any distance or viewing angle in this 3D space. He can also store the textures he finds interesting as images in his phone. Symmetricon provides the designer with the opportunity to generate and test various motifs under an unlimited set of continuous and constantly changing imaging conditions. This enables the app to be used as a visual exploration environment that allows its users to create infinite color and geometrical variations of symmetrical textures using as raw material real-world scenes captured by the camera of a mobile device. In this case, the artist/designer can control the camera position and motion of the mobile device during the capturing of the raw material while also applying on it various animation patterns and projecting it on various surfaces. A video showing a selection of patterns created by an extended version of Symmetricon while randomly scanning a typical office space with the camera of the mobile device can be found in [2]. The particular app version used in the video is able to project the textures simultaneously on an unlimited set of texturing surfaces where each of these surfaces is specified by the user. Figure 2 provides some more examples of the possible tessellations created with Symmetricon.

Fig. 2.
figure 2

Three examples of tessellations generated by Symmetricon.

Full size image

3 Conclusions and Future Work

Our goal is to create systems for the design and exploration of nonrepresentational environments that can be easily deployed and used. This is the reason we base our development efforts on mobile platforms. Quoternion and Atypical run on the Daydream platform while Symmetricon is available for Android phones. Our future research efforts in the case of Quoternion and Atypical will focus on the enrichment of their iconography and interaction opportunities. In the case of Symmetricon we will focus on speeding up further the tessellation process by parallelizing the primary cell generation which currently is executed on the CPU. Finally, we are developing synthesis environments for nonrepresentational graphics that can fully exploit multi-core processor architectures. In this case we create process grids, where each process corresponds to a pixel with full messaging capabilities between then to explore the dynamic generation of nonobjective patterns, their interactions and the iconography they produce.