Keywords

1 Smart Participation Culture

Today we live in a society where more and more people own a smartphone or a tablet device. According to the United States (US)-American Network Digital Trends the number of smartphone users is expected to reach over 6.1 billion by 2020 (Boxall 2015). Today’s number is also impressive: as the Hamburg based company Statista has evaluated, the smartphone users worldwide will reach over 2 billion this year (Statista 2015). The predictions vary extremely but the trend is clear: smartphones will be the worldwide standard communication technology in the near future.

Alongside with the massive distribution comes a diversification of the devices. E.g. a wide variety of displays is available: these range from common off-the-shelve-devices like tablets and smartphones that can be employed in BYOD (bring your own device) fashion, over specialized devices [Samsung has the first augmented reality (AR)-glasses for smartphones on the market], head-mounted displays (HMD) to custom solutions like GeoScope and projection-systems on physical three-dimension (3D)-city-models. These options vary in their quality of visual augmentation, support for group activities in discussions of proposals and respective cost for hardware and content.

We also live in a world where networks and participation is increasing massively. Facebook alone had 1.591 billion users at the end of 2015. Other networks like snapchat are growing at high speed (over 350% in the last 8 months) (Statista 2016).

The consequence is that user culture and user habits are rapidly changing. Openness, transparency and participatory potential are key factors for today’s citizens and their everyday life. That again has influence on a lot of social environments and decision-making processes that were accessible only for professionals before, but that have to open themselves for citizen-knowledge and non-professionals. One of these environments is smart city development. The success of every city development project now and in the future depends on how the changed user culture is addressed and used.

Keeping in mind that everybody has a smart device and interconnects with many things on several levels, one can state a grown interest in the topic of urban re-design. People are engaging at large numbers in urban gardening projects and other initiatives, trying to shape the urban environment. People come together in smaller and larger groups, online and offline, and talk about urban change and smart use strategies.

What happens is that with growing knowledge about city development, growing interaction and participation in urban processes, citizens develop a demand for creating knowledge and want to be asked and be part of the development of their urban surroundings.

What does that mean for the future of smart city projects? It means that these wishes and demands need to be picked up in their progression. Participation will keep growing. The number of smart devices will keep growing and people will always claim their right to be asked.

The “People’s Smart Sculpture” (EU-PS2), a large scale Creative Europe co-funded project, addresses these issues. EU-PS2 raises questions about the design of smart city technologies; and creates surroundings where citizens and artists as well as professionals are enabled to create their urban visions by using these technologies. The background is that actual software and tools for smart city planning target only the relevant administration staff, architects and other professionals. These systems create significant barriers for citizens’ participation in the planning process as they were designed with the professional user in mind. Professionals are used to work with highly abstract data, while citizens would require much more direct visualization. Best practices are smart in-situ Mixed Reality (MR), combining the real location with planning data.

The technologies tested and implemented in EU-PS2 show how smart participation in urban contexts is possible to be in use. Cities will become interactive corpuses through the new user culture, enabled by combining new technologies and social developments.

The remaining is organized as follows: the first two sections introduce the related work in context of urban development tools, MR and gamification. The fourth section presents the “Express Yourself/city” use case with the Betaville project at the Public Urban Lab Bremen, as part of the EU-PS2 project. In the fifth section, AR tracking will be explained and our AR framework will be introduced. The last section contains the conclusions and future steps.

2 Urban Development Tools

Urban development has been a widely discussed topic in a lot of scientific disciplines in recent years. Along with the digital revolution the field has opened up; once only a topic for architects and city planners it has become a favorite among social and cultural scientists, ecologists, computer scientists, and many more—they are all lining up together to interpret and shape the urban future. Modern approaches towards urban development bundle knowledge and know-how from different disciplines in theoretical or practical methods and technologies.

The reason for these broad, cross-disciplinary, activities in urban development is quite evident. We live in a rapidly transforming world. Urbanization is one future key challenge for humanity—quite as urgent as dealing with climatic change. According to the United Nations (UN) World Urbanization Prospects of 2014 “by 2050, 66% of the world’s population is projected to be urban” (UN 2015). That creates great societal changes. A lot of theories on urban development discuss ideas on how we should adapt the design of our urban spaces to the increasing number of people living in cities.

Very often theories about urban development refer to the smart city term. Mostly smart cities are associated with smart technologies, i.e. technology developed during the digital age like social media, big data, robotics, cloud-computing, AR or of course new and more efficient mobile devices. But the large number of existing definitions involves more aspects. A simple but very powerful definition was given by Caragliu et al. (2009):

We believe a city to be smart when investments in human and social capital and traditional (transport) and modern (ICT) communication infrastructure fuel sustainable economic growth and a high quality of life, with a wise management of natural resources, through participatory governance.

This rough definition contains all relevant aspects discussed in current research (see Anthopoulos 2015). Although urban development should not be mixed up completely with the smart city debate, the mentioned characteristics feature certain aspects of particular interest for the conclusion of this paper:

  • the human factor: smart cities cannot be realized without the smartness of humans;

  • use of modern communication devices like smartphones and tablet devices;

  • participatory strategies enabling to play a new role in urban development.

AR and MR applications, using novel interaction and visualization techniques, have high potential to provide attractive user experiences in smart city planning. A central challenge is the choice of an appropriate display technology. A wide variety of displays is available for the implementation of MR: tablets and smartphones, AR glasses, HMD, projection systems, or custom solutions like GeoScope. These options vary widely in the quality of the visual augmentation provided, the cost of the hardware, and their suitability for different usage settings.

While technical data such as resolution and cost are important for the selection, other aspects that are not easily established from data-sheets like ergonomics, comfort and reliability are essential for acceptance. We have gathered design experiences and user feedback from over 10 years of work with MR applications for smart city planning and public participation. Based on this, we provide a structured approach for the informed selection of MR displays that we use in our development activities.

Gamification is a fun way to engage people to solve real life problems. It picks up elements of games like goals, levels, bonuses, progression, challenges, status, statistics, rewards and many others; and applies them with a game design technique in a non-game context. Game design techniques involve the process of conceiving a game. To design a game you need to know your target group. Once you know your target group, you can design a suitable game for this, combining the elements of game. A gamer will spend time and energy in useful work and can simultaneously collaborate with other gamers to solve large-scale problems. Gamification can be applied in many domains, such as security, computer vision, adult content filtering, and Internet search. One such application is Labelling of Random Images on the Internet (Von Ahn 2006): On the Web, millions of images exist without any appropriate textual descriptions or classification. Describing the content of an image is quite complex, using computer vision algorithms. However, this problem can be solved using the above-mentioned game. It is played by two players at a time in which they cannot communicate with each other. The goal of this game is to guess what label the other player has given to a particular image. If the labels match, the players get points and the image can be labeled with the matching word.

3 AR for Smart Urban Development

The realization of a MR system requires some means of visually combining computer-generated graphics with the real physical environment. The three principal technologies to achieve this are: video-see-through, optical-see-through, and spatial projection. The choice of technology has important impacts on the design of the application, as some are not applicable in all application settings; and they differ significantly in the interaction modalities supported. More specifically:

  • Video-see-through systems use a camera to capture a current live-view of the real-world environment. This video stream is then augmented with additional graphical information and displayed on a conventional display.

  • Optical-see-through systems use an optical combiner, e.g. a semi-transparent mirror, to overlay computer graphics over the user’s view of the real-world environment.

  • Spatial projection systems use video- or laser-projectors to project the augmentation information directly onto the physical environment.

The choice of a display technology, and consequently of a specific display that implements it, influences important aspects of the resulting application. These factors include technical data like resolution, contrast and dynamic range; environmental aspects like waterproofing and ruggedness; ergonomic aspects like adjustability, balance and weight; usability and user experience aspects like utility, comfort and attractiveness. These factors must be considered early in the design process to make a suitable choice of display technology, since changing display technologies later on will require significant and costly redesign.

Over the last decade, we have developed a variety of MR systems for smart-cities, urban planning and architecture, addressing both the requirements of end-users and content creators. In this section, we present several display technologies that were used successfully in these projects; and discuss specific considerations that are relevant for designers.

Mobile Video-See-Through Devices: e.g. Smartphones and Tablets

Smartphones and tablets can be used as video-see-through MR devices. Attractive properties for designers are the relatively low prices and the fact that many potential users already own a smartphone, which enables the implementation of BYOD scenarios, in which users use their own devices. Significant limitations are the small display size and frequent issues with tracking, since only the sensors already integrated into the device are available [usually global positioning system (GPS), camera and inertial sensors]. Smartphones and tablets can be used as a baseline, as they are currently the most frequently used platform.

Located Video-See-Through Devices: e.g. GeoScope

Video see-through devices that are installed at a fixed location enable precise localization and the use of additional sensors. An example is the GeoScope (Paelke and Brenner 2007), a device that works similar to a telescope in a fixed location, where the user can adjust jaw and elevation angles by pointing the device at specific points of interest. The mechanical sensors that measure these angles, combined with the precisely known location, provide very precise tracking. Disadvantages are the limited flexibility caused by the fixed location; and the need for a custom designed and build device, including the need for making it robust or even vandalism proof if operated in public settings.

Mobile Optical-See-Through Devices: e.g. AR Glasses

AR glasses using optical-see-through techniques like Epson’s Moverio and Microsoft’s HoloLens are used in a small, but increasing percentage of MR applications. A key advantage is that the user retains vision of the real environment in addition to the augmentations; and that glasses allow hands-free use. Central limitations of these devices are the relatively high costs, limited autonomy due to small batteries, and significant weight.

Projection Systems: e.g. Projection of Physical Models or Maps

Projection systems can be used to provide augmentation on physical real-world environments (video mapping), on physical models (e.g. on 3D printed city models) or onto paper maps. While projections in outdoor settings are limited by the lack of mobility, the high setup cost, a small augmented area and issues caused by environmental light (e.g. video mapping on facades can usually only be used at night), indoor projections are a flexible technique that encourages collaboration as these can usually be viewed by many users at the same time. To provide a spatial reference for the viewers, white physical 3D models (which can be created by rapid prototyping techniques like 3D printing) can be used as the background, onto which the augmentation information is projected. Such setups are attractive but costly. A cheaper alternative is the use of paper maps as the projection background. However, some users are unfamiliar with relating maps to physical reality, making physical 3D models more useful for such clients.

The choice of a suitable display is an essential building block of a MR system. We have presented an overview of proven solutions from past projects (Fig. 1); and are continuing research and evaluation to extend our knowledge-base to optimize our selection process and provide consulting based on results. The work presented in the remainder of this paper relies on mobile video-see-through devices.

Fig. 1
figure 1

Choice of appropriate display technology

Full size image

4 Express Yourself/City Use Cases

This section includes four parts. The first introduces the EU-PS2 project. Next, the Betaville system for participatory urban design is presented. The third part talks about the experimentation with the Betaville system in the Express Yourself/city context. Finally, the last subsection enumerates the Betaville outcomes and shows the measures taken in order to enhance the Betaville platform.

4.1 The EU-PS2 Project

“Express Yourself/city” is a sub-project of EU-PS2Footnote 1 that is a creative research and innovation project about the design of methodologies and tools for the participative cultural evolution of urban spaces in Europe. The approach blends different levels of access: public participation, collaborative creativity, exploratory and game-based learning. It consists of 12 project partners in 11 sub-projects in 8 European countries involving citizens, artists and creatives from 29 European countries. The approach works on two levels: the implementation of cultural participation projects by researchers, artists, creatives on the one hand; and the ongoing optimization of the participation aspects in these projects through reflection and evaluation in a series of participation workshops in the 8 countries. Thereby EU-PS2 integrates diverse groups of people into a practical dimension of re-design of the urban environment, all working together towards realizing the vision of a hybrid open environment where everybody can follow—even change—the ideas concerning collaborative re-design and development of urban art (Koplin 2014). The project is also the base for a new deal between artists, experts, citizens, learners, creators and the government. It is a performative sort of integrated art to combine social and cultural sustainability in the city (Koplin et al. 2016).

Innovative technologies and tools are key to realize the project’s visions. That is why in a lot of sub-projects new technologies are developed and tested. Within the sub-project “Express Yourself/city”, the aim is to use digital media and gamification experimentally in participatory urban development. In art projects, workshops and public urban labs, 3D objects are to be placed on sites that are critical for the urban development of the city of Bremen. These 3D objects illustrate new ideas and alternative suggestions for the development possibilities of places and re-design of town virtually. The activities in Bremen include a series of creative and artistic interventions as participation actions in different parts of the town and also social art and experimental, informal learning activities. “Express Yourself/city” addresses and profits from the fast growing percentage of people already making use of pads, tablets and smartphones; and invites all citizens to participate in attending, responding and modifying these sculptures at the real spaces in the city of Bremen. One of the technologies used to achieve that is the participatory platform Betaville.

4.2 Betaville

Every city is “in beta”, every city is incomplete and under on-going development. For the development of a livable city, the citizens’ demands, potential interests of authorities, and technical restrictions should be taken into consideration. This requires the active participation of different parties, such as a city’s residents, experts and potential investors in the urban development and decision-making process. Starting out from this premise, the Betaville project at M2C (Institute of Applied Media Technology and Culture Bremen), the City University of Applied Sciences Bremen, the Gotham Innovation Greenhouse New York and the BXMC (Brooklyn Experimental Media Center) of New York University aim at providing online collaboration platforms in the field of urban development (Koplin and Skelton 2012).

Urban development projects in Betaville are represented by an area in the real world that is defined by a center coordinate and a number of GPS coordinates bounding this center. Each project can be associated with proposals, where a proposal constitutes a suggestion for urban development within the regarded area. Proposals form a hierarchy with the first published proposal at the root and alternative suggestions (e.g. modifications to a proposal) as its descendants. The actual urban development suggestion is expressed using a textual description and a set of 3D models of e.g. buildings, lakes or trees that a proposal can be associated with. Since projects, proposals, and models are user-generated entities, each of them has a user as its creator–a city’s engaged citizen. The class diagram in Fig. 2 illustrates a simplified view of the Betaville data model.

Fig. 2
figure 2

Class diagram of Betaville data model

Full size image

The Betaville platform comprises two mobile clients (smartphone and tablet), a multi-touch table and a web client, which are connected to a server that serves the data storage and exchange of project and proposal data via a representational state transfer—application program interface (REST-API) (Fig. 3).

Fig. 3
figure 3

Betaville platform overview

Full size image

Using state-of-the-art technologies, every member of the community has the chance to participate in a city’s on-going development projects online, e.g. by proposing new ideas in the shape of 3D models, by refining and extending already published proposals or by commenting on them. The process of repeatedly voting for one out of two proposals suggested by the system makes it possible to include the users’ opinions in the selection process of the most popular proposal, which represents the voice of the community. This gamification feature can help to increase the participation of people by making Betaville more fun and enjoyable (ISEA 2011).

The mobile client is implemented as an Android app for smartphones and tablets. The mobile client leverages its mobility and location-awareness features: using the camera, GPS and compass that are built-in in the current generation of mobile phones and tablets, its AR mode provides the user with impressions of development proposals right on the spot. Figure 4 presents an arts’ installation, created during an event within the EU-PS2 project.

Fig. 4
figure 4

Arts installation in AR view on Betaville tablet client

Full size image

In AR mode, the mobile client shows its true potential: users may experience the different proposals of a project in the real world at their designated location. The impressions collected by the user in this situation may stimulate further user activities, such as engaging in discussions, voting for or against a proposal or designing a new proposal. The mobile client also contains a map view, enabling the user to browse through all projects in the system; and locate them on a map.

In off-site mode, when the AR feature is not available, the mobile clients offer an interactive 3D view of proposals, where photographs of the surroundings are used instead of reality. As a result, users are still able to get an impression of a proposal within its designated environment. Moreover, they are able to interact with the 3D view, i.e. to apply transformations such as rotation or scaling, and hence look at a proposal from different perspectives. The right-hand side of Fig. 5 illustrates this feature, while the left-hand side shows a list of proposals in the browse view.

Fig. 5
figure 5

Betaville tablet client of City University of Applied Sciences Bremen

Full size image

The main purpose of the multi-touch application in Betaville is to support the collaboration of multiple users in the urban development and decision-making process. Due to the large screen of a multi-touch table, people are able to work simultaneously on such a system (Fig. 6). Hence, the application offers a possibility to a group of people to browse, compare, discuss and vote for urban proposals together.

Fig. 6
figure 6

Team work at Betaville multi-touch table

Full size image

The web client serves as the central point for project and user management in the Betaville system. Moreover, the web client offers functionalities for more advanced tasks (than those provided on the clients), which can be performed from a desktop computer. Since the web client is available on the internet, it can be linked to activities in social networks or other public announcements, and therefore also serves as a reference point for getting new users interested into the system.

The Betaville server architecture supports reusable components and is extendable for future additions, due to the use of many features of the Spring Framework (Spring 2016). The most integrated Inversion of Control (IoC) and Dependency Injection (DI) in the Betaville server implementation, authorization is based on Uniform Resource Locators (URL); and a custom user-authentication is provided, which interacts with the Spring security framework. The Betaville server provides web services, which can be consumed from the mobile and multi-touch clients through a JavaScript object notation (JSON)-based hypertext transfer protocol - application program interface (HTTP-API). In order to achieve this, the web service layer is implemented, using the Spring model-view-controller (MVC) framework. The server manages the data generated and consumed by the aforementioned clients. Further information about the Betaville project can be found on the project’s website (Betaville 2016).

4.3 Betaville and Express Yourself/City

Betaville was tested and evaluated during a one-week Public Urban Lab in Bremen in June 2016. Therefore, an overseas container with a digital media lab was positioned at a prominent place in the Bremen urban space, in front of the main station. Artists, scientists, students, teachers, and professionals started a dialogue with citizens and creatives on the topic of urban participation. They presented Betaville within the public, stimulated the use of the different clients and collected valuable feedback for future working steps.

At the public urban lab, citizens, artists and professionals had been also able to learn and discuss technologies developed in other EU-PS2 sub-projects, like apps and tools that can inspire and motivate people to become active participants in the smart city shaping processes. This also can help to find new approaches to enhance Betaville and, more importantly, to define strategies towards smart participation methods and the design of participatory tools and technologies in smart city planning. The Public Urban Lab Bremen therefore works as an empowerment or at least awareness-raising machine for the topics discussed in this paper among the Bremen population.

In EU-PS2, Betaville is of significant importance in the sub-project “Express Yourself/city”. In this sub-project, the platform is used for creative design and learning experiences. Betaville supports the aim of “Express Yourself/city”, which is to create and experiment with new digital media and gamification in participatory urban development. Students, artists and other stakeholders will use Betaville in art projects, workshops and Open Labs to place 3D objects on sites that are critical for the urban development of the city of Bremen; and would strongly prefer smart tools instead of town hall meetings. 3D objects illustrate new ideas and alternative proposals for the development possibilities of places; and support re-design of the town virtually. Citizens, artists and creatives shape 3D objects, modify and comment them, and create alternative visions for the development of the critical areas. Figure 7 shows the user interaction with the tablet clients during the art event Rememberti on urban change in Bremen in September 2015. The event created about a dozen of new visions for an urban area that has been subject to controversial debates over the last decades.

Fig. 7
figure 7

Experimenting with the Betaville clients at the art event Rememberti

Full size image

An important reason why Betaville can serve as an ideal tool to create educational effects in the EU-PS2 project is that it allows access for everyone from everywhere, which helps to reach a huge number of participants for the events. Users can profit from Betaville because it combines several advanced mobile smart city learning concepts, including AR, integration of mobile and social media channels and mobile tagging in a single platform approach. As Buchen and Pérez-Sanagustín (2013) have pointed out, the concept of smart cities goes far beyond technologies and technological infrastructures. For them smart cities need:

Locally and globally interconnected citizens who use smart technologies to learn by using, sharing, remixing and co-constructing learning resources, and in this way actively contribute to solving societal, environmental, political and economic challenges.

The design of Betaville contains all of these prepositions, especially because it puts the “Smart Citizen” in the central role on the way to Smart Cities. With its application in the EU-PS2 project, it perfectly serves the main characteristics of Smart City Learning scenarios as Buchen and Pérez-Sanagustín (2013) see them.

4.4 Directions

The experiences of the users in the workshops and events will directly influence the further development of the web client, which is also part of EU-PS2, and of the Betaville platform in general. Among the strengths of the Betaville system are: the in situ experience; the fact that it can activate a wide range of people usually not interested in participating in the processes of urban development; and the assembling of public opinions, prior to the actual planning action in an urban area. The Betaville web and mobile clients, however, show two major shortcomings:

  • Users’ possibilities to actually create 3D models and hence the personal design possibilities are very limited. Due to these limitations and the fact that 3D manipulation functionality might also be required on the web client, a next logical step in the development of Betaville could be the re-design of its client-technologies, based on hypertext markup language 5 (HTML5). Since HTML5 contains web graphics library (WebGL) to display real-time 3D graphics and works with different operating systems and browsers, it might be the technology of choice for client-independent 3D rendering and manipulation.

  • Betaville uses location-based data, provided by a mobile device’s GPS sensor to place the 3D models in the desired area. The delivered location data, however, show a tolerance of ±5 m, which is unacceptable for a convincing AR experience. To make the visualization more accurate, we propose the use of optical AR, tracking along with location-based AR tracking.

Our approach for marker-less optical tracking is described in the following section.

5 Work in Progress: A Configurable Solution for Mobile AR

In the following two sections, tracking is presented as one of the key challenges for developing an AR system (Azuma et al. 2001); and the “Marker-less Adaptive Mobile AR in Games” (MadMAGs) framework for configurable mobile AR solutions is introduced.

5.1 Tracking in AR

Tracking means to determine the pose of an object, i.e. its position and orientation with respect to some coordinate system in real time. Zhou et al. (2008) categorized the AR tracking techniques in sensor-based and hybrid tracking techniques.

As the name suggests, sensor-based tracking uses several types of sensors. Tracking can be based on a variety of sensors (Welch and Foxlin 2002; Lane et al. 2010) like camera, magnetometer for measuring the magnetic field (compass), gyroscope for measuring rotation, accelerometer for measuring translation, and finally the GPS sensor. These sensors provide us with information about reality, like position and motion. All these information can be used in a tracking method to recognize objects around us; and to estimate their position relative to us.

Vision-based tracking techniques are a part of sensor-based tracking, which use image information to track the position and orientation of a camera (Yang et al. 2008). There are two approaches for vision-based tracking: marker-based and marker-less. Marker-based tracking on the one hand uses a marker, which typically is a black and white 2-dimensional texture and has specific properties that make it easy to identify their position in the real world. Marker-less tracking, on the other hand, does not use any specific marker or references, but relies on an object’s innate (visual) features; knowledge of these features can be used for tracking in an unprepared environment.

While each sensor comes with its inaccuracies, hybrid-tracking techniques combine various sensor data into a merged data stream in order to enhance the quality of tracking data (Azuma et al. 1998). However, hybrid technologies increase the complexity of the tracking process (Rolland et al. 2001).

Tracking still remains a major challenge for AR applications. It has to be as precise, accurate and robust as possible in order to create the illusion that the virtual content is a part of the real world (Azuma 1997). An accurate tracking system is required for AR systems because even a small tracking error may cause a noticeable misalignment between virtual and real objects (Wang and Dunston 2007). (Visual) tracking has been the most popular research area in AR (Zhou et al. 2008). The following section exhibits the necessity of computational framework in AR system and MadMAGs Framework will be presented.

5.2 Marker-Less Adaptive Mobile AR in Games

Besides tracking, registration, i.e. the process of aligning a virtual object in the real environment, and interaction, Höllerer and Feiner (2004) mention three more requirements for a mobile AR system:

  • computational framework;

  • wireless networking;

  • data storage and access technology.

AR systems have to perform some typical tasks like tracking, sensing, display and interaction as shown in Fig. 8 (Van Krevelen and Poelman 2010). A computational framework can generate and manage the virtual material to be layered on top of the physical environment, process the tracker information and control the AR display (Höllerer and Feiner 2004). There are two kinds of framework architectures in AR systems: autonomous and distributed (Lopez et al. 2010). In autonomous frameworks, the components run in the same physical environment; whereas the components in distributed frameworks are located in networked computers.

Fig. 8
figure 8

Typical AR system framework tasks. Reproduced from Van Krevelen and Poelman (2010)

Full size image

A modular architecture is the way to design AR tracking algorithms in a flexible manner. Frameworks are desirable that allows to flexibly exchanging these principle steps of tracking to allow feedback loops and additional processing steps. The framework’s components can be combined with each other to assemble new configurable algorithms. Different settings such as indoor or outdoor and domains from architecture visualization to mobile gaming or machines maintenance pose different requirements for mobile AR (Skelton et al. 2011).

The focus of the MadMAGs project is to design and implement a mobile, platform independent and flexible framework, promoting the development and (re-)combination of innovative algorithms in the fields of image processing and computer vision in mobile AR applications. At the core of the targeted AR framework, we identified the needs to be able to process streams of (image) data on the one hand; and to represent a tracking and registration solution as a configurable network of algorithms on the other. Such a network can be illustrated by directed graph composed of:

  • exchangeable processing nodes implementing independent and concurrent algorithms; and

  • dedicated coordination nodes taking care of the synchronized transmission of data between processing nodes.

On top of these fundamental requirements, we have to cover additional functional requirements, such as integrating sensor data from various sources (e.g. camera, gyroscope, GPS) as a means of enhancing (optical) tracking quality, augmenting a scene by overlaying virtual objects on the camera image, and facilitating interaction both with the virtual camera and virtual objects. Moreover, non-functional requirements like extensibility and performance are key factors to the success of the aspired solution.

The software architecture is based on the Pipes and Filters style (Buschmann et al. 1996), where processing nodes are represented as filters and coordination nodes as pipes. A network of pipes and filters is then referred to as pipeline. To prove that the proposed software architecture meets the defined requirements, a first prototype is implemented and run in a desktop environment. A pipeline configuration gives first evidence regarding the platform’s flexibility and performance: the pipeline processes a camera’s images, extracts features, and tracks these features in consecutive frames using the optical flow algorithm, thus generating a list of movement vectors (red arrows in Fig. 9). These movement vectors serve to update the camera’s position and orientation within the 3D coordinate system, and hence the display of the scene.

Fig. 9
figure 9

3D tracking based on optical flow using the MadMAGs framework. Reproduced from Pooya (2016) (color fig online)

Full size image

6 Conclusions and Future Work

The current work shows that participation in the urban and smart city context relies on the implementation and use of new smart tools and technologies that support the interaction of people by making their visions for future city developments visible e.g. with AR features and by combining it with entertaining and motivating interactive aspects of gamification.

The paper lined out that the EU-PS2 project serves as an ideal surrounding for innovation, testing and improving participative tools for urban development. The “Express Yourself/city” sub-project provides a perfect use case for the application of the Betaville system and MadMAGs technologies. Betaville enables people to step into the processes of urban planning—something non-professionals usually have no or very limited access to.

It has been shown that the Betaville feature with the biggest participatory potential is the in-situ experience achieved through AR presentation of urban development proposals. Currently, this feature is not working to its full extent due to the inherent inaccuracies of GPS-based positioning of 3D objects.

The MadMAGs project aims to provide a solution to this issue by using innovative 3D optical tracking methods. Optical tracking is often used in AR to attach virtual objects, like prospective architecture to the camera image of a site seen through a mobile device. The advantage of optical tracking is that it reacts much faster and more accurate to changes in position and orientation, compared to gyroscope and GPS sensor data.

The next step in the development process will be to integrate the optical tracking results of MadMAGs as well as further gamification features into the Betaville system and to implement the revised Betaville in future use case scenario of EU-PS2 to enhance participation of people and creatives to design and develop the citizens-friendly European city of the future.