Keywords

1 Introduction

Recently, as media such as SNS, smart phone, tablet, and Internet TV are developed, it is not aimed at people who are providing content, but it is moving towards providing personal goal [1]. In addition, with the development of real-time streaming protocols, more and more producers are increasingly trying to provide content for individuals or a small number of people [2]. In addition to creating and delivering content that meets the public’s needs, you need to move to a custom content creation system that can reflect your ideas. However, most commercial software used in film, broadcast, and commercial production is well suited to those who have acquired specialized skills. Although commercial programs can easily display subtitles or images, blending between objects, faces, and backgrounds requires a lot of manipulation to naturally synthesize. Common commercial programs have the following effects.

Motion tracking technology to extract motion [3], matte choker effect [3] to soften blending boundaries, and curve filter to obtain similar colors [3]. To get natural shadows there is Drop Shadow effect [3]. In this paper, we propose a method to generate a similar effect through one or two of the above effects. And we have designed and implemented a simpler video playback system in a suggested way so that end users can more easily create custom video content.

1.1 Video Regeneration System Overview

In this paper, we propose a video reconstruction system based on original source video created by content providers and image extracted from candidate image obtained through tag-based searching. And a new video is finally generated through a combining process of correcting them. The proposed system consists of three steps as shown in Fig. 1. First, a video classifier which distinguishes the frames to will be blended in the original video source. Second, an image search extractor that search a video to be blend and extracts a necessary image from the video. Finally, video blender that regenerates a new video blended with original frames and the image through frame comparison and correction. In addition, the video blender provides three main functions: object blending for inserting a new object into the original video frames, face blending for replacing original face from original frames with the new face from the extracted image, and background blending for replacing the background of the original video with extracted image.

Fig. 1.
figure 1

System overview

Full size image

2 Design and Its Implementation

In this chapter, we will explain the 3 stages of video regeneration system step by step. In addition, it also refers to the techniques to blend objects, faces, and backgrounds, respectively.

2.1 Video Classifier

In Video Classifier, first, it splits reference sections have frames composed of similar images from an original source video. Second, based on scene change, it is indexed as the candidate frames from scene change section. Third, it selects the changing section include the background range or objects of interest, and then, Lastly, extracts frames will be changed from the reference section (Fig. 2).

Fig. 2.
figure 2

Video classifier

Full size image

2.2 Image Search Extractor

In Image Search Extractor, the user input text and tags of various videos stored in the candidate video storage are compared with each other, and candidate videos are extracted first. Then, the user selects a desired frame from the video, and extracts a target image for blending the original source video (Fig. 3).

Fig. 3.
figure 3

Image search and extractor

Full size image

2.3 Video Blender - Object Blending

Video blender includes three effects; object blending, face blending and background blending. Here, an Object refers to an object other than a face, a background, and the like among the objects constituting the video. The result should show the desired object in the video. The manipulation is as follows. First, a video and an image are extracted through the above steps. Second, the image is placed on the blending position of the video and rescaled by user interaction. After that, the Poisson Image Matting technique was applied to blend objects automatically, and the boundary was processed smoothly and naturally [4].

2.4 Video Blender - Face Blending

Face blending should automatically extract a face from the original image and replace it to another face image extracted from above steps. The manipulation is as follows. First, a video and an image are extracted through the above steps. Then, the extracted image is mapped to the 3D face model. After that, the system automatically employs Haar-like Feature algorithm to automatically detect the position, angle and size of the face in the original video [5], and it replace the face with the 3D face model and blending them.

2.5 Video Blender - Background Blending

Background blending separates background and foreground from the original image, and then remove the background, finally blend the extracted background to the original video instead of the existing background. The manipulation is as follows. First, draw two lines to separate the background and the foreground. After that, it automatically employs Watershed Segmentation technique to segment foreground and background to each region [6]. The background region is removed, and then original background image is replaced with another extracted background area (Fig. 4).

Fig. 4.
figure 4

Background blending

Full size image

3 Experimental Results

To verify the quality of the proposed system, we compared the results with Adobe After Effects, one of the commercial programs. Figure 5(a) shows object blending result of our system, and Fig. 5(b) represents object blending of the after effect. Proposed system can obtain similar result by After Effect employing Matte Choker Effect and Curve Filtering. However, it cannot give shadow effects. Figure 5(c) and (d) are the background blending of our system and After Effect, respectively. Figure 5(e) is our face blending, and Fig. 5(f) shows face blending by After Effects. Figure 5(e)-1, (f)-1 are extracted face images, and (e)-2 and (f)-2 are a frame of original video. (e)-3, (e)-4, (f)-3 and (f)-4 represent result of image that replacing original face with extracted face image. As shown in Fig. 5, (f) face blending extracts the movement of the face of the original video through Motion Tracking and employ the new face image the movement. However, Fig. 5(e) our face blender can extract movements automatically by Haar-like Feature algorithm, but it cannot be used when there are more than two faces. As shown in Fig. 5, the results of the commercial system are largely different in quality depending on the user’s skill. On the other hand, this system has confirmed that the user’s ability has little effect on the quality of the result, but the similar effect to the commercial system with little manipulation.

Fig. 5.
figure 5

(a) Object blending of proposed system, (b) object blending of the after effect, (c) background blending of proposed system, (d) background blending of the after effect, (e) face blending of our system, (f) face blending of the after effect

Full size image

4 Concluding Remarks

The number of customized video content producers has increased. However, current commercial systems are inconvenient to use for unskilled people. Because there are many manipulations to obtain natural results in video blending. To satisfy the needs of the individual video creating, we propose a video regeneration system that can produce similar effects to commercial system while minimizing manipulation. The blending boundary automatically have softened by the Poisson image matting technique. In the case of face blending, the movement of the face in an original video automatically have tracked through the Haar-like feature algorithm. In case of background blending, we could distinguish foreground and background area by simple operation through Watershed Segmentation. We think that our system can be helpful for individual content producers.