Abstract
Image compositing is widely used to combine visual elements from separate source images into a single image. Although recent image compositing techniques are capable of achieving smooth blending of the visual elements from different sources, most of them implicitly assume the source images are taken in the same viewpoint. In this paper, we present an approach to compositing novel image objects from multiple source images which have different viewpoints. Our key idea is to construct 3D proxies for meaningful components of the source image objects, and use these 3D component proxies to warp and seamlessly merge components together in the same viewpoint. To realize this idea, we introduce a coordinate-frame based single-view camera calibration algorithm to handle general types of image objects, a structure-aware cuboid optimization algorithm to get the cuboid proxies for image object components with correct structure relationship, and finally a 3D-proxy transformation guided image warping algorithm to stitch object components. We further describe a novel application based on this compositing approach to automatically synthesize a large number of image objects from a set of exemplars. Experimental results show that our compositing approach can be applied to a variety of image objects, such as chairs, cups, lamps, and robots, and the synthesis application can create novel image objects with significant shape and style variations from a small set of exemplars.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Perez P, Gangnet M, Blake A. Poisson image editing. ACM Transactions on Graphics, 2003, 22(3): 313–318.
Jia J, Sun J, Tang C K, Shum H Y. Drag-and-drop pasting. ACM Transactions on Graphics, 2006, 25(3): 631–637.
Farbman Z, Hoffer G, Lipman Y, Cohen-Or D, Lischinski D. Coordinates for instant image cloning. ACM Transactions on Graphics, 2009, 28(3): Article No. 67.
Tao MW, Johnson M K, Paris S. Error-tolerant image compositing. In Proc. the 11th European Conference on Computer Vision, Sept. 2010, pp.31-44.
Sunkavalli K, Johnson M K, Matusik W, Pfister H. Multiscale image harmonization. ACM Transactions on Graphics, 2010, 29(4): Article No. 125.
Agarwala A, Dontcheva M, Agrawala M, Drucker S, Colburn A, Curless B, Salesin D, Cohen M. Interactive digital photomontage. ACM Transactions on Graphics, 2004, 23(3): 294–302.
Rother C, Kumar S, Kolmogorov V, Blake A. Digital tapestry [automatic image synthesis]. In Proc. IEEE CVPR, June 2005, pp.589-596.
Rother C, Bordeaux L, Hamadi Y, Blake A. AutoCollage. ACM Transactions on Graphics, 2006, 25(3): 847–852.
Wang J, Quan L, Sun J, Tang X, Shum H Y. Picture collage. In Proc. IEEE CVPR, June 2006, pp.347-354.
Chen T, Cheng M M, Tan P, Shamir A, Hu S M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics, 2009, 28(5): 124:1–124:10.
Eitz M, Richter R, Hildebrand K, Boubekeur T, Alexa M. Photosketcher: Interactive sketch-based image synthesis. IEEE Computer Graphics and Applications, 2011, 31(6): 56–66.
[12] Kalogerakis E, Chaudhuri S, Koller D, Koltun V. A probabilistic model for component-based shape synthesis. ACM Trans. Graph., 2012, 31(4): 55:1–55:11.
Xu K, Zhang H, Cohen-Or D, Chen B. Fit and diverse: Set evolution for inspiring 3D shape galleries. ACM Trans. Graph., 2012, 31(4): 57:1–57:10.
Burt P J, Adelson E H. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 1983, 2(4): 217–236.
Ogden J M, Adelson E H, Bergen J R, Burt P J. Pyramidbased computer graphics. RCA Engineer, 1985, 30(5): 4–15.
Porter T, Duff T. Compositing digital images. ACM SIGGRAPH Comput. Graph., 1984, 18(3): 253–259.
Xue S, Agarwala A, Dorsey J, Rushmeier H. Understanding and improving the realism of image composites. ACM Transactions on Graphics, 2012, 31(4): Article No. 84.
Diakopoulos N, Essa I, Jain R. Content based image synthesis. In Proc. the 3rd CIVR, July 2004, pp.299-307.
Johnson M, Brostow G J, Shotton J et al. Semantic photo synthesis. Computer Graphics Forum, 2006, 25(3): 407–413.
Lalonde J F, Hoiem D, Efros A A, Rother C, Winn J, Criminisi A. Photo clip art. ACM Transactions on Graphics, 2007, 26(3): Article No. 3.
Hall P, Cai H, Wu Q, Corradi T. Cross-depiction problem: Recognition and synthesis of photographs and artwork. Computational Visual Media, 2015, 1(2): 91–103.
Huang H, Zhang L, Zhang H C. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): Article No. 155.
Yu Z, Lu L, Guo Y, Fan R, Liu M, Wang W. Content-aware photo collage using circle packing. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(2): 182-195.
Risser E, Han C, Dahyot R, Grinspun E. Synthesizing structured image hybrids. ACM Transactions on Graphics, 2010, 29(4): Article No. 85.
Carroll R, Agarwala A, Agrawala M. Image warps for artistic perspective manipulation. ACM Transactions on Graphics, 2010, 29(4): Article No. 127.
Zheng Y, Chen X, Cheng M M, Zhou K, Hu S M, Mitra N J. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph., 2012, 31(4): 99:1–99:11.
Chen T, Zhu Z, Shamir A, Hu S M, Cohen-Or D. 3-sweep: Extracting editable objects from a single photo. ACM Transactions on Graphics, 2013, 32(6): Article No. 195.
Miao Y, Hu F, Zhang X, Chen J, Pajarola R. SymmSketch: Creating symmetric 3D free-form shapes from 2D sketches. Computational Visual Media, 2015, 1(1): 3–16.
Funkhouser T, Kazhdan M, Shilane P, Min P, KieferW, Tal A, Rusinkiewicz S, Dobkin D. Modeling by example. ACM Trans. Graph., 2004, 23(3): 652–663.
Shin H, Igarashi T. Magic canvas: Interactive design of a 3-D scene prototype from freehand sketches. In Proc. Graphics Interface, May 2007, pp.63-70.
Lee J, Funkhouser T. Sketch-based search and composition of 3D models. In Proc. the 5th SBM, June 2008, pp.97-104.
Xu K, Chen K, Fu H, Sun W L, Hu S M. Sketch2Scene: Sketch-based co-retrieval and co-placement of 3D models. ACM Transactions on Graphics, 2013, 32(4): Article No. 123.
Kreavoy V, Julius D, Sheffer A. Model composition from interchangeable components. In Proc. the 15th PG, Oct. 2007, pp.129-138.
Chaudhuri S, Koltun V. Data-driven suggestions for creativity support in 3D modeling. ACM Trans. Graph., 2010, 29(6): 183:1–183:10.
Chaudhuri S, Kalogerakis E, Guibas L, Koltun V. Probabilistic reasoning for assembly-based 3D modeling. ACM Trans. Graph., 2011, 30(4): 35:1–35:10.
Li Y, Sun J, Tang C K, Shum H Y. Lazy snapping. ACM Transactions on Graphics, 2004, 23(3): 303–308.
Russell B C, Torralba A, Murphy K P, Freeman W T. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 2008, 77(1/2/3): 157–173.
Barnes C, Shechtman E, Finkelstein A, Goldman D B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24:1–24:11.
Criminisi A, Reid I, Zisserman A. Single view metrology. International Journal of Computer Vision, 2000, 40(2): 123-148.
Sinha S N, Steedly D, Szeliski R, Agrawala M, Pollefeys M. Interactive 3D architectural modeling from unordered photo collections. ACM Transactions on Graphics, 2008, 27(5): 159:1–159:10.
Wilczkowiak M, Sturm P, Boyer E. Using geometric constraints through parallelepipeds for calibration and 3D modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(2): 194–207.
Jiang N, Tan P, Cheong L F. Symmetric architecture modeling with a single image. ACM Transactions on Graphics, 2009, 28(5): 113:1–113:8.
Shen C H, Fu H, Chen K, Hu S M. Structure recovery by part assembly. ACM Transactions on Graphics, 2012, 31(6): Article No. 180.
O’Donovan P, Agarwala A, Hertzmann A. Color compatibility from large datasets. ACM Transactions on Graphics, 2011, 30(4): Article No. 63.
Yu L F, Yeung S K, Terzopoulos D, Chan T F. DressUp!: Outfit synthesis through automatic optimization. ACM Transactions on Graphics, 2012, 31(6): 134:1–134:14.
Xu K, Zheng H, Zhang H, Cohen-Or D, Liu L, Xiong Y. Photo-inspired model-driven 3D object modeling. ACM Trans. Graph., 2011, 30(4): 80:1–80:10.
Cootes T F, Taylor C J, Cooper D H, Graham J et al. Active shape models — Their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38–59.
Schwarz G. Estimating the dimension of a model. The Annals of Statistics, 1978, 6(2): 461–464.
Author information
Authors and Affiliations
Corresponding author
Additional information
Special Section of CVM 2016
This work is partially supported by the National Natural Science Foundation of China under Grant Nos. 61272305, 61303136, 61272392, and 61322204, and the National Program for Special Support of Eminent Professionals of China.
Rights and permissions
About this article
Cite this article
Chen, X., Xu, WW., Yeung, SK. et al. View-Aware Image Object Compositing and Synthesis from Multiple Sources. J. Comput. Sci. Technol. 31, 463–478 (2016). https://doi.org/10.1007/s11390-016-1640-8
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-016-1640-8