Abstract
Building a cloud geodatabase for a sponge city is crucial to integrate the geospatial information dispersed in various departments for multi-user high concurrent access and retrieval, high scalability and availability, efficient storage and management. In this study, Hadoop distributed computing framework, including Hadoop distributed file system and MapReduce (mapper and reducer), is firstly designed with a parallel computing framework to process massive spatial data. Then, access control with a series of standard application programming interfaces for different functions is designed, including spatial data storage layer, cloud geodatabase access layer, spatial data access layer and spatial data analysis layer. Subsequently, a retrieval model is designed, including direct addressing via file name, three-level concurrent retrieval and block data retrieval strategies. Main functions are realised, including real-time concurrent access, high-performance computing, communication, massive data storage, efficient retrieval and scheduling decisions on the multi-scale, multi-source and massive spatial data. Finally, the performance of Hadoop cloud geodatabases is validated and compared with that of the Oracle database. The cloud geodatabase for the sponge city can avoid redundant configuration of personnel, hardware and software, support the data transfer, model debugging and application development, and provide accurate, real-time, virtual, intelligent, reliable, elastically scalable, dynamic and on-demand cloud services of the basic and thematic geographic information for the construction and management of the sponge city.
摘要
构建海绵城市云空间数据库可以整合不同管理部门分散的地理空间信息, 实现海量数据的多用户高并发访问和检索、 高可扩展性和可用性、 高效存储和管理。 为了处理海量空间数据, 本研究首先利用并行计算技术设计了包括 Hadoop 分布式文件系统和 MapReduce 的 Hadoop 分布式计算框架。 其次, 用一系列标准 API 设计了访问控制模块, 包括空间数据存储层、 云空间数据库访问层、 空间数据访问层和空间数据分析层。 然后, 设计了一个检索模型, 包括利用文件名直接寻址、 3 层并行检索和数据块检索策略。 实现了多尺度、 多源和海量空间数据的实时并行访问, 高性能计算、 通讯、 存储、 高效检索和时序安排等功能。 最后, 通过与 Oracle 数据库的比较, 验证了 Hadoop 云空间数据库的性能。 海绵城市云空间数据库能避免人员、 硬件和软件资源的冗余配置, 支持数据传输、 模型调试和应用开发, 为海绵城市建设和管理提供基础和专题地理信息的精确、 实时、 虚拟、 智能、 可靠、 动态、 按需和弹性可扩展的云服务。
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
HOU Jing-wei, LI Long-tang, HE Jie. Detection of grapevine leafroll disease based on 11-index imagery and ant colony clustering algorithm [J]. Precision Agriculture, 2016, 17(4): 488–505. DOI: https://doi.org/10.1007/s11119-016-9432-2.
HOU Jing-wei, FAN Xin-gang, LIU Ren-tao. Optimal spatial allocation of irrigation water under uncertainty using the bilayer nested optimisation algorithm and geospatial technology [J]. International Journal of Geographical Information Science, 2016, 30(12): 2462–2485. DOI: https://doi.org/10.1080/13658816.2016.1181264.
ZHOU Xiao-guang, CHEN Jun, JIANG Jie, ZHU Jian-jun, LI Zhi-lin. Event-based incremental updating of spatiotemporal database [J]. Journal of Central South University of Technology, 2004, 11(2): 192–198.
LEI Xiang-dong, ZHAO Yue-long, CHEN Song-qiao, YUAN Xiao-li. Scheduling transactions in mobile distributed real-time database systems [J]. Journal of Central South University of Technology, 2008, 15: 545–551. DOI: https://doi.org/10.1007/s11771-008-0103-y.
HOU Jing-wei, MI Wen-bao, LI Long-tang. Spatial quality evaluation for drinking water based on GIS and ant colony clustering algorithm [J]. Journal of Central South University of Technology, 2014, 21(2): 1051–1057. DOI: https://doi.org/10.1007/s11771-014-2036-y.
JANAKIRAMAN K K, ORGUN M A, NAYAK A. Geospatial editing over a federated cloud geodatabase for the state of NSW [C]// Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems. San Jose: ACM, 2010: 144–151. DOI: https://doi.org/10.1145/1869790.1869813.
VORA A, JAIN D, MAVANI G, SHAH S. Incorporating database management system in cloud [J]. International Journal of Scientific Research and Education, 2016, 5(11): 61–64. DOI: https://doi.org/10.17148/IJARCCE.2016.51112.
LI Wei-yue, LIU Chun, HONG Yang, ZHANG Xin-hua, WAN Zhan-ming, SAHARIA M, SUN Wei-wei, YAO Dong-jing, CHEN Wen, CHEN Sheng, YANG Xiu-qin, YUE Jing. A public cloud-based China’s landslide inventory database (CsLID): Development, zone, and spatiotemporal analysis for significant historical events, 1949–2011 [J]. Journal of Mountain Science, 2016, 13(7): 1275–1285. DOI: https://doi.org/10.1007/s11629-015-3659-7.
DŐRNEMANN T, JUHNKE E, FREISLEBEN B. Ondemand resource provisioning for BPEL workflows using Amazon’s elastic computer cloud [C]// Processing of the 9th IEEE/ACM Int Symposium on Cluster Computing and the Grid. Shanghai, China: TEEE, 2009: 140–147.
KANG C. Cloud computing and its Applications in GIS [D]. Ann Arbor: Clark University, 2011.
BALKIĆ Z, ŠOŠTARIĆ D, HORVAT G. GeoHash and UUID identifier for multi-agent systems [M]//Agent and Multi-Agent Systems Technologies and Applications. Berlin, Springer, 2012, 7327: 290–298. DOI: https://doi.org/10.1007/978-3-642-30947-2_33.
ELDAWY A. MOKBEL M F. SpatialHadoop: A MapReduce framework for spatial data [C]// IEEE International Conference on Data Engineering, Seoul: IEEE, 2016: 1352–1363.
SRIVASTAVA P, BINH N T, KHARE A. Content-based image retrieval using moments [C]// Context-Aware Systems and Applications. ICCASA 2013. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Berlin: Springer, 2014, 128: 228–237. DOI: https://doi.org/10.1007/978-3-319-05939-6_23.
ZHANG Ming-jin, WANG Hui-bo, LU Yun, LI Tao, GUANG Yu-dong, LIU Chang, EDROSA E, LI Hong-tai, RISHE N. TerraFly GeoCloud: An online spatial data analysis and visualization system [J]. ACM Transactions on Embedded Computing Systems, 2010, 9(4): 1–24. DOI: https://doi.org/10.1145/2700494.
DING Zhi-ming, GUO Li-min, YANG Qi. RDB-KV: A cloud database framework for managing massive heterogeneous sensor stream data [C]// Second International Conference on Intelligent System Design and Engineering Application. Sanya: IEEE, 2011: 653–656. DOI: https://doi.org/10.1109/ISdea.2011.33.
LEE D W, LIANG S H L. Geopot: A Cloud-based geolocation data service for mobile applications [J]. International Journal of Geographical Information Science, 2011, 25(8): 1283–1301. DOI: https://doi.org/10.1080/13658816.2011.558017.
BHAT M A, SHAH R M, AHMAD B. Cloud computing: a solution to geographical information systems (GIS) [J]. International Journal on Computer Science and Engineering, 2011, 3(2): 594–600.
YANG Chao-wei, GOODCHILD M, HUANG Qun-ying, NEBERT D, RASKIN R, XU Yan, BAMBACUS M, FAY D. Spatial cloud computing: How can the geospatial sciences use and help shape cloud computing [J]. International Journal of Digital Earth, 2011, 4(4): 305–329. DOI: https://doi.org/10.1080/17538947.2011.587547.
AJI A, SUN Xi-ling, VO H, LIU Qiao-ling, LEE R, ZHANG Xiao-dong, SALTZ J, WANG Fu-sheng. Demonstration of Hadoop-GIS: A spatial data warehousing system over MapReduce [J]. Advances in Geographic Information Systems, 2013, 6(11): 528–531. DOI: https://doi.org/10.1145/2525314.2525320.
CHEN Da-lun, CHEN Rong-guo, XIE Jiong. Research of the parallel spatial database proto system based on MPP architecture [J]. Journal of Geo-information Science, 2016, 18(2): 151–159. DOI:https://doi.org/10.3724/SP.J.1047.2016.00151.
LIU Bao-ling, LI Gang, YOU Hong, SUI Ming-rui, WANG Shu-tao. Evaluation of dynamic groundwater quality simulation based on Cloud-GIS: A case study in Harbin urban area, China [J]. Water Science & Technology: Water Supply, 2014, 14(6): 1095–1103. DOI: https://doi.org/10.2166/ws.2014.070.
WEI Ling-yin, HSUA Ya-ting, PENGA Wen-chih, LEE Wang-chien. Indexing spatial data in cloud data managements [J]. Pervasive and Mobile Computing, 2014, 15(12): 48–61. DOI: https://doi.org/10.1016/j.pmcj.2013.07.001.
MATEUS R C, SIQUEIRA T L L, TIMES V C, CIFERRI R R, CIFERRI C D D A. Spatial data warehouses and spatial OLAP come towards the cloud: design and performance [J]. Distributed and Parallel Databases, 2015, 4: 1–37. DOI: https://doi.org/10.1007/s10619-015-7176-z.
KUNE R, KONUGURTHI P K, AGARWAL A, CHILLARIGE R R, BUYYA R. XHAMI–extended HDFS and MapReduce interface for big data image processing applications in cloud computing environments [J]. Software: Practice and Experience, 2017, 47: 455–472. DOI: https://doi.org/10.1002/spe.2425.
WANG Yong, LIU Zhen-ling, LIAO Hong-yan. Improving the performance of GIS polygon overlay computation with MapReduce for spatial big data processing [J]. Cluster Computing, 2015, 18(2): 507–516. DOI: https://doi.org/10.1007/s10586-015-0428-x.
YAN Ji-ning, MA Yan, WANG Li-zhe. A cloud-based remote sensing data production system [J]. Future Generation Computer Systems, 2018, 86(9): 1154–1166. DOI: https://doi.org/10.1016/j.future.2017.02.044.
LIN Feng-cheng, CHUNG Lan-kun, WANG Chun-ju. Storage and processing of massive remote sensing images using a novel cloud computing platform [J]. GIScience & Remote Sensing, 2013, 50(3): 322–336. DOI: https://doi.org/10.1080/15481603.2013.810976.
WANG Peng-yao, WANG Jian-qin, CHEN Ying. Rapid processing of remote sensing images based on cloud computing [J]. Future Generation Computer Systems, 2013, 29(8): 1963–1968. DOI: https://doi.org/10.1016/j.future.2013.05.002.
CHEN Hung-ming, CHANG Kai-chuan, LIN Tsung-hsi. A cloud-based system framework for performing online viewing, storage, and analysis on big data of massive BIMs [J]. Automation in Construction, 2016, 71: 34–48. DOI: https://doi.org/10.1016/j.autcon.2016.03.002.
CARY A, YESHA Y, ADJOUADI M, RISHE N. Leveraging cloud computing in geodatabase management [C]// IEEE International Conference on Granular Computing. San Jose: IEEE, 2010: 73–78.
CAFARELLA M, CHANG E, FIKES A, HALEVY A, HSIEH W, LERNER A, MADHAVAN J, MUTHUKRISHNAN S. Data management projects at Google [J]. ACM Signod Record, 2008, 37(1): 34–38. DOI: https://doi.org/10.1145/1374780.1374789.
ZHENG Kun, FU Yan-li. Research on vector spatial data storage schema based on Hadoop platform [J]. International Journal of Database Theory and Application, 2013, 6(5): 85–94. DOI: https://doi.org/10.14257/ijdta.2013.6.5.08.
XU Xiao-long, ZHANG Qi-tong, ZHOU Jing-lan. NC-MACPABE: Non-centered multi-authority proxy re-encryption based on CP-ABE for cloud storage systems [J]. Journal of Central South University, 2017, 24(4): 807–818. DOI: https://doi.org/10.1007/s11771-017-3483-z.
RYOTA H, KIMIO K. MapQL: Map-based Ontology Query Language [C]// DEWS2008, C3-2. 2008: 1–8. (in Japanese)
DUAN L F. The research and application of cloud database technology for remote sensing data [D]. Kaifeng: Henan University, 2014. (in Chinese)
ZHANG Xiu-ling, GAO Wu-yang, LAI Yong-jin, CHENG Yan-tao. Flatness predictive model based on T-S cloud reasoning network implemented by DSP [J]. Journal of Central South University, 2017, 24(10): 2222–2230. DOI: https://doi.org/10.1007/s11771-017-3631-5.
YAO Guang-shun, DING Yong-sheng, HAO Kuang-rong. Multi-objective workflow scheduling in cloud system based on cooperative multi-swarm optimization algorithm [J]. Journal of Central South University, 2017, 24(5): 1050–1062. DOI: https://doi.org/10.1007/s11771-017-3508-7.
VIJAYA P, RAJU G, RAY S K. Artificial neural network-based merging score for Meta search engine [J]. Journal of Central South University, 2016, 23(10): 2604–2615. DOI: https://doi.org/10.1007/s11771-016-3322-7.
Author information
Authors and Affiliations
Corresponding author
Additional information
Foundation item: Project(NZ1628) supported by the Natural Science Foundation of Ningxia, China
Rights and permissions
About this article
Cite this article
Hou, Jw., Sun, Sq., Liu, Rt. et al. Design and achievement of cloud geodatabase for a sponge city. J. Cent. South Univ. 25, 2423–2437 (2018). https://doi.org/10.1007/s11771-018-3926-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11771-018-3926-1