Keywords

1 Introduction

With the advent of the era of the cloud, the data format and size is growing at an unprecedented speed. Reasonable storage and management on growing huge amounts of data will be beneficial to provide support to industry analysis which is applied to forecast, as well as to effectively take advantage of big data under the background of opportunities and challenges, to the integrate traditional decision method and the concept of the decision method of big data, to build a platform of innovative management of computer technology for intelligent analysis prediction and evaluation of prediction. Eventually the ability of enterprise to apply new technology on management and decision making will be improved [1].

Big data is put forward for the first time in 2009, and then have found application in the field of multiple business and development, especially the mature usage on medical field. In the era of big bang data, using data service in the industry is the inevitable result of the activation of era. To use data efficiently and accurately, the premise is the efficient storage and management of data, to take appropriate data storage model according to different application requirements, so as to more efficiently real-time process and analyze data. Relevant data is the collected and under storage management for analysis and process so as to dig out the potential value of the information provided to the senior leadership for decision making judgment. In big data environment, users put forward higher request to the storage service on the availability, reliability and durability of data. In order to prevent the data from being lost or damaged and ensure data privacy, the environment of users’ storage system is essential [2]. Google’s programming model can effectively parallel-process mass data, which adopt the GFS and BigTable model for data management, the big data platform based on Hadoop implements Google’s big data storage system with source opened.

Early in the information economy, the enterprise only plays as a resource to collect and store the data with simple statistical analysis at most, while the intrinsic value of the data is usually ignored. Along with the progress of the storage and analysis technology, the enterprise further mined and processed the data collected with growing awareness of the importance of active mastery on data. Ability to develop potential value of data becomes one of the core competitiveness of enterprises. The value of data shows its important position in the era of intelligence science and technology.

Hysteresis phenomenon exists in the application of traditional data storage technology on mass data storage management and security. At present, in the continuous development of industry data, there are vast amounts of unstructured data and semi-structured data. Reasonable process and analysis on these data to unearth valuable information complied with the requirements for big data policy decisions and service provision. Data model is the precondition and foundation of big data analysis and forecast. This paper builds the big data application platform and studies the big data storage system architecture based on the Hadoop data platform.

Application platform provides industry analysis, reporting, prediction and decision making, etc. At the same time feasible technical solutions are provided in storage management for big data.

2 Big Data Platform Design

2.1 Platform Environment

How to collect and store huge amounts of data, how to integrate heterogeneous data, how to mining and process large data sets, which are of concern to the third party service provider. Data collection, data storage, data process, data analysis, data application will be the basic task of the enterprise performance in wisdom economy era. Judgment and decision based on data will become the skills and means of enterprises for development. A platform, which conform to the big data management and support the development in the field of application, is designed for the analysis of relevant industry development, integration of heterogeneous data and customized user requirements, etc. The platform is based on Hadoop big data platform architecture, relying on HDFS, MapReduce and MongoDB, etc. distributed framework which are deployed into more cheap hardware equipment, for the application with high throughput data access mechanism. HDFS as open source distributed file system supports data storage and management of high fault tolerance. HDFS is open source implementation of GFS, which can deploy to cheap PC devices and is a suitable application for big data [3]. HDFS takes a master-slave mode structure. In the cluster, there is a NameNode and multiple DataNodes, which is in charge of data storage option and namespace, database storage and optimal strategy choice [4]. MapReduce is proposed by Google to concurrent processing mass data parallel programming model. It has the characteristics of simple and efficient for the shielding mechanism of the underlying implementation details, effectively reduction on programming difficulty. Cheap machine deployment clusters can be used to achieve high performance with good scalability and provide high-efficient scheduling interface, implementing task scheduling, load balancing and fault tolerance and consistency management etc. [5–8]. MongoDB supports multiple binaries and two kinds of storage ways as storage subsystem automatically divided and user-defined divided. In order to realize load balance, MongoDB implements distributed storage in multiple servers for the same data file directed into a multiple block. There will be no necessity for the user to know where the data is stored for the servers will record the shard [9, 10].

2.2 Architecture Design

Business application requirements include data acquisition, data gathering, and data analysis and data application. A unified data application platform will be built for data real-time loading, storage and processing different types of data. Data processing tools and services are integrated for the management of heterogeneous data. Structured and unstructured data warehouse analysis tool are also integrated. The platform can implement the concentration of large data sharing and collaborative access at anytime, anywhere, by any terminal equipment; Application platform can support the modeling of new business development and business strategy, and promote the development of the industry insight, real-time early warning analysis.

Big data application platform can satisfy the processing requirements of data of large amount, multiple style and fast flow. It also possess the ability to implement huge amounts of data collection, storage, processing and analysis, meet the basic requirements of high reliability, easy extensibility, strong fault tolerance, high availability, high security and high secrecy of the enterprise application, and ensure the compatibility of existing technology with the platform and realize data storage and processing.

Big data application platform conforms to two standard systems, the system safety standard system and service management standard system. Big data platform in Hadoop and data integration platform in data warehouse implement a lot of data storage, analysis, processing and usage, including the front layer, core layer, manager layer, data layer, and application layer, etc. Office automation, risk assessment, data acquisition, smart analysis, real-time processing is integrated to create a data integration and management platform. With integration of data warehouse and analysis tools, this paper puts forward a real-time forecast analysis solution.

A connection is realized among core layer, big data platform and data warehouse integration platform, in order to realize intelligent warning and real-time analysis as well as integration of data warehouse and intelligent analysis system, using analytical tools for analyzing visualization and form the electronic report and analysis report. As shown in Fig. 1 as the big data general architecture.

Fig. 1.
figure 1

General architecture

Full size image

3 Key Technologies

3.1 Storage System Design Based on Cloud Computing

Cloud computing virtualization technology is used in the design of storage system to achieve high concurrency and high fault tolerance under the condition of the consistency. The master-slave distribution pattern is used in the design of data storage to avoid the data loss and damage caused by the outage under the traditional storage technology which adopts a single mode of storage system. System uses partitioned storage in different physical and backup storage device, So as to improve the security and integrity of data. Virtualized physical resources are integrated into the master node as the system management node, which is responsible for the management and monitoring the daily operation of the slave nodes as well as to ensure the normal state of nodes. The master node network virtualization and distributed management are put forward as the design idea to solve the problems of master node management bottleneck of the traditional technology. The master node is visited by the principle of recent-visit, to a certain extent, alleviate the access and management of the master node burden, avoid server failures caused by centralized access to the master node which will lead to the collapse of whole system, as well as to solve the problem of the bottleneck of the whole system operation, thus improve the overall efficiency. Multiple slave nodes as the data storage, achieves load balanced distribution of stored data. The difficulty of storage of different data types under the traditional relational database is solved, and multiple of backup storage of data is distributed into data nodes. The loss of data is avoided on the premise of superior system operation performance, the storage architecture design is shown in Fig. 2.

Storage system access flow begins with access request sent by client, then the message accesses near master node server through network. After received and responded to users request to read and write, the node will locate block of data in specified slave node according to the address to implement specified operation. Data is stored by partition distributed storage, convenient in management. With the use of analysis of visualization tools, electronic report is generated in the users’ terminal display, which provides convenience in analysis and decision. As long as connecting to the Internet client terminal can access the system. The terminal can be hardware, fixed and mobile devices and embedded devices, etc.

Fig. 2.
figure 2

Storage architecture

Full size image

3.2 Updating Algorithm Design Based on File Block

Store system adopts distributed master-slave mode, which automatically implement load balanced block storage. Data may be stored in different data nodes and blocks. In order to ensure transactional consistency operation when read and write updates, necessary coordination mechanism need to be taken to realize the feasible operation. Traditional technology solutions using the famous paxos algorithm, while this paper uses the Chubby coarse-grained lock service to solve consistency problem when the file block update. On one hand, the design is consistent to data set partitioned storage design patterns; on the other hand, ensure normal network traffic during system data update, to avoid network congestion phenomena caused by data update in the system. Update process is the communication between the client and the Server by RPC. Redundant technology strategy with lock service is used to ensure the consistency of data update. The server is made of five machines. The single server consists of fault tolerant database and fault tolerant log file. Communication between servers is through protocols. Snapshots are stored locally. Data in the form of file blocks is under parallel transmission and synchronization update, thus update efficiency is improved.

When data update normally, data block is updated in server, while update operation is written into the fault tolerant database and fault-tolerant log files, at the same time, the client is notified to update data. When data update abnormally, an inconsistency occurs during data block update, recovery on error data in the data block will be implemented according to the copy of the rest data block and fault tolerant database and log file, and notify the client to update the abnormal data or to ignore the update of the data. When data failure occurs, the server will revoke to normal transactions according to the log file. In order to achieve the goal that data is always up-to-date, the system will real-time test the data state, and avoid unnecessary updates take up normal network traffic, reduce client and server communication flow. Cache of log files of sever are saved by the client. When the server data update, on the premise of lock the client will update the data in the local file with synchronization, basic structure is shown in Fig. 3.

Fig. 3.
figure 3

Basic structure

Full size image

3.3 Fault Recovery Mechanism Design Based on Cloud Storage

When system failure occurs during the running of application, without timely recovery on failure site, huge economic and customer losses will be caused to the enterprise. Storage service based on cloud is a kind of network storage service, which hold a high efficient and safe design of data and fault recovery, on the one hand, the efficiency of data computing, communication and storage is improved, on the other hand, the efficiency of detection is improved as high precision in fault detection of data storage and in data recovery. Master-slave distributed design method is used for the mass data storage system, as well as master-slave mode design on fault recovery mechanism based on cloud storage. According to the global and local fault detection and recovery mechanism, the operational efficiency of the system and the accuracy of fault positioning recovery are improved and unnecessary waste of resources reduced. Data error occurrence, security and reliability are real-time detected by the integration of idle resources of network through the network communication mechanism of the cloud storage system. For data resources are in cloud storage, detect movement can be implemented while user is offline, which makes the system always in a state of the data accuracy, safety and reliability.

In this paper, the fault recovery mechanism integrates global action, distributed management and local action at an organic whole, which makes a balance of the independent operation of the master node and slave nodes. A middle tier of the proxy server is added between the master node and slave nodes so as to reduce the workload of the master node and avoid the bottleneck of system management. When data inconsistency occurs, cloud storage server will lessen the fault detected to the management master node. After receiving the fault message, through the cloud storage servers, the node send the fault report back to the proxy server to process. This move is a global action. The fault message through the external interface service is sent to a proxy server. Proxy server implements the distributed management and share part of the master node work, which is responsible for the records and addressing the fault point. After received fault message, according to the error log file, slave node starts to recover the data to normal status and update and respond to proxy server the current status of storage systems. The process is a local action. Fault recovery mechanism is shown in Fig. 4.

Fig. 4.
figure 4

Fault recovery mechanism

Full size image

4 Conclusions

The importance of the application of data for the enterprise is analyzed in this paper in the first step. To satisfy the demand of big data processing platform, data application platform in Hadoop and data integration platform in data warehouse are put forward. Cloud computing technology is adopted in the design of storage system and improves the traditional technology in the master node server management in distributed storage. Through analyzing and realizing key technologies of storage system, such as file block update strategy and fault recovery mechanisms, feasible technical solutions in storage management for big data are provided.