基于HBase的交通流数据实时存储与查询优化方案的设计与实现

发布时间：2018-03-11 15:22

本文选题：HBase　切入点：智能交通　出处：《江苏大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着经济的飞速发展,互联网存储技术日益成熟,智能交通领域得到空前进步,以海量交通流数据为基础的智能交通技术得到业界广泛关注。交通数据采集技术迅速完善,采集手段多样化,数据量激增。我国沿海地区一个中型规模城市每天产生数千万条交通流数据,每年汇集的数据量达到数百TB级。面对如此庞大的数据量,必须对其所搭载的系统进行优化,以适应实际应用需求。Hadoop作为较成熟的海量数据处理计算框架,其核心HDFS和MapReduce为用户提供了高效的数据存储能力和分布式计算模型。HBase分布式数据库采用Hadoop的分布式文件系统实现存储功能,支持Hadoop并行计算框架,使用HBase作为海量数据的存储媒介,具有更稳定的可靠性和数据检索能力。为解决传统关系型数据库进行海量数据读写时普遍性能低下的缺陷,本文致力于研究基于HBase的交通流数据实时存储与查询优化方案,实现交通工程中对数据的实时处理要求,主要内容和创新点如下:(1)本文以道路车辆信息数据为模型,分别针对数据存储和查询两个方面,设计出一种基于HBase的集群架构模型和优化方案。(2)数据存储方面,本文分析交通流数据特征设计了HBase复合主键存储模型。首先介绍了一种依据数据特征社群进行分片的Region预分区策略用以解决Region拆分导致的数据“热点”问题。其次,为了解决集群节点变更导致的数据丢失问题,提出了一种基于散列技术和一致性哈希算法的存储调度算法。然后,对于数据缓冲和数据写入给出具体的实现方案,最后,本文从数据写入性能和缓冲队列写入阈值测试两个方面进行实验,验证本优化方案中数据查询模块相比现有数据存储模块具有更好的性能。(3)数据查询方面,本文利用Redis分布式集群服务器和本地磁盘设计了多级缓存策略并给出了实现方案。本文首先提出了一种Redis分布式缓存服务器系统架构,设计一种缓存记录值存储模型。然后,根据访问频率的不同引入热度值的概念,设计出一种基于热度积累的缓存淘汰算法。最后,本文从数据读取效率和缓存淘汰策略对比两个方面进行实验,验证使用当前优化方案的交通流数据查询相比现有查询方案具有更高的查询效率。
[Abstract]:With the rapid development of economy, the technology of Internet storage is maturing day by day, and the field of intelligent transportation has made unprecedented progress. The intelligent transportation technology based on massive traffic flow data has been widely concerned by the industry. A medium-sized city in the coastal area of our country produces tens of millions of traffic flow data every day, and the amount of data collected each year amounts to hundreds of terabytes. Faced with such a large amount of data, The system must be optimized to meet the practical application requirements. Hadoop is a mature computing framework for mass data processing. Its core HDFS and MapReduce provide users with efficient data storage capability and distributed computing model. HBase distributed database uses distributed file system of Hadoop to realize storage function and supports Hadoop parallel computing framework. Using HBase as the storage medium of mass data, it has more stable reliability and data retrieval ability. This paper is devoted to study the real-time storage and query optimization scheme of traffic flow data based on HBase, and realize the real-time data processing requirements in traffic engineering. The main contents and innovations are as follows: 1) this paper takes the road vehicle information data as the model. Aiming at data storage and query, this paper designs a cluster architecture model based on HBase and its optimization scheme. In this paper, the HBase composite primary key storage model is designed by analyzing the traffic flow data features. Firstly, a Region prepartitioning strategy based on the data feature community is introduced to solve the data "hot spot" problem caused by the Region split. In order to solve the problem of data loss caused by the change of cluster nodes, a storage scheduling algorithm based on hash technology and consistent hash algorithm is proposed. In this paper, the performance of data writing and the threshold test of buffer queue writing are tested to verify that the data query module in this optimization scheme has better performance than the existing data storage module. In this paper, we design a multi-level cache strategy using Redis distributed cluster server and local disk. Firstly, we propose a Redis distributed cache server architecture, and design a cache record value storage model. According to the different access frequency, the concept of calorific value is introduced, and a cache elimination algorithm based on heat accumulation is designed. Finally, the experiment is carried out from two aspects: data reading efficiency and cache elimination strategy comparison. Verify that the traffic flow data query using the current optimization scheme is more efficient than the existing query scheme.
【学位授予单位】：江苏大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：U495;TP311.13

【参考文献】