一种海量小文件存储系统中通信模块的设计与实现

发布时间：2018-04-04 20:20

  本文选题：分布式文件系统　切入点：海量小文件　出处：《南京大学》2017年硕士论文

【摘要】：近年来,电信运营商传统业务受到了互联网浪潮的猛烈冲击,运营商不得不寻求转型,以适应互联网的发展潮流。运营商在发展动漫基地和音乐基地业务时,随着业务的发展,产生的数据日益增多,产生了海量的小文件,这使得传统文件系统面临巨大的挑战,因此解决海量小文件存储的问题具有非常重要的意义。目前存在的分布式文件系统可以实现海量文件的存储,但是同时也存在一些痛点。首先,现存的分布式文件系统主要提供的核心功能是文件的上传、下载、更新和删除。其次,目前的分布式文件系统存在的不足主要是无法对海量小文件的元数据进行有效的管理以及系统架构过于复杂。电信运营商需要一个能够拥有核心功能,又能克服这些不足的文件存储系统。本论文设计实现了统一的文件存储系统,该系统主要为电信运营商的动漫基地和音乐基地业务提供数据存储服务。整个系统主要由MetadataServer、BlockServer及通信模块组成。系统通过使用Netty框架为业务提供高并发、高可用的存储访问服务;通过文件多副本存储,系统在提供良好读写性能的基础上,也提供了高可用性的保障;通过将元数据管理和存储数据访问分离,来减小元数据管理机器的压力,提升元数据管理效率;为了减少inode资源的占用,系统采用小文件合并的方式,大大减少了存储文件所需要的inode资源,使得元数据信息可以全部缓存到内存中,提升存储效率和访问效率。通信模块为整个系统提供高性能的通信服务,通信模块包括内部通信和外部通信。内部通信为节点之间提供稳定的通信服务,外部通信使得用户通过HTTP和HTTPS协议访问系统服务。通过本文设计的海量小文件系统,用户可以有效的对海量小文件进行存储及管理。目前整个系统已经上线,为电信运营商提供了高效、可靠的数据存储服务。
[Abstract]:In recent years, the traditional business of telecom operators has been severely impacted by the tide of the Internet, operators have to seek transformation to adapt to the trend of development of the Internet.Operators in the development of animation base and music base business, with the development of business, the data generated by the increasing, produced a large number of small files, which makes the traditional file system is facing a huge challenge.Therefore, it is of great significance to solve the problem of storing large amounts of small files.The existing distributed file systems can store large amount of files, but there are also some pain points.First of all, the core functions of the existing distributed file system are file upload, download, update and delete.Secondly, the shortcomings of current distributed file systems are that they can not effectively manage the metadata of a large number of small files and the system architecture is too complex.Telecom operators need a file storage system that has core functions and can overcome these shortcomings.This paper designs and implements a unified file storage system, which mainly provides data storage services for the business of cartoon and music base of telecom operators.The whole system is mainly composed of Metadata Server Block Server and communication module.The system provides high concurrency and high availability storage access services for business by using Netty framework, and provides high availability on the basis of providing good read and write performance through file multiple replica storage.In order to reduce the pressure of metadata management machine and improve the efficiency of metadata management, the system adopts the method of small file merging in order to reduce the occupation of inode resources.The inode resources needed to store files are greatly reduced, so that metadata information can be completely cached into memory, thus improving storage efficiency and access efficiency.The communication module provides high performance communication service for the whole system. The communication module includes internal communication and external communication.Internal communication provides stable communication services between nodes. External communication enables users to access system services via HTTP and HTTPS protocols.Through the massive file system designed in this paper, users can effectively store and manage the massive small files.At present, the whole system has been online, providing efficient and reliable data storage service for telecom operators.
【学位授予单位】：南京大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333
，

本文编号：1711533