Distributed Signalling Analysis With Mapreduce

With the number of users and business size growth, the telecommunication companys daily product up to tens of TBs of signalling monitoring data. Traditional data processing methods, which are based on relational database, can no longer cope with such a huge amount of data. How to analyse signalling data effectively is a common problem in telecommunication industry. Hadoop includes an open source implementation of MapReduce, which can be used to build low-cost and high-performance distributed data analysis system to overcome the lack of throughput in stand-alone system. In addition, Hadoop also provides a fail-tolerant and scalable mass data storage solution.This paper describes how to achieve massive signalling data analysis on the base of MapReduce programming model. Designs a complete scheme of distributed signalling analysis, including data preprocessing, data analysis and results output. Introduce how to transform signalling analysis jobs into MapReduce tasks using HIVE. Discusses the optimization of tasks transforming and results storage. Data source comes from the Call Detail Records(CDR) generated by signalling monitoring system.A large number of jobs run in the signalling analysis system at the same time, sharing cluster resources. Task scheduling is to solve the problem of resource allocation, improving system performance. Hadoop default scheduler ignores the different needs of different operations, may cause some jobs never be able to get enough resources. This paper presents a MapReduce based Least Laxity First(LLF) scheduling algorithm, which ensures fairness in scheduling and supports preemptive.