Effective wide-area network performance monitoring and diagnosis from end systems

My dissertation research developed a light-weight active monitoring system to monitor, diagnose and react to network disruptions by purely using end hosts, which can help customers assess the compliance of their service-level agreements (SLAs). This thesis studies research problems from three indispensable aspects: efficient monitoring, accurate diagnosis, and effective mitigation. This is an essential step towards accountability and fairness on the Internet.To fully understand the limitation of relying on ISP data, I study and demonstrate the monitor selection's great impact on the monitoring quality and the interpretation of the results. In particular, this work analyzes the impact of public route monitoring system constraints on various research work in the Internet routing community.End-host based approach usually lacks of accuracy for diagnosing the causes of the disruptions because of the limited view from each end host. To overcome such limitations, this thesis demonstrates two techniques to diagnose two types of fine-grained causes accurately and scalably by exploring information across routing and data planes, as well as sharing information among multiple locations collaboratively.The first type of disruptions to diagnose is routing-induced performance degradation. The diagnosis component builds on the measurement results obtained. Correlating the traces from multiple locations, we design an inference algorithm to identify the minimum set of root causes that can explain most observations using a greedy algorithm.Complementary to the above system, we developed NVLens, a monitoring system to detect general performance differentiation caused by ISP policies, e.g. slowing down peer-to-peer traffic. It is motivated by the topic of "net neutrality" which has become a critical social and technical problem, as ISPs may use different ways to discriminate traffic, e.g., giving peer-to-peer application traffic lower priority, providing different qualities of service based on customer identities. In order to detect the discrimination, NVLens is designed with accurate and efficient method to monitor the service level agreement (SLA) compliance. NVLens uses novel application content-aware probing techniques to monitor the service provided for diverse applications and customers. This helps detect traffic discrimination and ensures fairness of the Internet.Finally, we demonstrate usefulness of the monitoring and diagnosis results with two mitigation applications. The first application is short-term prevention of avoiding choosing the problematic route by exploring the predictability from history. I built an efficient framework to capture coarse-grained but important performance degradation as a result of BOP routing events using light-weight probing. The framework is deployed across six vantage points and found that the data plane experienced serious performance degradation in the form of reachability loss and forwarding loops following a significant fraction of routing changes. Based on the stationarity of the correlation between routing changes and the data plane performance, we developed a model to accurately predict the severity of the impact due to routing changes. Such a model is directly helpful for making informed decisions for improved routing schemes such as overlay routing and backup path selection.In the second application, I propose the first system that can scalably compare multiple ISPs across four important performance metrics, namely reachability, loss rate, latency, and path diversity completely from end systems without any ISP cooperation, providing both short-term and long-term analysis. Cross ISP and cross metric comparisons enable more powerful decision analysis for selecting ISPs. By correlating these two metrics with latency and path diversity across multiple vantage points, we have this unique opportunity to comprehensively understand the detailed differences among ISPs. (Abstract shortened by UMI.)...