Protecting server programs and systems: Privilege separation, attack surface reduction, and risk assessment

In today's digitized world, server programs and systems have become an indispensable part of people's daily life and business, such as Web service, file service, database, etc. In the meanwhile, server programs and systems have been attracting more and more attacks and threats, resulting in the reality that they are constantly being targeted and compromised. Besides, the associated impact is becoming larger and larger, ranging from millions of stolen credit card numbers to innumerous Web servers vulnerable and waiting for an emergency security patch.;In this dissertation, we perform a three-dimensional research study emphasizing on protecting server programs and systems, including privilege separation, attack surface reduction, and risk assessment.;First, we explore applying privilege separation to enhance the security of server programs. We design and implement Arbiter, a runtime system targeting at fine-grained privilege separation in multithreaded server programs. In Arbiter, different principal threads can have different privileges to access shared data objects so that the compromise or malfunction of one thread does not lead to data contamination or data leakage of another thread. We leverage page table protection bits and devise a new memory allocation mechanism to achieve efficient reference monitoring. Programmers specify security policy through annotating the source code.;Second, reducing attack surface is an effective preventive measure to strengthen security in large-scale server systems. We propose an automated approach to accurately detect the idling (most likely unused) services and provide ways to reduce their attack surface. We implement this idea and deploy our system in a real working environment of a mid-sized enterprise to identify and constrain unused services that expose attack surface.;Finally, given a server program or system, it is important to evaluate the effectiveness of different security settings and understand the security risks of potential vulnerabilities. We study an emergent type of vulnerability, namely buffer over-read vulnerability, and propose a systematic methodology to model buffer over-read vulnerabilities and quantitatively measure the potential amount of information leakage.