Control and Inspection of Distributed Process Groups at Extreme Scale via Group File Semantics

Tools and middleware are crucial to the effective use of large distributed systems. Middleware enables efficient utilization of resources, and tools help to diagnose and fix problems in distributed programs. A common requirement among tools and middleware is operating on groups of distributed processes and files, but prior work has failed to provide a solution that both addresses the key scalability barriers and is easy to use within new and existing software. In this dissertation, we present our four-part solution to these problems.;First, we introduce a new programming idiom, group file operations , that eliminates iteration and includes explicit data aggregation. The idiom extends familiar POSIX I/O operations and provides intuitive semantics, which eases its adoption. The idiom frees developers from considerations about how to access distributed resources or parallelize group operations. Thus, developers can focus on devising scalable data aggregations that reduce centralized analysis.;Second, we present a flexible and efficient way to construct file groups using global name space composition. Users specify how independent, distributed name spaces are combined to form a global name space. Composition of hierarchical name spaces is modeled as intuitive tree operations. A merge operation over many trees provides key semantics for efficient composition of directories that define group membership.;Third, we design the TBON File System that provides scalable group file operations and global name space composition. TBON-FS leverages a tree-based overlay network to provide logarithmic scaling for multicast communication with remote servers and distributed aggregation of data.;Fourth, we present proc++, a synthetic file system for control and inspection of process and thread groups. proc++ improves upon existing process control interfaces by exposing new abstractions that hide context-sensitive information and reduce tool interactions with the operating system.;Together, these parts form a scalable platform for group operations on distributed processes and files. We evaluate the utility and performance of this platform in a suite of new tools and two widely-used software packages. Our evaluations show the idiom is easy to use and our platform provides excellent scalability for both global name space composition and group file operations.