Parallel Implementation of the Unified Curvilinear Ocean and Atmospheric (UCOAM) Model and Supporting Computational Environment

A new parallel framework and associated cyberinfrastructure has been verified to efficiently and accurately run Unified Curvilinear Ocean and Atmospheric Model (UCOAM) simulations. UCOAM is a high-resolution (sub-km), non-hydrostatic, large eddie simulation (LES) computational fluid dynamics model that is capable of running ocean and atmospheric simulations. UCOAM is the only environmental model in existence today that uses a full 3D curvilinear coordinate system, which results in increased accuracy, resolution, and reduced times to solution. UCOAM is a petascale model: for 10 meter resolutions, array sizes will scale as theta(1012) bytes, resulting in a large on node memory footprint; meaningful model simulations will generate terabytes of data and run for long periods of time; the non-hydrostatic model will require 3D communications. A parallel framework (PFW) has been developed that is capable of distributing data and computations across arbitrary 3D processor arrangements, of managing the Arikawa-C staggered grid variables and several finite difference stencils. The framework is written in Fortran 95, is component based and modular, and is completely decoupled from the application. The PFW communication model is based on the Message Passing Interface (MPI), but can be extended to include other parallel approaches. The framework uses a simple block-block data distribution scheme and can handle arbitrary processor arrangements. The framework manages and tracks communications between arbitrary groups of processors in 1, 2, and 3 dimensions, and is aware of the locations of axial, diagonal and tri-diagonal neighbors. The application framework is based on the PFW and can support any UCOAM based application. To facilitate simulations, a computational environment (CE) based on the Cyberinfrastructure Web Application Framework (CyberWeb) has been developed to support access to and development of web services, portals, and cyberinfrastructure. The parallel framework has been used to develop several applications, and including nesting the model within the global Regional Ocean Model System. Here we present results that demonstrate scaling of the parallel and application frameworks to nearly 2000 nodes, show that the applications scale similarly to other comparable coastal ocean models, thus verifying that UCOAM can be parallelized and produce accurate results.