Improvement of TOMCAT-GLOMAP File Access with User Defined MPI Datatypes

TOMCAT-GLOMAP is software that is used to model global atmospheric chemistry and aerosol processes in three dimensions. The model is written in FORTRAN and has evolved from originally running on vector machines to the current parallel code. It is widely used by the UK atmospheric science community on the Cray XE6 UK national computing resource (HECToR). HECToR uses a LUSTRE file system to support its 90112 computational cores. Recent developments in TOMCAT-GLOMAP include moving to a higher resolution atmosphere and more detailed chemistry processes making the file access more significant. This paper describes the modification of the file access patterns that occur throughout the simulation. The analysis identified subroutines where the workload was not actually the accessing of the data but the processing of data either before writing it or after reading it. The main gains in the project have been achieved by reducing the load on MPI task zero. The replacement methods include properly distributing the work such that local data is processed before any gather and subsequent writing. Also, after reading data, it is distributed out to other MPI tasks before it is processed. The overheads of some of the gathering category of subroutines have been reduced. For example, reporting on altitude data at specific locations. In the case using 80 MPI tasks, for the original coding the time per iteration is 8.9s on the writer iteration compared to 0.98s of a non-writing iteration. The time per writing iteration for the new method is 0.99s. When more MPI tasks were applied the overhead for the original method became more significant (e.g. with 400 MPI tasks the time per writing iteration is 38s compared to the time of 0.42s per non-writing iteration). In this case the time per iteration for the new method is 0.47s. These changes allow the higher resolution to be simulated more efficiently and to be more flexible within research work. Researchers can now choose to do more within an experiment without the overhead previously incurred when activating the functions that have been modified by this project.