Bulut, Serdar and Wright, Steven A. orcid.org/0000-0001-7133-8533 (2023) Optimizing Write Performance for Checkpointing to Parallel File Systems Using LSM-Trees. In: SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis. ACM , pp. 492-501.
Abstract
The widening gap between compute performance and I/O performance on modern HPC systems means that writing checkpoints to a parallel file system for fault tolerance is fast becoming a bottleneck to high-performance. It is therefore vital that software is engineered such that it can achieve the highest proportion of available performance on the underlying hardware; and this is a burden often carried by I/O middleware libraries. In this paper, we outline such an I/O library based on a Log-structured Merge Tree (LSM-Tree), not just for metadata, but also scientific data. We benchmark its performance using the IOR benchmark, demonstrating 2.4 to 76.7x better performance than alternative file formats, such as ADIOS2, HDF5, and IOR baseline when running on a Lustre Parallel File System. We further demonstrate that when our LSM-Tree I/O library is used as a storage layer for ADIOS2, the resulting I/O library still outperforms the default ADIOS2 implementation by 1.5x.
Metadata
Item Type: | Proceedings Paper |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2023 Copyright held by the owner/author(s). This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy. |
Dates: |
|
Institution: | The University of York |
Academic Units: | The University of York > Faculty of Sciences (York) > Computer Science (York) |
Depositing User: | Pure (York) |
Date Deposited: | 06 Oct 2023 15:40 |
Last Modified: | 17 Dec 2024 00:34 |
Published Version: | https://doi.org/10.1145/3624062.3624118 |
Status: | Published |
Publisher: | ACM |
Identification Number: | 10.1145/3624062.3624118 |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:204040 |