openPMD – the Open Standard for Particle-Mesh Data

The Open Standard for Particle-Mesh Data (openPMD) is a F.A.I.R. metadata standard for tabular (particle/dataframe) and structured mesh data in science and engineering.
We show the basic components of openPMD, its extensions to specific domains, applications from laser-plasma physics, particle accelerators, material physics to imaging and the ability to bridge multiple heterogeneous scientific models with a commonly-understood markup.

The openPMD-api builds upon established portable I/O formats such as HDF5 and ADIOS2, enabling workflows that scale from single-user computers up to Exascale simulations, in-transit data processing, 3D visualization, GPU-accelerated data analytics and AI/ML. openPMD links into the existing ecosystems of its scalable I/O backends and extends them with tooling that understands the openPMD data markup.
An overview over the openPMD ecosystem and community is shown.

Attention is given to recent developments in openPMD that interplay with HDF5, including mesh refinement and the Helmholtz Metadata Collaboration's HELPMI project which aims for an easier integration of openPMD with other HDF5-based standards, this way bringing openPMD closer to experiment workflows.

References:

[1] Axel Huebl, Remi Lehe, Jean-Luc Vay, David P. Grote, Ivo F. Sbalzarini, Stephan Kuschel, David Sagan, Christopher Mayes, Frederic Perez, Fabian Koller, and Michael Bussmann. “openPMD: A meta data standard for particle and mesh based data,” DOI:10.5281/zenodo.591699 (2015)
[2] Homepage: https://www.openPMD.org
[3] GitHub Organization: https://github.com/openPMD
[4] Projects using openPMD: https://github.com/openPMD/openPMD-projects
[4] Reference API implementation: Axel Huebl, Franz Poeschel, Fabian Koller, and Junmin Gu. “openPMD-api 0.14.3: C++ & Python API for Scientific I/O with openPMD,” DOI:10.14278/rodare.1234 (2021)
https://openpmd-api.readthedocs.io
[5] Selected earlier presentations on openPMD:
https://zenodo.org/search?page=1&size=20&q=openPMD&type=presentation
[6] Axel Huebl, Rene Widera, Felix Schmitt, Alexander Matthes, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, and Michael Bussmann. “On the Scalability of Data Reduction Techniques in Current and Upcoming HPC Systems from an Application Perspective,” ISC High Performance 2017: High Performance Computing, pp. 15-29, 2017. arXiv:1706.00522, DOI:10.1007/978-3-319-67630-2_2
[7] Franz Poeschel, Juncheng E, William F. Godoy, Norbert Podhorszki, Scott Klasky, Greg Eisenhauer, Philip E. Davis, Lipeng Wan, Ana Gainaru, Junmin Gu, Fabian Koller, Rene Widera, Michael Bussmann, and Axel Huebl. Transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2, Part of Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation, SMC 2021, Communications in Computer and Information Science (CCIS), vol 1512, 2022. arXiv:2107.06108, DOI:10.1007/978-3-030-96498-6_6
[8] The Helmholtz Metadata Collaboration's ongoing HELPMI project: https://helmholtz-metadaten.de/de/inf-projects/helpmi-helmholtz-laser-plasma-metadata-initiative