During the review of the Data Product Development Guide (DPDG) for Data Producers (https://doi.org/10.5067/DOC/ESCO/RFC-041VERSION2) prior to its publication, Patrick Quinn had the following comment regarding the first paragraph of Section 3.1.
"May need to revisit this. I agree with recommending netCDF-4, but the netCDF-4 library alone cannot (yet) produce cloud-optimized HDF5. It might be something like 'Use the netCDF-4 library, then use h5repack'."
(This comment has been been assigned an ID Quinn-2 for tracking purposes. Note that the first paragraph of Section 3.1 states: "While several acceptable formats are listed by ESCO [25], the highly preferred format for EOSDIS data products is network Common Data Form Version 4 (netCDF-4) [26], which uses the Hierarchical Data Format Version 5 (HDF5) [27] data storage model.")
DPDG V2.0 Section 3.1
-
DPDG - ramapriyan
- Subject Matter Expert
- Posts: 18
- Joined: Fri Nov 06, 2020 12:49 pm America/New_York
DPDG V2.0 Section 3.1
Last edited by DPDG - ramapriyan on Wed Feb 05, 2025 4:07 pm America/New_York, edited 1 time in total.
Filters:
-
DPDG - ramapriyan
- Subject Matter Expert
- Posts: 18
- Joined: Fri Nov 06, 2020 12:49 pm America/New_York
Re: DPDG V2.0 Section 3.1
The DPDG V2.0 editing team agrees with Patrick Quinn's sentiments in this comment, but the text in this section was written before knowledge about how to optimize HDF5 was known or documented. How best to cloud optimize netCDF4 is still under investigation. But we do state that netCDF "Readily allows for conversion to a cloud-optimized format." because you can use h5reack on it.
The revisions in response the comment will be considered for the next version of the DPDG.
The revisions in response the comment will be considered for the next version of the DPDG.