Page 1 of 1

MYD03.061 CMR entry changed, but the file didn't

Posted: Tue Aug 27, 2024 7:44 pm America/New_York
by earthengine_urs
Hi,

We just noticed that some MYD03.061 entries like MYD03.A2010123.2320.061.2018059034522.hdf have new values of the "updated" field in CMR.

Eg, https://cmr.earthdata.nasa.gov/search/granules.json?collection_concept_id=C1379841358-LAADS&temporal=2010-05-03T23:20:00 shows the "updated" field with the value of 2024-08-22, but in an earlier scan we saw the same filename with the "updated" field of 2022-12-10.

The actual file's checksum didn't change compared to the version we downloaded earlier, so the change in the "updated" field feels unwarranted. But if this is an expected situation, we can adjust our download procedure.

Advice appreciated!

Thanks,
Simon

Re: MYD03.061 CMR entry changed, but the file didn't

Posted: Wed Aug 28, 2024 6:55 am America/New_York
by LAADS-EDL_UserServices_M
Yes, CMR records for all LAADS data products have been updated starting August 1, 2024. The update is due to changing data access URL from on-premise to Cloud, see announcement at:
https://ladsweb.modaps.eosdis.nasa.gov/cloud/

Re: MYD03.061 CMR entry changed, but the file didn't

Posted: Wed Aug 28, 2024 6:33 pm America/New_York
by earthengine_urs
Thank you, noted. I would argue that because the actual data haven't changed, the "updated" field shouldn't change - the switch from one storage system to another is an implementation detail. Instead, the "revisionDate" field indicating the CMR metadata update time should change. But I realize this is too late to give feedback on the process.

Re: MYD03.061 CMR entry changed, but the file didn't

Posted: Thu Aug 29, 2024 8:40 am America/New_York
by LAADS-EDL_UserServices_M
Good point! However, a change in data is reflected by the change in last part of the granule id (the production date, e.g., "2018059034713" in MYD03.A2010123.2315.061.2018059034713.hdf) and the change in the "updated" field shows updates in metadata.

Re: MYD03.061 CMR entry changed, but the file didn't

Posted: Thu Aug 29, 2024 3:42 pm America/New_York
by earthengine_urs
The problem is that not every dataset uses the filename to indicate the data version (and if they do, there's no consistenvcy - eg, SMAP uses _01, _02, _03 etc instead of the timestamp). So it would have been better to rely on more structured entries in CMR, but I realize that achieving consistency across thousands of datasets is hard.

Re: MYD03.061 CMR entry changed, but the file didn't

Posted: Fri Aug 30, 2024 9:40 am America/New_York
by LAADS-EDL_UserServices_M
"I realize that achieving consistency across thousands of datasets is hard." --True!