Earthdata Forum

I noticed at least one change in MODIS 6.1 CMR ids - the MOD13Q1.061 collections used to be C1621383370-LPDAAC_ECS, but now queries (https://cmr.earthdata.nasa.gov/search/granules.json?online_only=true&collection_concept_id[]=C1621383370-LPDAAC_ECS&page_size=1000&temporal[]=2022-06-26T00:00:00/2022-06-27T00:00:00&page_num=1) for granules in that collection return nothing. Open search returns a new collection id C1748066515-LPCLOUD, and its files (https://cmr.earthdata.nasa.gov/search/granules.json?online_only=true&collection_concept_id[]=C1748066515-LPCLOUD&page_size=1&temporal[]=2022-06-26T00:00:00/2022-06-27T00:00:00&page_num=1) are no longer at LP DAAC - the URLs are
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/MOD13Q1.061/MOD13Q1.A2022177.h02v10.061.2022195120340/MOD13Q1.A2022177.h02v10.061.2022195120340.hdf
and
s3://lp-prod-protected/MOD13Q1.061/MOD13Q1.A2022177.h02v10.061.2022195120340/MOD13Q1.A2022177.h02v10.061.2022195120340.hdf"

Is this expected? I still see new files at LP DAAC at https://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.061

Thanks,
Simon

Hi Simon,

I spoke with the Science Data Lifecycle Lead. He mentioned this is related to the cloud migration efforts. As we flip the ACL's to the new cloud version the concept IDs will change. Please update your scripts to reflect the new concept IDs. Currently we have some data that are both in the LP DAAC's Data Pool and on the cloud. Earthdata Search and CMR are both pulling cloud available data from the cloud, thus you are seeing the concept id for the cloud collection, instead of the on-prem Data Pool collection. The on-prem collection for data that are available in the cloud is not available in Earthdata Search or CMR anymore. We are currently working on migrating all LP DAAC data to the cloud, so you will see this with our other datasets in Earthdata Search and CMR as well, as they complete the migration.

Thanks,
LP DAAC

Hi @LP DAAC - dgolon,

I'm still facing the same problem today. Multiple V061 data products of MODIS are still missing from the CMR, such as MYD11A1, MOD11A1 and MCD43A3, https://cmr.earthdata.nasa.gov/search/site/collections/directory/LPDAAC_ECS/gov.nasa.eosdis

You mention that this is due to the data migration to the cloud storage. However, the NASA CMR Cloud STAC endpoint currently is not returning these collections of data products (or any collection at all...) https://cmr.earthdata.nasa.gov/cloudstac

Can you give some pointer on how to programmatically query MODIS V061 products at the moment? When do we expect to see the MODIS V061 products appear on the CMR list?

Thanks,

Zhan.

Hi @zhan.li We will look into this and get back to you. Thanks!

@zhan.li & @earthengine_urs ,

As was mentioned before, the LP DAAC is in a state where on-prem data is actively being moved to the NASA Earthdata Cloud. When those on-prem (i.e., provider=LPDAAC_ECS) collections are moved to the cloud (i.e., provider=LPCLOUD) they receive a new concept ID, AND the on-prem collections are made unavailable. We don’t have a mechanism in place to let users know when those collections are now available in the cloud. We apologize for the inconvenience.

Regarding work arounds for the issues you’ve described here…I recommend a change to the CMR API search query you are using. I would recommend searching against both the “LPCLOUD” and “LPDAAC_ECS” providers. I would also query for a collection using the “short_name” parameter and “version”, rather than the “collection_concept_id” parameter. Below are example request queries:

Collection Search:
https://cmr.earthdata.nasa.gov/search/collections.json?provider=LPCLOUD&provider=LPDAAC_ECS&short_name=MYD11A1&version=061

Granules Search:
https://cmr.earthdata.nasa.gov/search/granules.json?provider=LPCLOUD&temporal=2022-06-26T00%3A00%3A00Z,2022-06-27T23%3A59%3A59Z&provider=LPDAAC_ECS&short_name=MYD11A1&version=061

Thank you for the response. Can all datasets in CMR be addressed by a short name + version query? I suppose that would work for me too, thought it's nice to deal with a single concept id string.

@earthengine_urs All LP DAAC datasets in CMR can be addressed by a short name + version query, I believe that would be the same case for all non-lp related datasets in CMR too.