HLS STAC Problem

[dyuu]

@dgolon I tested various search param settings. I noticed the same behavior - that longer periods (> 6 months) will result in the "oops" API error.

My MRE (small section of Central park, 9 months):

results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)

If there's any other observations I can provide to the CMR team that would be of help, please let me know. Thank you.

[ASDC - rkey]

Posting on behalf of @LP DAAC - dgolon :

Hello all,

Work on the STAC API continues. We apologize for the inconvenience. In the meantime, I'd like to suggest earthaccess, a Python library designed to simplify authentication and the process of searching for, downloading, or streaming NASA Earth science data. Here's a block of code that replicate the initial post:

```python

import earthaccess

earthaccess.login(persist=True)

collection_query = earthaccess.collection_query().provider('LPCLOUD').keyword('HLS')

hls_products = ['HLSL30', 'HLSS30']

collection_ids = [x['meta']['concept-id'] for x in collection_query.fields(['ShortName']).get() if x['umm']['ShortName'] in hls_products]

bbox = (-105.58650854045227,40.05184049311418,-105.5832945453711, 40.05424331298521)

date = ('2023-05-01','2023-09-30')

granules_request = earthaccess.search_data(

concept_id=collection_ids,

provider='LPCLOUD',

bounding_box=bbox,

temporal=date,

count=100

)

asset_links = [x.data_links() for x in granules_request]

asset_links

```

Here are some additional resources for getting started with earthaccess:

- https://github.com/nsidc/earthaccess/

- https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-python.html

- https://github.com/CU-ESIIL/HYR-SENSE/blob/main/notebooks/how_to/earthaccess_Introduction.ipynb
GitHub - nsidc/earthaccess: Python Library for NASA Earthdata APIs
Python Library for NASA Earthdata APIs. Contribute to nsidc/earthaccess development by creating an account on GitHub.

[dyuu]

Thank you, is there an ETA on the STAC fixes? It would help inform whether to proceed with the workaround, or wait for updates. Cheers.

[LP DAACx - dgolon]

Hi @dyuu The CMR STAC team is working on a resolution. I recommend following this thread: viewtopic.php?t=5917 as they are posting in there.

[primi1.qc.ca]

For those interested, I just been told by the user service that AppEEARS will always do some data treatment and geoprocessing. So better to simply go with "https://search.earthdata.nasa.gov/search" to quickly obtain the granule-list.

Cheers

[ircwaves]

This problem may be related to a pagination issue. To reproduce

1. visit https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30_2.0/items
2. select the next link

   Code: Select all

   jsonresponse.links[3].href

   which will probably be something like this https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30_2.0/items?cursor=eyJqc29uIjoiW1wibHBjbG91ZFwiLDEzNjU2MzkzMzI4ODgsMjI0NzA2MTU3MF0iLCJ1bW0iOiJbXCJscGNsb3VkXCIsMTM2NTYzOTMzMjg4OCwyMjQ3MDYxNTcwXSJ9
3. get error

   Code: Select all

   "Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."

[ircwaves]

My MRE (small section of Central park, 9 months):

results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)

Assuming your catalog is a pystac-client object, I believe that you're hitting the same pagination bug that I ran into. If not resolved soon, I'll need to break up my queries into smaller time-space windows, to avoid this bug. The side effect of such would be a rather large increase in the number of queries, which is probably not desirable for anyone involved.

[wvalenci]

Please try your request again walterdan, it should be working now.

[waltersdan]

Hi @wvalenci it looks like that fixed the image limit, but the metadata asset is still missing.

The metadata asset has info from the source image (Landsat or Sentinel) that is not otherwise contained in the STAC. It used to link to an .xml file (e.g. 'https://cmr.earthdata.nasa.gov/search/concepts/G2243943991-LPCLOUD.xml'). For example, in our pipeline we use the original image ID to parse out the relative orbit number, which we use in follow-on processing.

Ideally, all the metadata would be in the STAC itself under each item's properties (this would be much faster!), but it is not, so we have been querying the metadata asset link for each item to get it.

I can open a separate thread for the metadata issue if that is helpful
Cheers

[wvalenci]

We have a separate ticket for the missing metadata issue already. It is in the works.

William