Page 2 of 4

Re: HLS STAC Problem

Posted: Wed Sep 18, 2024 12:07 am America/New_York
by dyuu
@dgolon I tested various search param settings. I noticed the same behavior - that longer periods (> 6 months) will result in the "oops" API error.

My MRE (small section of Central park, 9 months):

results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)

If there's any other observations I can provide to the CMR team that would be of help, please let me know. Thank you.

Re: HLS STAC Problem

Posted: Wed Sep 18, 2024 3:21 pm America/New_York
by ASDC - rkey
Posting on behalf of @LP DAAC - dgolon :
Hello all,

Work on the STAC API continues. We apologize for the inconvenience. In the meantime, I'd like to suggest earthaccess, a Python library designed to simplify authentication and the process of searching for, downloading, or streaming NASA Earth science data. Here's a block of code that replicate the initial post:

```python

import earthaccess

earthaccess.login(persist=True)

collection_query = earthaccess.collection_query().provider('LPCLOUD').keyword('HLS')

hls_products = ['HLSL30', 'HLSS30']

collection_ids = [x['meta']['concept-id'] for x in collection_query.fields(['ShortName']).get() if x['umm']['ShortName'] in hls_products]

bbox = (-105.58650854045227,40.05184049311418,-105.5832945453711, 40.05424331298521)

date = ('2023-05-01','2023-09-30')

granules_request = earthaccess.search_data(

concept_id=collection_ids,

provider='LPCLOUD',

bounding_box=bbox,

temporal=date,

count=100

)

asset_links = [x.data_links() for x in granules_request]

asset_links

```

Here are some additional resources for getting started with earthaccess:

- https://github.com/nsidc/earthaccess/

- https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-python.html

- https://github.com/CU-ESIIL/HYR-SENSE/blob/main/notebooks/how_to/earthaccess_Introduction.ipynb
GitHub - nsidc/earthaccess: Python Library for NASA Earthdata APIs
Python Library for NASA Earthdata APIs. Contribute to nsidc/earthaccess development by creating an account on GitHub.

Re: HLS STAC Problem

Posted: Wed Sep 18, 2024 3:35 pm America/New_York
by dyuu
Thank you, is there an ETA on the STAC fixes? It would help inform whether to proceed with the workaround, or wait for updates. Cheers.

Re: HLS STAC Problem

Posted: Wed Sep 18, 2024 5:20 pm America/New_York
by LP DAACx - dgolon
Hi @dyuu The CMR STAC team is working on a resolution. I recommend following this thread: viewtopic.php?t=5917 as they are posting in there.

Re: HLS STAC Problem

Posted: Thu Sep 19, 2024 8:31 am America/New_York
by primi1.qc.ca
For those interested, I just been told by the user service that AppEEARS will always do some data treatment and geoprocessing. So better to simply go with "https://search.earthdata.nasa.gov/search" to quickly obtain the granule-list.

Cheers

Re: HLS STAC Problem

Posted: Fri Sep 20, 2024 11:53 am America/New_York
by ircwaves
This problem may be related to a pagination issue. To reproduce
  1. visit https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30_2.0/items
  2. select the next link

    Code: Select all

    jsonresponse.links[3].href
    which will probably be something like this https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30_2.0/items?cursor=eyJqc29uIjoiW1wibHBjbG91ZFwiLDEzNjU2MzkzMzI4ODgsMjI0NzA2MTU3MF0iLCJ1bW0iOiJbXCJscGNsb3VkXCIsMTM2NTYzOTMzMjg4OCwyMjQ3MDYxNTcwXSJ9
  3. get error

    Code: Select all

    "Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."

Re: HLS STAC Problem

Posted: Mon Sep 23, 2024 8:47 am America/New_York
by ircwaves
dyuu wrote: Wed Sep 18, 2024 12:07 am America/New_York
My MRE (small section of Central park, 9 months):

results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)
Assuming your catalog is a pystac-client object, I believe that you're hitting the same pagination bug that I ran into. If not resolved soon, I'll need to break up my queries into smaller time-space windows, to avoid this bug. The side effect of such would be a rather large increase in the number of queries, which is probably not desirable for anyone involved.

Re: HLS STAC Problem

Posted: Thu Sep 26, 2024 5:36 pm America/New_York
by wvalenci
Please try your request again walterdan, it should be working now.

Re: HLS STAC Problem

Posted: Fri Sep 27, 2024 10:38 am America/New_York
by waltersdan
Hi @wvalenci it looks like that fixed the image limit, but the metadata asset is still missing.

The metadata asset has info from the source image (Landsat or Sentinel) that is not otherwise contained in the STAC. It used to link to an .xml file (e.g. 'https://cmr.earthdata.nasa.gov/search/concepts/G2243943991-LPCLOUD.xml'). For example, in our pipeline we use the original image ID to parse out the relative orbit number, which we use in follow-on processing.

Ideally, all the metadata would be in the STAC itself under each item's properties (this would be much faster!), but it is not, so we have been querying the metadata asset link for each item to get it.

I can open a separate thread for the metadata issue if that is helpful
Cheers

Re: HLS STAC Problem

Posted: Fri Sep 27, 2024 2:53 pm America/New_York
by wvalenci
We have a separate ticket for the missing metadata issue already. It is in the works.

William