Page 2 of 4
Re: HLS STAC Problem
Posted: Wed Sep 18, 2024 12:07 am America/New_York
by dyuu
@dgolon I tested various search param settings. I noticed the same behavior - that longer periods (> 6 months) will result in the "oops" API error.
My MRE (small section of Central park, 9 months):
results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)
If there's any other observations I can provide to the CMR team that would be of help, please let me know. Thank you.
Re: HLS STAC Problem
Posted: Wed Sep 18, 2024 3:21 pm America/New_York
by ASDC - rkey
Posting on behalf of
@LP DAAC - dgolon :
Hello all,
Work on the STAC API continues. We apologize for the inconvenience. In the meantime, I'd like to suggest earthaccess, a Python library designed to simplify authentication and the process of searching for, downloading, or streaming NASA Earth science data. Here's a block of code that replicate the initial post:
```python
import earthaccess
earthaccess.login(persist=True)
collection_query = earthaccess.collection_query().provider('LPCLOUD').keyword('HLS')
hls_products = ['HLSL30', 'HLSS30']
collection_ids = [x['meta']['concept-id'] for x in collection_query.fields(['ShortName']).get() if x['umm']['ShortName'] in hls_products]
bbox = (-105.58650854045227,40.05184049311418,-105.5832945453711, 40.05424331298521)
date = ('2023-05-01','2023-09-30')
granules_request = earthaccess.search_data(
concept_id=collection_ids,
provider='LPCLOUD',
bounding_box=bbox,
temporal=date,
count=100
)
asset_links = [x.data_links() for x in granules_request]
asset_links
```
Here are some additional resources for getting started with earthaccess:
-
https://github.com/nsidc/earthaccess/
-
https://nasa-openscapes.github.io/earthdata-cloud-cookbook/how-tos/find-data/find-python.html
-
https://github.com/CU-ESIIL/HYR-SENSE/blob/main/notebooks/how_to/earthaccess_Introduction.ipynb
GitHub - nsidc/earthaccess: Python Library for NASA Earthdata APIs
Python Library for NASA Earthdata APIs. Contribute to nsidc/earthaccess development by creating an account on GitHub.
Re: HLS STAC Problem
Posted: Wed Sep 18, 2024 3:35 pm America/New_York
by dyuu
Thank you, is there an ETA on the STAC fixes? It would help inform whether to proceed with the workaround, or wait for updates. Cheers.
Re: HLS STAC Problem
Posted: Wed Sep 18, 2024 5:20 pm America/New_York
by LP DAACx - dgolon
Hi
@dyuu The CMR STAC team is working on a resolution. I recommend following this thread:
viewtopic.php?t=5917 as they are posting in there.
Re: HLS STAC Problem
Posted: Thu Sep 19, 2024 8:31 am America/New_York
by primi1.qc.ca
For those interested, I just been told by the user service that AppEEARS will always do some data treatment and geoprocessing. So better to simply go with "
https://search.earthdata.nasa.gov/search" to quickly obtain the granule-list.
Cheers
Re: HLS STAC Problem
Posted: Fri Sep 20, 2024 11:53 am America/New_York
by ircwaves
Re: HLS STAC Problem
Posted: Mon Sep 23, 2024 8:47 am America/New_York
by ircwaves
dyuu wrote: ↑Wed Sep 18, 2024 12:07 am America/New_York
My MRE (small section of Central park, 9 months):
results = catalog.search(
bbox=[-73.9819, 40.7681, -73.9580, 40.7851],
collections=["HLSL30_2.0"],
datetime="2019-01-01T00:00:00Z/2019-09-01T00:00:00Z",
)
Assuming your catalog is a pystac-client object, I believe that you're hitting the same pagination bug that I ran into. If not resolved soon, I'll need to break up my queries into smaller time-space windows, to avoid this bug. The side effect of such would be a rather large increase in the number of queries, which is probably not desirable for anyone involved.
Re: HLS STAC Problem
Posted: Thu Sep 26, 2024 5:36 pm America/New_York
by wvalenci
Please try your request again walterdan, it should be working now.
Re: HLS STAC Problem
Posted: Fri Sep 27, 2024 10:38 am America/New_York
by waltersdan
Hi
@wvalenci it looks like that fixed the image limit, but the metadata asset is still missing.
The metadata asset has info from the source image (Landsat or Sentinel) that is not otherwise contained in the STAC. It used to link to an .xml file (e.g. '
https://cmr.earthdata.nasa.gov/search/concepts/G2243943991-LPCLOUD.xml'). For example, in our pipeline we use the original image ID to parse out the relative orbit number, which we use in follow-on processing.
Ideally, all the metadata would be in the STAC itself under each item's properties (this would be much faster!), but it is not, so we have been querying the metadata asset link for each item to get it.
I can open a separate thread for the metadata issue if that is helpful
Cheers
Re: HLS STAC Problem
Posted: Fri Sep 27, 2024 2:53 pm America/New_York
by wvalenci
We have a separate ticket for the missing metadata issue already. It is in the works.
William