HLS STAC Problem

[gsfc_landslides]

Hi @wvalenci,

I am still getting the error when I try to increase the limit. I was once able to retrieve all imagery using the below request, excluding the 'limit' parameter (so I was only specifying the 'collections', 'bbox', and 'datetime' parameters). In this example, my query returns 387 matches. Yet I am only able to retrieve the first 249 before there is an error. I can imagine a few workarounds, but my hope is to better understand why the previously functional search is still failing. Thank you for all the help with this issue!

### REPRODUCIBLE CODE ###

collections = ['HLSL30_2.0', 'HLSS30_2.0']
start = '2021-01-01'
end = '2024-09-01'
bbox = [-105.55427843, 35.64105739, -105.31137177, 35.81262559]

from pystac_client import Client

STAC_URL = 'https://cmr.earthdata.nasa.gov/stac'
catalog = Client.open(f"{STAC_URL}/LPCLOUD")
catalog.add_conforms_to('COLLECTIONS')

search = catalog.search(collections = collections,
bbox = bbox,
datetime = start+'/'+end,
limit=250)

print('Total matches for search:',search.matched())
item_list = list(search.items()) ### <--- THIS RETURNS THE ERROR
print(len(item_list))

OUTPUT:

Total matches for search: 387

---------------------------------------------------------------------------
APIError Traceback (most recent call last)
Cell In[29], line 18
12 search = catalog.search(collections = collections,
13 bbox = bbox,
14 datetime = start+'/'+end,
15 limit=250)
17 print('Total matches for search:',search.matched())
---> 18 item_list = list(search.items())
19 print(len(item_list))

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:694, in ItemSearch.items(self)
687 def items(self) -> Iterator[Item]:
688 """Iterator that yields :class:`pystac.Item` instances for each item matching
689 the given search parameters.
690
691 Yields:
692 Item : each Item matching the search criteria
693 """
--> 694 for item in self.items_as_dicts():
695 # already signed in items_as_dicts
696 yield Item.from_dict(item, root=self.client, preserve_dict=False)

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:705, in ItemSearch.items_as_dicts(self)
698 def items_as_dicts(self) -> Iterator[Dict[str, Any]]:
699 """Iterator that yields :class:`dict` instances for each item matching
700 the given search parameters.
701
702 Yields:
703 Item : each Item matching the search criteria
704 """
--> 705 for page in self.pages_as_dicts():
706 for item in page.get("features", []):
707 # already signed in pages_as_dicts
708 yield item

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:737, in ItemSearch.pages_as_dicts(self)
735 if isinstance(self._stac_io, StacApiIO):
736 num_items = 0
--> 737 for page in self._stac_io.get_pages(
738 self.url, self.method, self.get_parameters()
739 ):
740 call_modifier(self.modifier, page)
741 features = page.get("features", [])

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:297, in StacApiIO.get_pages(self, url, method, parameters)
285 def get_pages(
286 self,
287 url: str,
288 method: Optional[str] = None,
289 parameters: Optional[Dict[str, Any]] = None,
290 ) -> Iterator[Dict[str, Any]]:
291 """Iterator that yields dictionaries for each page at a STAC paging
292 endpoint, e.g., /collections, /search
293
294 Return:
295 Dict[str, Any] : JSON content from a single page
296 """
--> 297 page = self.read_json(url, method=method, parameters=parameters)
298 if not (page.get("features") or page.get("collections")):
299 return None

File ~/envulmo2/lib/python3.12/site-packages/pystac/stac_io.py:205, in StacIO.read_json(self, source, *args, **kwargs)
188 def read_json(self, source: HREF, *args: Any, **kwargs: Any) -> dict[str, Any]:
189 """Read a dict from the given source.
190
191 See :func:`StacIO.read_text <pystac.StacIO.read_text>` for usage of
(...)
203 given source.
204 """
--> 205 txt = self.read_text(source, *args, **kwargs)
206 return self.json_loads(txt)

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:168, in StacApiIO.read_text(self, source, *args, **kwargs)
166 href = str(source)
167 if _is_url(href):
--> 168 return self.request(href, *args, **kwargs)
169 else:
170 with open(href) as f:

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:220, in StacApiIO.request(self, href, method, headers, parameters)
218 raise APIError(str(err))
219 if resp.status_code != 200:
--> 220 raise APIError.from_response(resp)
221 try:
222 return resp.content.decode("utf-8")

APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

[wvalenci]

Hi @gsfc_landslides,

I am able to replicate your issue. Looks like we may be running into size limits with AWS API Gateway or Lambda. Will need some time to look at this a bit more, but I think its best to break up your requests for now into separate calls or whatever work around you were thinking of.

Will

[gsfc_landslides]

Hello,

Thank you for letting me know. I will proceed with this workaround in the meantime! I appreciate the help here and look forward to hearing if/how things resolve.

[kristianbodolai]

to add to this - it looks like when you search the landsat and the sentinel collections separately, it does work, and then you can just combine the item collections (see attached pdf with a minimal example)

[rbrassfield]

I've been following this thread now since it was spotted early September. I made some workarounds (separate collections requests for HLSS and HLSL) which has been working fine. I'm still getting the same API error however:

APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

But I've noticed specifically around 3pm MST every day. I don't run into issues prior to that time, but right around 3 is when I have to be done with my API requests for the day since it will not run after that. Am I hitting a limit of some kind?

Relevant code:

START_DATE = "2023-04-01" #change/update if necessary
END_DATE = "2023-10-31"
stac_items = catalog.search(
collections=s,
bbox=bbox_4326,
query={"eo:cloud_cover": {"lt": 30}},
datetime=f"{START_DATE}/{END_DATE}",
)

stac_items2 = catalog.search(
collections=l,
bbox=bbox_4326,
query={"eo:cloud_cover": {"lt": 30}},
datetime=f"{START_DATE}/{END_DATE}",
)
items = list(stac_items.items())
items2 = list(stac_items2.items())

[LP DAACx - dgolon]

Hello @rbrassfield Please look for your issue here: https://github.com/nasa/cmr-stac/issues, if you do not see any related issues posted please post what you mentioned there. Thanks -- Danielle

[aaleman]

RE: the "oops" error— We're looking into it, but so far, it doesn't appear to be a CMR-STAC issue but rather something happening upstream with CMR. Status updates on the issue can be tracked in the repo: https://github.com/nasa/cmr-stac/issues/366

[aaleman]

Also want to share an update regarding the communication of changes and releases, as well as reporting and tracking issues for CMR-STAC...

• Release notes are now available in the CMR-STAC GitHub repo: https://github.com/nasa/cmr-stac/releases. The production release cadence for CMR-STAC is typically every two weeks on Wednesday, and release notes will be posted 2 weeks ahead of scheduled release dates. The next scheduled release to production is 11/20/2024.

• The CMR-STAC GitHub repo will also be The Place for reporting and tracking the status of issues and feature requests going forward. You can check the current list of open issues here: https://github.com/nasa/cmr-stac/issues.

I'm hopeful this streamlined approach to communications will make it easier for everyone to report and track the status of issues, fixes, and new feature requests for CMR-STAC. Please reach out if you have questions or concerns.

[LP DAAC - afriesz]

Hey all,

I wanted to let you know that there has been an update deployed to the CMR-STAC API that, I think, addresses many of the issues identified in this thread. Please let us know if you continue to experience issues.

@gsfc_landslides,

The code you provided in this thread does seem to be performing properly now. Let us know if you're still running into issues.

[dyuu]

Hi @LP DAAC - afriesz

Thanks for the update! Is this fix related to the "oops" error? I am still getting that when calling the STAC api.