Earthdata Forum

There seems to be an issue with the HLS STAC that started a couple days ago. A minimal reproducible example is: This search finds 22 items and returns the error: The search does work if the date range is shortened. In the above example, if I shorten to 2013-07-01/2013-09-20, then it finds 20 items and does not get the error. Likewise, if I add 'limit': 23 to params, then the first search works. However, this doesn't seem to be extensible to get the full archive as it was before. For example, if I change the date range to 2013-07-01/2024-07-01 and 'limit': 2000, the search finds 1040 items but fails with the same error.

Further, the limit where this occurs seems to change by ___location. In another ___location the issue occurred at 100 items. There was a recent post about the name of the collection, I tried both 'HLSL30_2.0' and 'HLSL30.v2.0', both produce the same results.

Finally (perhaps a separate issue), it appears that the 'metadata' asset has been removed from the items. The metadata link was the only way to get certain properties of the original images (e.g., orbit number) as these are not included in the STAC properties.

Thanks for your help!

Hello @waltersdan Thank you for reporting this issue. Our team is looking into it. We will post when we have additional information. Thanks -- Danielle

I am facing the same issue, seeing most datetime ranges returning the same API error.

Additionally, was there some recent change that resulted in the search API datetime param type changing from array to string? Would be great to know where I can stay up to date with these API changes.

Thanks for kickstarting the thread, following along here.

Hi,
I’m facing a similar issue and here’s my intakes:

- Based on your old python script (HLS_Su.py, HLS_Super.py, HLS_PER.py) I was able to download the 20th au August 2024. After the stac update: (https://lpdaac.usgs.gov/news/important-update-to-cmr-stac-new-identifier-and-search-parameter-format/) the problem occurred. No surprise here.
- Yes, the xml metadata file is not where it was. You can still find it somewhere in the stac, but not as directly as it was.
- Even after changing the product name from shortname.'v'versionId to shortname_versionId, I faced a date format issue. I found that the old version allowed us to write a single digit day without starting with a zero and not the new version. For exemple:

Old version: "2024-07-1T00:00:00Z/2024-07-31T00:00:00Z"
New version: "2024-07-01T00:00:00Z/2024-07-31T00:00:00Z"

- The most important problem in my case: my query tells there’s 1642 matched. In the parameters of my query, I can’t ask for limit that big or I receive and error message, and putting no limit seems to give a default limit value to 20. According to the python download script, we were setting in parameters limit=100 and incrementing pages until we reach an empty page.
search_response = r.post(lp_stac, json=params).json()
where params = {"bbox": bbox_string, "limit": 100, "datetime": dates, "collections": [prods[b]], "page": page}

But with this new stac version, the “page” parameter doesn’t change pages. Whatever the number I put, it always returns the same 100 observations. If it as to stay that way, I would need an indication of how to have access to the entire matched list.
Thank you very much for your efforts and for this discussion. I’ll stay in touch for the following.

Hi,
I’m facing a similar issue and here’s my intakes:

- Based on your old python script (HLS_Su.py, HLS_Super.py, HLS_PER.py) I was able to download the 20th au August 2024. After the stac update: (https://lpdaac.usgs.gov/news/important-update-to-cmr-stac-new-identifier-and-search-parameter-format/) the problem occurred. No surprise here.
- Yes, the xml metadata file is not where it was. You can still find it somewhere in the stac, but not as directly as it was.
- Even after changing the product name from shortname.'v'versionId to shortname_versionId, I faced a date format issue. I found that the old version allowed us to write a single digit day without starting with a zero and not the new version. For exemple:

Old version: "2024-07-1T00:00:00Z/2024-07-31T00:00:00Z"
New version: "2024-07-01T00:00:00Z/2024-07-31T00:00:00Z"

- The most important problem in my case: my query tells there’s 1642 matched. In the parameters of my query, I can’t ask for limit that big or I receive and error message, and putting no limit seems to give a default limit value to 20. According to the python download script, we were setting in parameters limit=100 and incrementing pages until we reach an empty page.
search_response = r.post(lp_stac, json=params).json()
where params = {"bbox": bbox_string, "limit": 100, "datetime": dates, "collections": [prods[b]], "page": page}

But with this new stac version, the “page” parameter doesn’t change pages. Whatever the number I put, it always returns the same 100 observations. If it as to stay that way, I would need an indication of how to have access to the entire matched list.
Thank you very much for your efforts and for this discussion. I’ll stay in touch for the following.

Chiming in to say I've noticed this same issue with my code that previously worked:

### REPRODUCIBLE CODE ####

collections = ['HLSL30_2.0', 'HLSS30_2.0']
start = '2021-01-01'
end = '2024-12-01'
bbox = [-105.55427843, 35.64105739, -105.31137177, 35.81262559]

from pystac_client import Client

STAC_URL = 'https://cmr.earthdata.nasa.gov/stac'
catalog = Client.open(f"{STAC_URL}/LPCLOUD")
catalog.add_conforms_to('COLLECTIONS')

search = catalog.search(collections = collections, bbox = bbox, datetime = start+'/'+end,)

print('Total matches for search:',search.matched())

In my case, the total matches is 390. Then when I try to retrieve the items via:

item_list = list(search.items())

It only returns a default of 20, up to about 150 if I tweak with the limit parameter. Otherwise it returns that same error if I try to go higher:

############ TRACEBACK ###############
---------------------------------------------------------------------------
APIError Traceback (most recent call last)
Cell In[40], line 1
----> 1 item_list = list(search.items())

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:694, in ItemSearch.items(self)
687 def items(self) -> Iterator[Item]:
688 """Iterator that yields :class:`pystac.Item` instances for each item matching
689 the given search parameters.
690
691 Yields:
692 Item : each Item matching the search criteria
693 """
--> 694 for item in self.items_as_dicts():
695 # already signed in items_as_dicts
696 yield Item.from_dict(item, root=self.client, preserve_dict=False)

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:705, in ItemSearch.items_as_dicts(self)
698 def items_as_dicts(self) -> Iterator[Dict[str, Any]]:
699 """Iterator that yields :class:`dict` instances for each item matching
700 the given search parameters.
701
702 Yields:
703 Item : each Item matching the search criteria
704 """
--> 705 for page in self.pages_as_dicts():
706 for item in page.get("features", []):
707 # already signed in pages_as_dicts
708 yield item

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/item_search.py:737, in ItemSearch.pages_as_dicts(self)
735 if isinstance(self._stac_io, StacApiIO):
736 num_items = 0
--> 737 for page in self._stac_io.get_pages(
738 self.url, self.method, self.get_parameters()
739 ):
740 call_modifier(self.modifier, page)
741 features = page.get("features", [])

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:297, in StacApiIO.get_pages(self, url, method, parameters)
285 def get_pages(
286 self,
287 url: str,
288 method: Optional[str] = None,
289 parameters: Optional[Dict[str, Any]] = None,
290 ) -> Iterator[Dict[str, Any]]:
291 """Iterator that yields dictionaries for each page at a STAC paging
292 endpoint, e.g., /collections, /search
293
294 Return:
295 Dict[str, Any] : JSON content from a single page
296 """
--> 297 page = self.read_json(url, method=method, parameters=parameters)
298 if not (page.get("features") or page.get("collections")):
299 return None

File ~/envulmo2/lib/python3.12/site-packages/pystac/stac_io.py:205, in StacIO.read_json(self, source, *args, **kwargs)
188 def read_json(self, source: HREF, *args: Any, **kwargs: Any) -> dict[str, Any]:
189 """Read a dict from the given source.
190
191 See :func:`StacIO.read_text <pystac.StacIO.read_text>` for usage of
(...)
203 given source.
204 """
--> 205 txt = self.read_text(source, *args, **kwargs)
206 return self.json_loads(txt)

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:168, in StacApiIO.read_text(self, source, *args, **kwargs)
166 href = str(source)
167 if _is_url(href):
--> 168 return self.request(href, *args, **kwargs)
169 else:
170 with open(href) as f:

File ~/envulmo2/lib/python3.12/site-packages/pystac_client/stac_api_io.py:220, in StacApiIO.request(self, href, method, headers, parameters)
218 raise APIError(str(err))
219 if resp.status_code != 200:
--> 220 raise APIError.from_response(resp)
221 try:
222 return resp.content.decode("utf-8")

APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

Thanks to everyone for flagging this so I know I'm not alone!

Hi @dgolon,
Any updates on this? Not being able to query the timeseries makes HLS unusable for our work, I imagine others are in the same boat...
Thanks!
Dan

@waltersdan I believe the issue you are seeing is related to the issue reported here, I saw you are also in the thread there too: viewtopic.php?t=5917 We are waiting more information from the CMR STAC team.

Ok, thanks for getting back!

@waltersdan I used AppEEARS as a workaround to make my query. The platform is very simple and well designed. There's many outputs, but I just needed this file: "[Query_NAME]-granule-list.txt" to be able to continue my workflow. It is the exact same output than what my Stac query was giving me.

A big disadvantage from this workaround is queries take a LOT MORE time to accomplish, like hours instead of seconds. I informed their team of that.

For my work, it was important to have access to HLS, so I'm pretty happy with this plan B, but it will only be a plan B.