Earthdata Forum

Hi LP DAAC folks!

Starting yesterday afternoon, accessing granules via a STAC client on LPCloud
started failing - specifically pystac, though I doubt that makes a difference. When attempting to retrieve an item collection, the client receives the error This has been occurring for at least 18 hours every time that I've attempted. I'm not seeing anything in the forums that would indicate that this is a known or anticipated issue. I don't think it's a username/password issue as I've double checked that I can log in to https://search.earthdata.nasa.gov using the same credentials.

Is there any more context that I can provide that would help resolve the issue?
Thanks,
Matthew

I'm sorry for the multiple posts - when attempting to post to the form, I was receiving the message that the confirmation challenge was incorrect (even though it appears that I entered correctly) and not redirecting me. Perhaps this is related to my issue? Anyways, feel free to delete this topic - it appears that I'm unable to do so.

Hi @matthew_cox_omnia Thank you for contacting the LP DAAC. I've removed the other copies of your post, I'll pass that along to our forum developer to see if that is a known issue with the forum. In the meantime, I'm talking with our LP DAAC science team to see if this issue is related to the known on-going STAC issues, although I wouldn't think it is since you just started seeing it yesterday, or if it is something new. I'll follow up when I have additional details. Thanks -- Danielle

Thanks Danielle-
In case it helps pinpoint timing, I'm pretty confident that it was working as expected as of 9PM CT on 10/15 but had issues as early as 4:30 CT on 10/16.
Matthew

Danielle-
I confirmed this morning that the STAC Error is no longer occurring. Thanks for looking into this.
Matthew

Hi all - I believe I continue to see this 'limit' related issue. Is there any new guidance on how to overcome this limit issue that seems to have arisen over the past 1.5 months? Is there something that I may have missed? This seems to be a very big problem for our ability to return the full set of search results that we then need to filter and work with.

reproducible code:
#---START CODE----------------------------------
from pystac_client import Client

MAX_N_RESULTS = 252

catalog = Client.open('https://cmr.earthdata.nasa.gov/stac/LPCLOUD')
MS_product_list = ['HLSL30_2.0', 'HLSS30_2.0']

print(f'\nConducting multispectral image search now with limit = {MAX_N_RESULTS}...')
print(f'Searching for:\t\t\t{MS_product_list}')

search = catalog.search(
collections=MS_product_list,
datetime=['2024-07-01T00:00:00Z' , '2024-08-31T23:59:59Z'],
bbox=[125.65378735345928, 70.23761759131604, 129.10749149273934, 71.38291000560454],
limit=MAX_N_RESULTS,
max_items=None
)
results = search.get_all_items_as_dict()

INFO_STR = ''
if len(results['features']) == MAX_N_RESULTS:
INFO_STR = f"(Heads up! # of search results is the same as your limit ({MAX_N_RESULTS}). Increase limit to filter from full set of data that meets your search criteria.)"
print("initial results:\t\t", len(results['features']), INFO_STR)
#--END CODE----------

Print out:
Conducting multispectral image search now with limit = 252...
Searching for: ['HLSL30_2.0', 'HLSS30_2.0']
initial results: 252 (Heads up! # of search results is the same as your limit (252). Increase limit to filter from full set of data that meets your search criteria.)

However, changing the limit to, in the case 255, will result in this error:
APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

Hi @montesano Issues are being addressed here: https://github.com/nasa/cmr-stac/issues if you don't see a similar issue already reported please feel free to add a new issue.

@montesano
You may be misunderstanding the limit and max_items parameters. Limit actually limits the results per call to CMR-STAC, but pystac's get_all_items_as_dict() will actually call CMR-STAC multiple times until it gets all the results(using the "next" links automatically for you). So using a limit of 250 or less will work for your case, it just may take a little time. The error you get with limit > 255 is a known issue with AWS Lambda limiting the return values to 6MB. So just use limit = 250 and you should be ok. If you set the max_items to 252, then you will only get 252 items and it should return faster.