Stac Error on LPCLOUD

Use this Forum to find information on, or ask a question about, NASA Earth Science data.
Post Reply
matthew_cox_omnia
Posts: 21
Joined: Thu Sep 21, 2023 4:40 pm America/New_York
Answers: 0
Has thanked: 2 times
Been thanked: 1 time

Stac Error on LPCLOUD

by matthew_cox_omnia » Thu Oct 17, 2024 11:17 am America/New_York

Hi LP DAAC folks!

Starting yesterday afternoon, accessing granules via a STAC client on LPCloud
started failing - specifically pystac, though I doubt that makes a difference. When attempting to retrieve an item collection, the client receives the error

Code: Select all

"Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."
This has been occurring for at least 18 hours every time that I've attempted. I'm not seeing anything in the forums that would indicate that this is a known or anticipated issue. I don't think it's a username/password issue as I've double checked that I can log in to https://search.earthdata.nasa.gov using the same credentials.

Is there any more context that I can provide that would help resolve the issue?
Thanks,
Matthew

Filters:

matthew_cox_omnia
Posts: 21
Joined: Thu Sep 21, 2023 4:40 pm America/New_York
Answers: 0
Has thanked: 2 times
Been thanked: 1 time

Re: Stac Error on LPCLOUD

by matthew_cox_omnia » Thu Oct 17, 2024 11:22 am America/New_York

I'm sorry for the multiple posts - when attempting to post to the form, I was receiving the message that the confirmation challenge was incorrect (even though it appears that I entered correctly) and not redirecting me. Perhaps this is related to my issue? Anyways, feel free to delete this topic - it appears that I'm unable to do so.

LP DAACx - dgolon
User Services
User Services
Posts: 422
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 31 times
Been thanked: 8 times
Contact:

Re: Stac Error on LPCLOUD

by LP DAACx - dgolon » Thu Oct 17, 2024 2:36 pm America/New_York

Hi @matthew_cox_omnia Thank you for contacting the LP DAAC. I've removed the other copies of your post, I'll pass that along to our forum developer to see if that is a known issue with the forum. In the meantime, I'm talking with our LP DAAC science team to see if this issue is related to the known on-going STAC issues, although I wouldn't think it is since you just started seeing it yesterday, or if it is something new. I'll follow up when I have additional details. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

matthew_cox_omnia
Posts: 21
Joined: Thu Sep 21, 2023 4:40 pm America/New_York
Answers: 0
Has thanked: 2 times
Been thanked: 1 time

Re: Stac Error on LPCLOUD

by matthew_cox_omnia » Thu Oct 17, 2024 3:58 pm America/New_York

Thanks Danielle-
In case it helps pinpoint timing, I'm pretty confident that it was working as expected as of 9PM CT on 10/15 but had issues as early as 4:30 CT on 10/16.
Matthew

matthew_cox_omnia
Posts: 21
Joined: Thu Sep 21, 2023 4:40 pm America/New_York
Answers: 0
Has thanked: 2 times
Been thanked: 1 time

Re: Stac Error on LPCLOUD

by matthew_cox_omnia » Fri Oct 18, 2024 9:17 am America/New_York

Danielle-
I confirmed this morning that the STAC Error is no longer occurring. Thanks for looking into this.
Matthew

montesano
Posts: 1
Joined: Wed Oct 23, 2024 9:15 am America/New_York
Answers: 0

Re: Stac Error on LPCLOUD

by montesano » Wed Oct 23, 2024 12:24 pm America/New_York

Hi all - I believe I continue to see this 'limit' related issue. Is there any new guidance on how to overcome this limit issue that seems to have arisen over the past 1.5 months? Is there something that I may have missed? This seems to be a very big problem for our ability to return the full set of search results that we then need to filter and work with.

reproducible code:
#---START CODE----------------------------------
from pystac_client import Client

MAX_N_RESULTS = 252

catalog = Client.open('https://cmr.earthdata.nasa.gov/stac/LPCLOUD')
MS_product_list = ['HLSL30_2.0', 'HLSS30_2.0']

print(f'\nConducting multispectral image search now with limit = {MAX_N_RESULTS}...')
print(f'Searching for:\t\t\t{MS_product_list}')

search = catalog.search(
collections=MS_product_list,
datetime=['2024-07-01T00:00:00Z' , '2024-08-31T23:59:59Z'],
bbox=[125.65378735345928, 70.23761759131604, 129.10749149273934, 71.38291000560454],
limit=MAX_N_RESULTS,
max_items=None
)
results = search.get_all_items_as_dict()

INFO_STR = ''
if len(results['features']) == MAX_N_RESULTS:
INFO_STR = f"(Heads up! # of search results is the same as your limit ({MAX_N_RESULTS}). Increase limit to filter from full set of data that meets your search criteria.)"
print("initial results:\t\t", len(results['features']), INFO_STR)
#--END CODE----------

Print out:
Conducting multispectral image search now with limit = 252...
Searching for: ['HLSL30_2.0', 'HLSS30_2.0']
initial results: 252 (Heads up! # of search results is the same as your limit (252). Increase limit to filter from full set of data that meets your search criteria.)

However, changing the limit to, in the case 255, will result in this error:
APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

LP DAACx - dgolon
User Services
User Services
Posts: 422
Joined: Mon Sep 30, 2019 10:00 am America/New_York
Answers: 0
Has thanked: 31 times
Been thanked: 8 times
Contact:

Re: Stac Error on LPCLOUD

by LP DAACx - dgolon » Thu Oct 24, 2024 8:56 am America/New_York

Hi @montesano Issues are being addressed here: https://github.com/nasa/cmr-stac/issues if you don't see a similar issue already reported please feel free to add a new issue.
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.

Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.

wvalenci
Posts: 11
Joined: Thu Sep 05, 2024 12:39 pm America/New_York
Answers: 0
Been thanked: 5 times

Re: Stac Error on LPCLOUD

by wvalenci » Wed Oct 30, 2024 6:42 pm America/New_York

@montesano
You may be misunderstanding the limit and max_items parameters. Limit actually limits the results per call to CMR-STAC, but pystac's get_all_items_as_dict() will actually call CMR-STAC multiple times until it gets all the results(using the "next" links automatically for you). So using a limit of 250 or less will work for your case, it just may take a little time. The error you get with limit > 255 is a known issue with AWS Lambda limiting the return values to 6MB. So just use limit = 250 and you should be ok. If you set the max_items to 252, then you will only get 252 items and it should return faster.
Attachments
Screenshot 2024-10-30 at 6.39.06 PM.png
Screenshot 2024-10-30 at 6.39.06 PM.png (334.23 KiB) Not viewed yet

Post Reply