Earthdata Forum

I have a Python script that I use to download OPERA DSWx data from the command line (https://github.com/ScienceCore/scipy-2024-climaterisk/blob/main/00_Introduction_Setup/test_netrc.py), and to also help test out that Earthdata credentials have been set up correctly. This script has now stopped working, and I don't see any results returned by the PySTAC query, even though data can be found and downloaded when performing a manual search using the GUI at search.earthdata.nasa.gov.

What gives?

Hello, your script works successfully when we try running it. Do you get an error message? If you try the pystac query directly in a browser, do you see results (https://cmr.earthdata.nasa.gov/stac/POCLOUD/collections/OPERA_L3_DSWX-HLS_V1_1.0/items?datetime=2024-04-30T00:00:00Z/2024-05-05T23:59:59Z)?

If you aren’t getting to the try/catch block or are seeing anything other than the Success message printed, it’s likely that something’s gone wrong with your .netrc file. You can check your .netrc file by opening it in a text editor, or use the following script to create a new one:
https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/04_NASA_Earthdata_Authentication.html

I updated the script for it to work - two changes were made, one was to update the collections name for OPERA DSWx-HLS (not sure why this was changed?), and the second was to reduce the time window over which the search is performed. The script still breaks if a search is performed for a 1 month window.

The queries seem to break when the search query returns more than 20 results.

Hi, I was wondering if there has been any update on this issue?

In the script, with the start and stop dates set as `start, stop = '2024-04-30', '2024-05-15'`, I receive the following error:

pystac_client.exceptions.APIError: {"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

Thank you for your follow-up inquiry.

For additional assistance, I have forwarded your question to our OPERA subject matter expert.

Many thanks for your continued patience.

Respectfully,
PO.DAAC User Services

I appreciate your help in this matter! However, I don't think the issue is limited to OPERA data products, as has been noted by other users: viewtopic.php?t=5922&sid=353377393ac757496463beaceb408d67

If I change my code to have the following values to search for HLS data, I can reproduce the error as well:

start, stop = '2024-04-30', '2024-07-31'
STAC_URL, COLLECTIONS = 'https://cmr.earthdata.nasa.gov/stac', ["HLSL30_2.0"]
catalog = Client.open(f'{STAC_URL}/LPCLOUD/')

I have a feeling that this extends beyond any single data product, and is an issue with the CMR system itself, but I would be curious to hear your thoughts.

Hello,
You're right, this looks like the same issue that was found with the HLS data. It has been reported to the CMR team and they are working on a fix in the underlying query API. I have added your findings with the OPERA data to the bug report so they can test it when verifying the fix.
I don't know yet the timeframe for this to be resolved but will update here when we know more.

Thanks,
Victoria
PO.DAAC Data Engineer

By adapting your script to use the earthaccess python package, you can get the search results you're looking for -- I've adapted your script to use earthaccess:
https://gist.github.com/jhkennedy/3c3b181516801f0e66ba1a8e35c9f103

This skips the STAC wrapper around CMR and hits CMR directly, so doesn't have the problems you're experiencing.

I'm able to expand the search window to and get 237 results.

Please retry your cmr-stac request with the expanded monthly time range. Should be working now.

jhkennedy wrote:
> By adapting your script to use the
> [url=https://earthaccess.readthedocs.io/en/latest/]earthaccess python
> package[/url], you can get the search results you're looking for -- I've
> adapted your script to use earthaccess:
> https://gist.github.com/jhkennedy/3c3b181516801f0e66ba1a8e35c9f103
>
> This skips the STAC wrapper around CMR and hits CMR directly, so doesn't
> have the problems you're experiencing.
>
> I'm able to expand the search window to
> [code]start, stop = '2024-04-30', '2024-08-31'[/code]
> and get 237 results.

I can confirm that this works for me; thank you, jhkennedy.

I am considering skipping the STAC API if it adds another layer of complexity to CMR needlessly. I confess I don't have a sufficiently deep understanding to assess the relative merits of each approach.