CMR STAC GET requests are broken

[henrydevseed]

I believe this is the underlying cause of STAC-related issues that users are having in recent weeks:
https://forum.earthdata.nasa.gov/viewtopic.php?t=5953
https://forum.earthdata.nasa.gov/viewtopic.php?t=5922

POST requests to the /search endpoint are working properly, but GET requests are returning empty results when they clearly should be returning items in an ItemCollection.

For example, here is a request for some HLS items that returns the specified number of items (1) in an item collection:
[code]
!curl -X POST \
-H "Content-Type: application/json" \
-d '{ \
"bbox": [-105.55, 35.64, -105.31, 35.81], \
"datetime": "2024-01-01T00:00:00Z/2024-09-01T00:00:00Z", \
"collections": ["HLSL30_2.0", "HLSS30_2.0"],\
"limit": 1 \
}' \
https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search | jq
[!code]

Results (abbreviated):
[code]
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"id": "HLS.S30.T13SDV.2024003T174731.v2.0",
"stac_version": "1.0.0",
"stac_extensions": [
"https://stac-extensions.github.io/eo/v1.0.0/schema.json"
],
"properties": {
"datetime": "2024-01-03T17:54:08.034Z",
"eo:cloud_cover": 45,
"start_datetime": "2024-01-03T17:54:08.034Z",
"end_datetime": "2024-01-03T17:54:08.034Z"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-106.0982929,
35.1499121
],
[
-104.8928408,
35.1548427
],
[
-104.8915104,
36.1448493
],
[
-106.1119251,
36.1397366
],
[
-106.0982929,
35.1499121
]
]
]
},
"bbox": [
-106.1119251,
35.1499121,
-104.8915104,
36.1448493
],
"assets": {
...
}
},
"links": [
{
"rel": "self",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSS30_2.0/items/HLS.S30.T13SDV.2024003T174731.v2.0",
"type": "application/geo+json"
},
{
"rel": "parent",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSS30_2.0/",
"type": "application/geo+json"
},
{
"rel": "collection",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSS30_2.0/",
"type": "application/geo+json"
},
{
"rel": "root",
"href": "https://cmr.earthdata.nasa.gov/stac",
"type": "application/json"
},
{
"rel": "provider",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD",
"type": "application/json"
},
{
"rel": "via",
"href": "https://cmr.earthdata.nasa.gov/search/concepts/G2832232406-LPCLOUD.json",
"title": "CMR JSON metadata for item",
"type": "application/json"
},
{
"rel": "via",
"href": "https://cmr.earthdata.nasa.gov/search/concepts/G2832232406-LPCLOUD.umm_json",
"title": "CMR UMM_JSON metadata for item",
"type": "application/vnd.nasa.cmr.umm+json"
}
],
"collection": "HLSS30_2.0"
}
],
"links": [
{
"rel": "self",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search",
"type": "application/geo+json",
"title": "This search"
},
{
"rel": "root",
"href": "https://cmr.earthdata.nasa.gov/stac",
"type": "application/json",
"title": "Root Catalog"
},
{
"rel": "parent",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD",
"type": "application/json",
"title": "Provider Catalog"
},
{
"rel": "first",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55%2C35.64%2C-105.31%2C35.81&datetime=2024-01-01T00%3A00%3A00Z%2F2024-09-01T00%3A00%3A00Z&collections=HLSL30_2.0%2CHLSS30_2.0&limit=1",
"type": "application/geo+json",
"title": "First page of results"
},
{
"rel": "next",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55%2C35.64%2C-105.31%2C35.81&datetime=2024-01-01T00%3A00%3A00Z%2F2024-09-01T00%3A00%3A00Z&collections=HLSL30_2.0%2CHLSS30_2.0&limit=1&cursor=eyJqc29uIjoiW1wibHBjbG91ZFwiLDE3MDQzMDQ0NDgwMzQsMjgzMjIzMjQwNl0iLCJ1bW0iOiJbXCJscGNsb3VkXCIsMTcwNDMwNDQ0ODAzNCwyODMyMjMyNDA2XSJ9",
"type": "application/geo+json",
"title": "Next page of results"
}
],
"context": {
"returned": 1,
"limit": 1,
"matched": 77
}
}
]
[/code]

If a client tries to use the link to the next page of results ("rel": "next"), it returns an empty ItemCollection when it should clearly return at least one result:
[code]
!curl -X GET "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55%2C35.64%2C-105.31%2C35.81&datetime=2024-01-01T00%3A00%3A00Z%2F2024-09-01T00%3A00%3A00Z&collections=HLSL30_2.0%2CHLSS30_2.0&limit=1&cursor=eyJqc29uIjoiW1wibHBjbG91ZFwiLDE3MDQzMDQ0NDgwMzQsMjgzMjIzMjQwNl0iLCJ1bW0iOiJbXCJscGNsb3VkXCIsMTcwNDMwNDQ0ODAzNCwyODMyMjMyNDA2XSJ9" | jq
[\code]

Result:
[code]
{
"type": "FeatureCollection",
"features": [],
"links": [
{
"rel": "self",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55%2C35.64%2C-105.31%2C35.81&collections=HLSL30_2.0%2CHLSS30_2.0&cursor=eyJqc29uIjoiW1wibHBjbG91ZFwiLDE3MDQzMDQ0NDgwMzQsMjgzMjIzMjQwNl0iLCJ1bW0iOiJbXCJscGNsb3VkXCIsMTcwNDMwNDQ0ODAzNCwyODMyMjMyNDA2XSJ9&datetime=2024-01-01T00%3A00%3A00Z%2F2024-09-01T00%3A00%3A00Z&limit=1",
"type": "application/geo+json",
"title": "This search"
},
{
"rel": "root",
"href": "https://cmr.earthdata.nasa.gov/stac",
"type": "application/json",
"title": "Root Catalog"
},
{
"rel": "parent",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD",
"type": "application/json",
"title": "Provider Catalog"
},
{
"rel": "first",
"href": "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55%2C35.64%2C-105.31%2C35.81&collections=HLSL30_2.0%2CHLSS30_2.0&datetime=2024-01-01T00%3A00%3A00Z%2F2024-09-01T00%3A00%3A00Z&limit=1",
"type": "application/geo+json",
"title": "First page of results"
}
],
"context": {
"returned": 0,
"limit": 1,
"matched": 0
}
}
[\code]

Most STAC client applications use the paged result link structure to iterate through search results, which is why many users are experiencing issues when running searches that should return more than the default limit (20).

[ircwaves]

I believe this is the underlying cause of STAC-related issues that users are having in recent weeks:
viewtopic.php?t=5953
viewtopic.php?t=5922

.... snipped ...

Most STAC client applications use the paged result link structure to iterate through search results, which is why many users are experiencing issues when running searches that should return more than the default limit (20).

Agreed, and I've posted on that first thread, and opened an issue w/ reproduction scenario over on the cmr-stac github.

[waltersdan]

Following

[b.coerver]

Have the same problem, following.

[ASDC - ingridgs]

All,

CMR STAC issues are currently being worked on.

CMR STAC last announcement was:

To: Users of the CMR Spatio Temporal Asset Catalogs (STAC)
Subject: CMR STAC To Use ‘EntryId’ instead of ‘shortname’.’v’versionId’ as the Identifier and Search Parameter
What’s Happening: An updated version of CMR-STAC will be deployed on Wednesday, September 4, 2024 to use ‘EntryId’ instead of ‘shortname’.’v’’versionId’, which replaces the current ‘.v’ delimiter with a ‘_’. Therefore, the ‘EntryId’ will now be in the form ‘shortname_versionId’. For example, an old link in PROD that was represented as https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30.v2.0 will now be represented as https://cmr.earthdata.nasa.gov/stac/LPCLOUD/collections/HLSL30_2.0.
Action Needed: Users of CMR-STAC, including those who use Jupyter notebooks will have to update them to use the new ‘EntryId’ format.
Who is Affected: Users of CMR-STAC
Support: If you have further questions, please contact us at support@earthdata.nasa.gov or in NASA Slack at ‘cmr-stac’.

[aaleman]

by ircwaves » 24 Sep 2024 13:04

henrydevseed wrote: ↑24 Sep 2024 12:08
I believe this is the underlying cause of STAC-related issues that users are having in recent weeks:
viewtopic.php?t=5953
viewtopic.php?t=5922

.... snipped ...

Most STAC client applications use the paged result link structure to iterate through search results, which is why many users are experiencing issues when running searches that should return more than the default limit (20).

Agreed, and I've posted on that first thread, and opened an issue w/ reproduction scenario over on the cmr-stac github.

A fix for this issue has been pushed to the CMR-STAC production environment. Please retest and let us know if you're still running into the problem.

[henrydevseed]

aaleman wrote:

> A fix for this issue has been pushed to the CMR-STAC production
> environment. Please retest and let us know if you're still running into the
> problem.

GET requests for more than one collection are still broken.

This request for HLSL30_2.0 returns 30 results:
curl -X GET "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55,35.64,-105.31,35.81&datetime=2024-01-01T00:00:00Z/2024-09-24T00:00:00Z&limit=1&collections=HLSS30_2.0&collections=HLSL30_2.0"

This request for HLSS30_2.0 returns 52 results:
curl -X GET "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55,35.64,-105.31,35.81&datetime=2024-01-01T00:00:00Z/2024-09-24T00:00:00Z&limit=1&collections=HLSS30_2.0&collections=HLSS30_2.0"

This request for HLSL30_2.0 and HLSS30_2.0 should return 82 results but it returns an empty item collection instead:
curl -X GET "https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55,35.64,-105.31,35.81&datetime=2024-01-01T00:00:00Z/2024-09-24T00:00:00Z&limit=1&collections=HLSL30_2.0,HLSS30_2.0"

[ASDC - ingridgs]

Hello @henrydevseed,


Please get in touch with support@earthdata.nasa.gov via email.

Thank you,

[wvalenci]

Hello henrydevseed

https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?bbox=-105.55,35.64,-105.31,35.81&datetime=2024-01-01T00:00:00Z/2024-09-24T00:00:00Z&limit=1&collections=HLSL30_2.0,HLSS30_2.0

is now returning 82 matches. Please try it again.

William

[aaleman]

Apologies for the cross-posts, but I want to reach as many of you as possible.

An update regarding the communication of changes and releases, as well as reporting and tracking issues for CMR-STAC...

• Release notes are now available in the CMR-STAC GitHub repo: https://github.com/nasa/cmr-stac/releases. The production release cadence for CMR-STAC is typically every two weeks on Wednesday, and release notes will be posted 2 weeks ahead of scheduled release dates. The next scheduled release to production is 11/20/2024.

• The CMR-STAC GitHub repo will also be The Place for reporting and tracking the status of issues and feature requests going forward. You can check the current list of open issues here: https://github.com/nasa/cmr-stac/issues.

I'm hopeful this streamlined approach to communications will make it easier for everyone to report and track the status of issues, fixes, and new feature requests for CMR-STAC. Please reach out if you have questions or concerns.