Hello,
I am trying to use the API to automate download of DORIS ephemeris data.
I figured out how to make a query to list the files:
https://cmr.earthdata.nasa.gov/search/granules.json?collection_concept_id=C1602851234-CDDIS&page_size=2000&pretty=true&created_at=2022-01-01T00:00:00Z,2022-06-01T00:00:00Z
(Return 170 hits)
However some time periods fail, and i am not sure why:
https://cmr.earthdata.nasa.gov/search/granules.json?collection_concept_id=C1602851234-CDDIS&page_size=2000&pretty=true&created_at=2022-01-01T00:00:00Z,2022-09-01T00:00:00Z
(Maxes out at 2000 hits with most of them pre 2022)
Is this a bug?
Regards
API issue when searching for DORIS files
API issue when searching for DORIS files
Last edited by scmember on Sat Sep 28, 2024 11:30 am America/New_York, edited 1 time in total.
Filters:
-
CDDISx – RB
- Posts: 13
- Joined: Thu Sep 12, 2019 10:23 am America/New_York
- Location: NASA Goddard
- Contact:
Re: API issue when searching for files
Hello scmember,
To better assist you, we need to ask for a point of clarification. You stated that some time periods fail, but you also state that it maxes out at 2,000 hits, with most of them pre-2022.
So your search appears to have been successful, but that you ran into a max? Or were you specifically looking for the pre-2022 time periods for results?
Ross Bagwell
CDDIS
To better assist you, we need to ask for a point of clarification. You stated that some time periods fail, but you also state that it maxes out at 2,000 hits, with most of them pre-2022.
So your search appears to have been successful, but that you ran into a max? Or were you specifically looking for the pre-2022 time periods for results?
Ross Bagwell
CDDIS
Re: API issue when searching for files
(Deleted)
Last edited by scmember on Wed Sep 25, 2024 2:49 pm America/New_York, edited 2 times in total.
Re: API issue when searching for files
Upon closer inspection it looks like the query is working but it's filtering on the "updated" field.
I am looking through the API but struggling to figure out how to filter on "start_time".
Ideally, I would also like to be able to filter on object name and analysis center name.
API doc: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html
Any suggestions?
I am looking through the API but struggling to figure out how to filter on "start_time".
Ideally, I would also like to be able to filter on object name and analysis center name.
API doc: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html
Any suggestions?
Last edited by scmember on Wed Sep 25, 2024 2:51 pm America/New_York, edited 1 time in total.
Re: API issue when searching for DORIS files
I have completely given up on using the API...
In theory it is convenient but in practice I have encountered too many issues, including:
1) It only returns a small subset of files listed on the web.
2) Many of the files it lists do not exist.
3) The file sizes it return are often incorrect.
4) There is no obvious way to filter on fields, so user has to download large listings and filter manually.
I have reverted to scraping 100s of html pages to generate file lists.
I have resolved my issue with not being able to load some html pages due to this error:
"curl: (56) Recv failure: Connection was reset"
Solution was to update the default windows 10 curl to the latest version from:
8.7.1 (Windows) libcurl/8.7.1 Schannel zlib/1.3 WinIDN, Release-Date: 2024-03-27
to:
curl 8.10.1 (x86_64-w64-mingw32) libcurl/8.10.1 LibreSSL/3.9.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 ngtcp2/1.7.0 nghttp3/1.5.0 Release-Date: 2024-09-18
In theory it is convenient but in practice I have encountered too many issues, including:
1) It only returns a small subset of files listed on the web.
2) Many of the files it lists do not exist.
3) The file sizes it return are often incorrect.
4) There is no obvious way to filter on fields, so user has to download large listings and filter manually.
I have reverted to scraping 100s of html pages to generate file lists.
I have resolved my issue with not being able to load some html pages due to this error:
"curl: (56) Recv failure: Connection was reset"
Solution was to update the default windows 10 curl to the latest version from:
8.7.1 (Windows) libcurl/8.7.1 Schannel zlib/1.3 WinIDN, Release-Date: 2024-03-27
to:
curl 8.10.1 (x86_64-w64-mingw32) libcurl/8.10.1 LibreSSL/3.9.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 ngtcp2/1.7.0 nghttp3/1.5.0 Release-Date: 2024-09-18