Issues in accessing and using data from CDDIS
Posted: Sat Sep 28, 2024 4:12 am America/New_York
CDDIS provides access to lots of unique and valuable data.
However it is is often very difficult to find, filter, download and parse the data which in turn makes it difficult to integrate the data with other tools, as it often involves manual steps that can't be easily automated.
Two recurring issues include:
- Known and long-standing (at least 2023) download problems outside of US, resulting in curl error 56 "Recv failure: Connection was reset"
- The dependence on lookup tables that are not available in machine readable formats.
Here is an example that involves both issues:
To find and use SP3 data files user need to download and parse SP3c codes as a table.
However the table is only available in HTML form (which is difficult to parse due to many irregularities),
AND the html page is failing to download using curl from Australia (about 9 out of 10 times).
URL = https://cddis.nasa.gov/sp3c_satlist.html
Below is a link with a windows batch file which attempts to download the URL 20 times and 3 output files for each attempt: the file fragment, trace output, stdout
https://drive.google.com/file/d/1Q9zGjxyA1p5yOH4HjpT5WtLj5gLIT6rt/view?usp=sharing
Remarks:
- The download fails to download the full 91908b on 18 out of 20 attempts.
"curl: (56) Recv failure: Connection was reset"
- When downloading files from CDDIS, it is sometimes necessary to enable curl's continue option "-C -" which vastly reduces the number of required download re-tries. However this option is not available for html pages and causes this error on the second attempt: "curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume."
Requests to CDDIS:
- Please investigate and resolve the 'error 56' download issue.
- Please provide lookup tables in JSON or CSV as an alternative to the HTML format.
However it is is often very difficult to find, filter, download and parse the data which in turn makes it difficult to integrate the data with other tools, as it often involves manual steps that can't be easily automated.
Two recurring issues include:
- Known and long-standing (at least 2023) download problems outside of US, resulting in curl error 56 "Recv failure: Connection was reset"
- The dependence on lookup tables that are not available in machine readable formats.
Here is an example that involves both issues:
To find and use SP3 data files user need to download and parse SP3c codes as a table.
However the table is only available in HTML form (which is difficult to parse due to many irregularities),
AND the html page is failing to download using curl from Australia (about 9 out of 10 times).
URL = https://cddis.nasa.gov/sp3c_satlist.html
Below is a link with a windows batch file which attempts to download the URL 20 times and 3 output files for each attempt: the file fragment, trace output, stdout
https://drive.google.com/file/d/1Q9zGjxyA1p5yOH4HjpT5WtLj5gLIT6rt/view?usp=sharing
Remarks:
- The download fails to download the full 91908b on 18 out of 20 attempts.
"curl: (56) Recv failure: Connection was reset"
- When downloading files from CDDIS, it is sometimes necessary to enable curl's continue option "-C -" which vastly reduces the number of required download re-tries. However this option is not available for html pages and causes this error on the second attempt: "curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume."
Requests to CDDIS:
- Please investigate and resolve the 'error 56' download issue.
- Please provide lookup tables in JSON or CSV as an alternative to the HTML format.