Issues in accessing and using data from CDDIS

[scmember]

CDDIS provides access to lots of unique and valuable data.

However it is is often very difficult to find, filter, download and parse the data which in turn makes it difficult to integrate the data with other tools, as it often involves manual steps that can't be easily automated.

Two recurring issues include:
- Known and long-standing (at least 2023) download problems outside of US, resulting in curl error 56 "Recv failure: Connection was reset"
- The dependence on lookup tables that are not available in machine readable formats.

Here is an example that involves both issues:
To find and use SP3 data files user need to download and parse SP3c codes as a table.
However the table is only available in HTML form (which is difficult to parse due to many irregularities),
AND the html page is failing to download using curl from Australia (about 9 out of 10 times).
URL = https://cddis.nasa.gov/sp3c_satlist.html

Below is a link with a windows batch file which attempts to download the URL 20 times and 3 output files for each attempt: the file fragment, trace output, stdout
https://drive.google.com/file/d/1Q9zGjxyA1p5yOH4HjpT5WtLj5gLIT6rt/view?usp=sharing

Remarks:
- The download fails to download the full 91908b on 18 out of 20 attempts.
"curl: (56) Recv failure: Connection was reset"
- When downloading files from CDDIS, it is sometimes necessary to enable curl's continue option "-C -" which vastly reduces the number of required download re-tries. However this option is not available for html pages and causes this error on the second attempt: "curl: (33) HTTP server doesn't seem to support byte ranges. Cannot resume."

Requests to CDDIS:
- Please investigate and resolve the 'error 56' download issue.
- Please provide lookup tables in JSON or CSV as an alternative to the HTML format.

[scmember]

I have found a workaround for the issue with html pages and files not loading completely using default version of curl supplied with Windows 10.

The solution was to manually update curl to latest version.
Note that all my Win10 browsers still do not work and load html pages partially!

Example URLs:
https://cddis.nasa.gov/archive/doris/products/orbits/lca/en1/
https://cddis.nasa.gov/archive/doris/products/orbits/lca/en1/*?list

These fail:
- Latest Chrome 129.0.6668.71 (Official Build) (64-bit)
- Latest Edge 129.0.2792.65 (Official build) (64-bit)
- Latest Firefox 130.0.1 (64-bit)
- Default Win10 curl 8.7.1 (Windows) libcurl/8.7.1 Schannel zlib/1.3 WinIDN, Release-Date: 2024-03-27

These work:
- Latest Win10 curl 8.10.1 (x86_64-w64-mingw32) libcurl/8.10.1 LibreSSL/3.9.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 ngtcp2/1.7.0 nghttp3/1.5.0 Release-Date: 2024-09-18
- Default win10 WSL curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16 Release-Date: 2022-01-05

Google suggests this is could be an issue with how SSL is handled.

CDDIS please investigate further and add support for default version of Win10 curl.

[miketaku111]

scmember wrote:
> CDDIS provides access to lots of unique and valuable data.
>
> However it is is often very difficult to find, filter, download and parse
> the data which in turn makes it difficult to integrate the data with other
> tools, as it often involves manual steps that can't be easily automated.
>
> Two recurring issues include:
> - Known and long-standing (at least 2023) download problems outside of US,
> resulting in curl error 56 "Recv failure: Connection was reset"
> - The dependence on lookup tables that are not available in machine
> readable formats.
>
> Here is an example that involves both issues:
> To find and use SP3 data files user need to download and parse SP3c codes
> as a table.
> However the table is only available in HTML form (which is difficult to
> parse due to many irregularities),
> AND the html page is failing to download using curl from Australia (about 9
> out of 10 times).
> URL = https://cddis.nasa.gov/sp3c_satlist.html
>
> Below is a link with a windows batch file which attempts to download the
> URL 20 times and 3 output files for each attempt: the file fragment, trace
> output, stdout
>
> https://drive.google.com/file/d/1Q9zGjxyA1p5yOH4HjpT5WtLj5gLIT6rt/view?usp=sharing
>
> Remarks:
> - The download fails to download the full 91908b on 18 out of 20 attempts.
> "curl: (56) Recv failure: Connection was reset"
> - When downloading files from CDDIS, it is sometimes necessary to enable
> curl's continue option "-C -" which vastly reduces the number of
> required download re-tries. However this option is not available for html
> pages and causes this error on the second attempt: "curl: (33) HTTP
> server doesn't seem to support byte ranges. Cannot resume."
>
> Requests to CDDIS:
> - Please investigate and resolve the 'error 56' download issue.
> - Please provide lookup tables in JSON or CSV as an alternative to the HTML
> format.

How can CDDIS resolve the 'error 56' download issue and offer lookup tables in machine-readable formats like JSON or CSV instead of HTML?