Earthdata Forum

Hello,

We teach a Sat. Oceanography lab every year and rely on the OceanColor data and Seadas for part of the lab during the semester. I am sniffing out problems, (we had a LOT of connectivity issues last year) and am experiencing download timeouts from both the L1&L2 browser, and the "Direct Data Access" files, which I ralize are the same CGI source...

I am working from a fuily up-to-date Centos 7 (7.4.1708) workstation. I have tried to make some sense of the openssl "cookbook" but it's pretty opaque.

The first test results in:

$ openssl s_client -connect oceancolor.gsfc.nasa.gov:443                   
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO ECC Domain Validation Secure Server CA
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Multi-Domain, CN = oceancolor.sci.gsfc.nasa.gov
verify return:1
---
Certificate chain
0 s:/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=oceancolor.sci.gsfc.nasa.gov
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Certification Authority
2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE5zCCBI6gAwIBAgIQYO599t+yPkKgiQdZJ7rZ7TAKBggqhkjOPQQDAjCBkDEL
.....lines deleted.......
QlpDJ2UMm0dBQSjVnqA6CmCtGxGq3K7S2AIgLY1SU/UogJgP3L7zGob4fLgx+Eve
VQ5C+P8Pr+V8fzw=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=oceancolor.sci.gsfc.nasa.gov
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3502 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 384 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx:
    Master-Key: 806E7C7CA724472CCA74FA29703C5053F6E7E4740B51E79CAEBC11F2B28E3E94F832FC67065366F887000907F5A31B35
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1521494303
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed


Using a browser, the download seems to start, and then times out ("Failed - network error" is all Chrome says). Using curl, I am seeing failures as shown below. If I wait several seconds I _might_ see success, or I might see a repeated failure. This was pretty much the behavior I fought last year. I can assure that a firewall at my end is NOT the issue. My attempts are from the 130.39.x.x ___domain, around 16:25 Central time. As I write this, I am seeing repeated successes and failures. I am simply waiting a few to many seconds, hitting up-arrow to repeat the "curl" command, so no scripts running at my end. As I understand the "throttle" I'd have to make multiple request PER second. The goal will be to have 8 students able to reliably download a granule (or two) for class use in teaching theory, etc. Any bulk data requests would be  handled through my own (lab admin) subscription, or by directing the student to register themselves for any research, etc.


$ curl -vvv -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)
*   Trying 2001:4d0:2418:128::84...
* Connected to oceandata.sci.gsfc.nasa.gov (2001:4d0:2418:128::84) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=oceancolor.sci.gsfc.nasa.gov,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
*   start date: Dec 14 00:00:00 2016 GMT
*   expire date: Dec 14 23:59:59 2019 GMT
*   common name: oceancolor.sci.gsfc.nasa.gov
*   issuer: CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB

> GET /cgi/getfile/A2018074193500.L1A_LAC.bz2 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: oceandata.sci.gsfc.nasa.gov
> Accept: */*
>

< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 19 Mar 2018 21:18:50 GMT
< Content-Type: application/octet-stream
< Content-Length: 231070539
< Connection: keep-alive
< Keep-Alive: timeout=60
< Last-Modified: Thu, 15 Mar 2018 20:54:37 GMT
< Content-Disposition: attachment; filename=A2018074193500.L1A_LAC.bz2
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
{ [data not shown]
34  220M   34 75.2M    0     0  6217k      0  0:00:36  0:00:12  0:00:24 6217k* SSL read: errno -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
36  220M   36 79.5M    0     0  6552k      0  0:00:34  0:00:12  0:00:22 6551k
* Closing connection 0
curl: (56) TCP connection reset by peer


Many thanks!
Alaric Haag
Systems Admin
LSU Earth Scan Laboratory

Adding more information: It's not just curl, wget exhibits the same behavior. It downloads a bit and then resets the connection.


$ wget  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
wget https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
--2018-03-20 11:41:48--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... 2001:4d0:2418:128::84, xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|2001:4d0:2418:128::84|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 231070539 (220M) [application/octet-stream]
Saving to: 'A2018074193500.L1A_LAC.bz2.1'

36% [===========================>                                                   ] 83,951,241  4.72MB/s   in 17s   

2018-03-20 11:42:06 (4.72 MB/s) - Read error at byte 83951241/231070539 (Connection reset by peer). [alaric@io Downloads]$


Traceroute gets to the router, but not the final node. Not sure if that's "nomral"...

$ traceroute oceandata.sci.gsfc.nasa.gov
traceroute to oceandata.sci.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
1  Howe-e241A-4006-dsw-1.net.lsu.edu (xx.xxx.xx.xx)  0.733 ms  1.043 ms  1.364 ms
2  mfcacsw-howedsw.cnet3.lsu.edu (xx.xxx.xx.xx)  0.263 ms  0.289 ms  0.270 ms
3  xx.xxx.xx.xx (xx.xxx.xx.xx)  0.413 ms xx.xxx.xx.xx (xx.xxx.xx.xx)  0.782 ms  0.931 ms
4  xx.xxx.xx.xx (xx.xxx.xx.xx)  0.544 ms  0.588 ms xx.xxx.xx.xx (xx.xxx.xx.xx)  0.558 ms
5  atha.bdr-csc.edge.frwl-cnet1.lsu.edu (xx.xxx.xx.xx)  1.346 ms  1.387 ms  1.419 ms
6  LONI-1427-LSUD-LONI.loni.org (xx.xxx.xx.xx)  0.990 ms  0.914 ms  0.914 ms
7  * * *
8  rtr.houh.net.internet2.edu-et-10-2-0.loni.org (xx.xxx.xx.xx)  1.182 ms  1.160 ms  1.145 ms
9  et-7-0-0.4079.rtsw.jack.net.internet2.edu (xx.xxx.xx.xx)  13.649 ms  13.747 ms  13.626 ms
10  et-3-3-0.4079.rtsw.atla.net.internet2.edu (xx.xxx.xx.xx)  19.454 ms  19.367 ms  19.515 ms
11  ae-4.4079.rtsw.wash.net.internet2.edu (xx.xxx.xx.xx)  32.273 ms  32.258 ms  32.344 ms
12  ae-1.4079.rtsw.ashb.net.internet2.edu (xx.xxx.xx.xx)  32.329 ms  32.387 ms  32.401 ms
13  et-11-3-0-1275.clpk-core.maxgigapop.net (xx.xxx.xx.xx)  33.452 ms  33.541 ms  33.524 ms
14  xx.xxx.xx.xx (xx.xxx.xx.xx)  35.799 ms  36.232 ms  36.207 ms
15  rtr-s28-hecn-40g.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  34.578 ms  34.281 ms  34.403 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *


And here are three tries, two successful, and the third failure. Is any host in 130.39.x.x showing a trigger of the throttling mechanism?

Many thanks!

Alaric

$ curl -vvv -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)
*   Trying 2001:4d0:2418:128::84...
* Connected to oceandata.sci.gsfc.nasa.gov (2001:4d0:2418:128::84) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=oceancolor.sci.gsfc.nasa.gov,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
*   start date: Dec 14 00:00:00 2016 GMT
*   expire date: Dec 14 23:59:59 2019 GMT
*   common name: oceancolor.sci.gsfc.nasa.gov
*   issuer: CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB



> GET /cgi/getfile/A2018074193500.L1A_LAC.bz2 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: oceandata.sci.gsfc.nasa.gov
> Accept: */*
>

< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 20 Mar 2018 18:51:28 GMT
< Content-Type: application/octet-stream
< Content-Length: 231070539
< Connection: keep-alive
< Keep-Alive: timeout=60
< Last-Modified: Thu, 15 Mar 2018 20:54:37 GMT
< Content-Disposition: attachment; filename=A2018074193500.L1A_LAC.bz2
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
{ [data not shown]
100  220M  100  220M    0     0  9136k      0  0:00:24  0:00:24 --:--:-- 9644k
* Connection #0 to host oceandata.sci.gsfc.nasa.gov left intact
[alaric@io Downloads]$ curl -vvv -O https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2018074193500.L1A_LAC.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to oceandata.sci.gsfc.nasa.gov port 443 (#0)
*   Trying 2001:4d0:2418:128::84...
* Connected to oceandata.sci.gsfc.nasa.gov (2001:4d0:2418:128::84) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=oceancolor.sci.gsfc.nasa.gov,OU=PositiveSSL Multi-Domain,OU=Domain Control Validated
*   start date: Dec 14 00:00:00 2016 GMT
*   expire date: Dec 14 23:59:59 2019 GMT
*   common name: oceancolor.sci.gsfc.nasa.gov
*   issuer: CN=COMODO ECC Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB

> GET /cgi/getfile/A2018074193500.L1A_LAC.bz2 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: oceandata.sci.gsfc.nasa.gov
> Accept: */*
>

< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 20 Mar 2018 18:52:28 GMT
< Content-Type: application/octet-stream
< Content-Length: 231070539
< Connection: keep-alive
< Keep-Alive: timeout=60
< Last-Modified: Thu, 15 Mar 2018 20:54:37 GMT
< Content-Disposition: attachment; filename=A2018074193500.L1A_LAC.bz2
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<
{ [data not shown]
35  220M   35 77.5M    0     0  6647k      0  0:00:33  0:00:11  0:00:22 6646k* SSL read: errno -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
37  220M   37 83.4M    0     0  6932k      0  0:00:32  0:00:12  0:00:20 6933k
* Closing connection 0
curl: (56) TCP connection reset by peer

See what-is-nss-error-5961-pr-connect-reset-error.    I too had this problem and was able to get improvements from IT after months of "the remote site must be down, our firewall is fine".    You may need to document the problem over a period of weeks.  If IT isn't helpful, you may need to raise the issue with higher level managers.   It is a big help making your case if you can show that the site is working for transfers to some external site at the same time it is failing from your regular connection.  Your IT people may have a non-firewalled connection that can be used for testing.

You are not tripping over any of our firewall rules and we see nothing odd in our logs coming from your IPs.
So, we believe that some machine between the client and our server is closing the connection on you, but it is not one of our servers.

As George suggests, you may want to contact the network administrators at your facility.

Sean

Many thanks to you both for the response!

I do have my campus network staff looking into it, and hope we can zero in on it quickly!

If you (or your network folks) need to poke someone on this end to help in the diagnosis, send an email to: connection_problems@oceancolor.sci.gsfc.nasa.gov

Regards,
Sean

Many (belated) thanks Sean! I’ve let my IT guys know.