Connections to oceandata.sci.gsfc.nasa.gov timing out!

[wahidmoufaddal]

Dear Dr Tommy / Sean

I am managing the operation of the MODIS DB ground station at ROPME in Kuwait. We experience some stuck in the automatic processing of the received MODIS-Terra data. Our internet browsers also are no longer able to connect to your new https connection at oceancolor and ocean data portals. Therefore we are not able to get necessary ancillary files to proceed with processing of Terra data. This stuck has started since switching of NASA to the https connection. We have tried to change our local scripts to tune with this change and also tried to check ssl, security certificates, etc but none of these trials led to any solution. We didn't experience any problem so far with processing of MODIS Aqua. I suspect that our IP has been blocked and that we have a similar case to the case of rkannenb at this forum: https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6437

Could you please check and advise about possible reasons of this stuck and possible solutions?. Your reply to this will be highly appreciated.
our IP address is: xx.xxx.xx.xx

Best wishes.

Wahid

[pfsmith]

Wahid,

We've been experiencing problems resulting from asymmetric network routes for several weeks, and some sites have experienced SSL issues since our government-mandated switch to https.  I was able to successfully traceroute to your IP this morning, and have seen connection attempts from your site in our firewall logs. If you could send us the output from a traceroute to oceancolor.gsfc.nasa.gov and try a "wget -d https://oceancolor.gsfc.nasa.gov/" we can try to identify whether it is a routing issue or a SSL issue.

Paul Smith

[wahidmoufaddal]

Dear Paul Smith

Thanks a lot for your response. Herein the inquired tracerouting info:

[apex@localhost ~]$ traceroute oceancolor.gsfc.nasa.gov
traceroute to oceancolor.gsfc.nasa.gov (xx.xxx.xx.xx), 30 hops max, 60 byte packets
1  xx.xxx.xx.xx (xx.xxx.xx.xx)  8.824 ms  8.993 ms  8.809 ms
2  xx.xxx.xx.xx (xx.xxx.xx.xx)  10.922 ms  11.007 ms  10.963 ms
3  xx.xxx.xx.xx (xx.xxx.xx.xx)  12.589 ms *  12.638 ms
4  skb-ace-10g62VL401.fasttelco.net (xx.xxx.xx.xx)  12.761 ms  12.659 ms  12.704 ms
5  kdc2-10g000-x-skb-ace.fasttelco.net (xx.xxx.xx.xx)  12.762 ms  12.780 ms  13.207 ms
6  ix-pos-2-1-1.core1.JSD-Jeddah.as6453.net (xx.xxx.xx.xx)  40.106 ms  43.684 ms  43.662 ms
7  if-xe-11-1-1-50.tcore2.WYN-Marseille.as6453.net (xx.xxx.xx.xx)  99.274 ms  99.401 ms  99.279 ms
8  if-ae-2-2.tcore1.WYN-Marseille.as6453.net (xx.xxx.xx.xx)  99.404 ms  99.423 ms  99.482 ms
9  if-ae-8-1600.tcore1.PYE-Paris.as6453.net (xx.xxx.xx.xx)  98.482 ms  98.440 ms  98.567 ms
10  if-ae-11-2.tcore1.PVU-Paris.as6453.net (xx.xxx.xx.xx)  98.702 ms  99.206 ms  99.219 ms
11  xx.xxx.xx.xx (xx.xxx.xx.xx)  130.260 ms  130.201 ms  130.299 ms
12  * * *
13  SIMS-INC.ear2.Washington1.Level3.net (xx.xxx.xx.xx)  1321.139 ms  1320.505 ms  1320.567 ms
14  xx.xxx.xx.xx (xx.xxx.xx.xx)  1303.351 ms  1303.374 ms  1303.343 ms
15  xx.xxx.xx.xx (xx.xxx.xx.xx)  1303.889 ms  1303.886 ms  1303.445 ms
16  xx.xxx.xx.xx (xx.xxx.xx.xx)  1312.497 ms  1312.525 ms  1312.438 ms
17  rtr-s28-hecn-test.sci.gsfc.nasa.gov (xx.xxx.xx.xx)  1441.055 ms  1441.159 ms  1440.992 ms
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

C:UsersWahid>tracert -4 oceancolor.gsfc.nasa.gov
Tracing route to oceancolor.sci.gsfc.nasa.gov [xx.xxx.xx.xx]
over a maximum of 30 hops:

  1     1 ms     1 ms     3 ms  xx.xxx.xx.xx
  2     2 ms     2 ms     2 ms  xx.xxx.xx.xx
  3    10 ms     3 ms     3 ms  xx.xxx.xx.xx
  4     6 ms     3 ms     3 ms  skb-ace-10g62VL401.fasttelco.net [xx.xxx.xx.xx]
  5     5 ms     5 ms     7 ms  kdc2-10g000-x-skb-ace.fasttelco.net [xx.xxx.xx.xx]
  6    33 ms    33 ms    34 ms  ix-pos-2-1-1.core1.JSD-Jeddah.as6453.net [xx.xxx.xx.xx]
  7    91 ms   105 ms    94 ms  if-xe-11-1-1-50.tcore2.WYN-Marseille.as6453.net [xx.xxx.xx.xx]
  8    97 ms   145 ms    93 ms  if-ae-2-2.tcore1.WYN-Marseille.as6453.net [xx.xxx.xx.xx]
  9    89 ms    94 ms    94 ms  if-ae-8-1600.tcore1.PYE-Paris.as6453.net [xx.xxx.xx.xx]
10    91 ms    92 ms    89 ms  if-ae-11-2.tcore1.PVU-Paris.as6453.net [xx.xxx.xx.xx]
11   122 ms   122 ms   122 ms  xx.xxx.xx.xx
12     *        *      193 ms  ae-2-3601.ear2.Washington1.Level3.net [xx.xxx.xx.xx]
13     *     1612 ms  1469 ms  SIMS-INC.ear2.Washington1.Level3.net [xx.xxx.xx.xx]
14  1340 ms  1206 ms  1351 ms  xx.xxx.xx.xx
15  1268 ms  1247 ms  1358 ms  xx.xxx.xx.xx
16  1407 ms  1443 ms  1455 ms  xx.xxx.xx.xx
17  1246 ms  1205 ms  1250 ms  rtr-s28-hecn-test.sci.gsfc.nasa.gov [xx.xxx.xx.xx]
18  1321 ms  1386 ms  1399 ms  oceancolor.sci.gsfc.nasa.gov [xx.xxx.xx.xx]

Trace complete.

[apex@localhost ~]$ wget -d https://oceancolor.gsfc.nasa.gov/
DEBUG output created by Wget 1.13.4 on linux-gnu.

URI encoding = `UTF-8'
--2017-01-12 05:53:56--  https://oceancolor.gsfc.nasa.gov/
Resolving oceancolor.gsfc.nasa.gov (oceancolor.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::44
Caching oceancolor.gsfc.nasa.gov => xx.xxx.xx.xx 2001:4d0:2418:128::44
Connecting to oceancolor.gsfc.nasa.gov (oceancolor.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
Created socket 3.
Releasing 0x08e97cb8 (new refcount 1).
Initiating SSL handshake.
SSL handshake failed.
Closed fd 3
Unable to establish SSL connection.


I will be highly indebted if you can advise how to deal with this stuck.
Looking forward to hearing from you.
Best wishes.

Wahid

[gnwiii]

Wget 1.13.4 is quite old, so there is a good chance it was built an old version of openssl or gnutls that lacks (Mozilla Modern) crypto support required with many sites that enforce https. The two most common crypto libraries on linux are openssl and gnutls, so it might be useful to check that ciphers required by NASA are supported, e.g.,:

$ gnutls-cli -l  | egrep "ECDHE_ECDSA_CHACHA20_POLY1305|ECDHE_ECDSA_AES_256_GCM_SHA384|ECDHE_ECDSA_AES_128_GCM_SHA256|ECDHE_ECDSA_AES_256-SHA384|ECDHE_ECDSA_AES_128_SHA256"
TLS_ECDHE_ECDSA_AES_128_GCM_SHA256                  0xc0, 0x2b  TLS1.2
TLS_ECDHE_ECDSA_AES_256_GCM_SHA384                  0xc0, 0x2c  TLS1.2
TLS_ECDHE_ECDSA_CHACHA20_POLY1305                   0xcc, 0xa9  TLS1.2

$ openssl ciphers -tls -v 'HIGH:!ADH:!MD5:@STRENGTH' | egrep "ECDHE-ECDSA-CHACHA20-POLY1305|ECDHE-ECDSA-AES256-GCM-SHA384|ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-AES256-SHA384|ECDHE-ECDSA-AES128-SHA256"
ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256

[pfsmith]

Wahid,

Thanks for the traceroute and wget information.  Comparing that to traceroutes from us to you shows a definite asymmetry.  I'll pass that on to our ISP for diagnosis.

The recent response concerning old versions of wget is a good point. We've had several cases where older versions of wget do not support the ciphers or certificates that our site does. Unfortunately the older ciphers have known weaknesses that we need to avoid.

As an alternative, if you have access to the curl utility, you could try "curl -IL --verbose https://oceancolor.gsfc.nasa.gov" as a debug tool. It may give you more information about the SSL connection.

Paul

[pfsmith]

Wahid,

Our ISP checked the routing and found that it is not the cause of the connection problem. That shifts the focus into establishing the SSL connection. Perhaps a newer version of wget, linked against the openssl library, will be successful. It comes down to what ciphers a given version of the application supports. You might also try the curl utility if it is available. If you are successful in reaching us via a current web browser from that system, that eliminates all of the network as the cause of the problem. I realize that the switch to https has caused difficulties for a number of sites

Paul