problem with https

[melinfr]

Dear colleagues,
after the switch to https in december we encountered various problems, including in running the updates that could bring solutions - importantly we're operating behind a corporate http proxy. So this post might be related to the post https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?tid=6431 .
Below is an example with a proposed work-around.

Particularly we encountered problems downloading the 'luts' files.
The error on command "./install_ocssw.py --install-dir=/opt/seadas-7.3.2/ocssw --viirsn"  was:
....
Installing viirsn-luts (10 of 11)
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.
Error - Could not install luts for viirsn

To work around this issue we did the following changes to the file /run/scripts/modules/ProcUtils.py :

run/scripts/modules/ProcUtils.py:68
#    if proxy is None:
#        full_request = request
#    else:
#       full_request = ''.join(['http://', url, request])
    if proxy is not None:
        urlConn.set_tunnel(url, 443)

    try:
#       req = urlConn.request('GET', full_request, headers=reqHeaders)
        req = urlConn.request('GET', request, headers=reqHeaders)
    except:
        err_msg = '\n'.join(['Error! could not establish a network connection. Check your network connection.',
                             'If you do not find a problem, please try again later.'])
        sys.exit(err_msg)

With these changes we are able to download all the luts successfully. But of course any feedback is very much welcome.
Cheers,
F.

[ana_dogliotti]

Hello,

I have exactly the same problem, but I tried this work-around, but didn't work for me. I removed the proxy and had the same error...
Could be there any other problem?
Thanks

Ana

[OB.DAACx - SeanBailey]

Thanks :smile:
Another code modification was previously proposed and will be included in a soon-to-be-released update to SeaDAS.

Regards,
Sean

[mpenalver]

The installation script for SeaDAS up to the current version 7.5.3 uses the Python 2 httplib (or Python 3 http.client) library to download ancillary files via HTTPS (or HTTP). First, an HTTPS connection is created to the proxy with httplib.HTTPSConnection, and then a GET request for the relevant file located on the endpoint server is sent through that connection with httplib.HTTPSConnection.request.

In earlier versions of the library, the request had to include the URL of the endpoint, which was passed on to the proxy. However, in the version included in Python 2.7 (and 3.2) the new method httplib.HTTPSConnection.set_tunnel was introduced with which the endpoint must be specified for the connection and doesn't need to be included in every request sent through it. This change has not yet been incorporated into the SeaDAS installation code, which prevents the successful setup of proxied connections. Below, the required modifications to seadas-7.5.3/ocssw/scripts/modulesProcUtils.py.

httpinit:
        #=======================================================================
        # elif proxy.scheme == 'https':
        #     urlConn = hclient.HTTPSConnection(proxy.hostname, proxy.port,
        #                                      timeout=timeout)
        # else:
        #     urlConn = hclient.HTTPConnection(proxy.hostname, proxy.port,
        #                                      timeout=timeout)
        #=======================================================================
        else:
            if proxy.scheme == 'https':
                urlConn = hclient.HTTPSConnection(proxy.hostname, proxy.port,
                                                  timeout=timeout)
            else:
                urlConn = hclient.HTTPConnection(proxy.hostname, proxy.port,
                                                 timeout=timeout)
            urlConn.set_tunnel(url)

_httpdl:
#=======================================================================
#    if proxy is None:
#        full_request = request
#    else:
#        full_request = ''.join(['https://', url, request])
#
#    try:
#        req = urlConn.request('GET', full_request, headers=reqHeaders)
#=======================================================================
    try:
        req = urlConn.request('GET', request, headers=reqHeaders)

[OB.DAACx - SeanBailey]

Thanks :grin:
Looks like your changes are fine.  We will incorporate them in a future update.

Sean

[zhigang]

Exactly, the set_tunnel is very useful. I have been using this function to access the https connection.

Zhigang