HICO data download

[mkahru]

I am trying to download HICO level-2 data using a list of file links that I got from the Level1&2 browser but I get files with 0 length.
My scripts work with regular http_manifest.txt files but not for the list I get for HICO.
Can anybody give a script using either Curl or Wget for Windows?
Thanks!
Mati

[OB WebDev - norman]

Hi Mati,

Have you specifically registered to download HICO data?

https://oceancolor.gsfc.nasa.gov/registration/

If so, can you share a specific URL that is not working for you?

Regards,
Norman

[mkahru]

Hi Norman,A sample link is: https://oceandata.sci.gsfc.nasa.gov [/hico/getfile/H2014252205906.L2_ISS_OC.ncI ]can download the NC file when I manually copy this address to my browser. I have over 100 or these links and it would take a long time to do it manually. I cannot use wget or curl to download from a link like that. Can you give an example?Thanks!

[gnwiii]

Downloading data from GES DISC has instructions for using wget in windows.

[mkahru]

I am using wget 1.18 and I am downloading all kinds of gibberish (css, js, html, etc) but NOT the needed netCDF files.

[gnwiii]

The "gibberish" may provide details for the failure to retrieve your requested file.   It may become readable if you can load it in a browser (maybe rename to .html).   Wget has a number of debugging options that may be helpful.   When NASA moved to https there were problems with wget and curl built with older libraries that didn't handle the redirects or didn't support the (more secure) https configuration NASA is using.   At that time, many of the binaries you could download didn't work.  The current versions of wget from Msys2 or Cygwin should be OK.

Google found 64-bit wget version 1.20.3 zip archives at Windows binaries of GNU Wget. 

C:UsersseadasDesktop>wget --version
GNU Wget 1.20.3 built on mingw32.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

Wgetrc:
    C:Usersgwhiteeosdis.wgetrc (env)
    C:Usersgwhite/.wgetrc (user)
    /win32dev/misc/wget/out64/etc/wgetrc (system)
Compile:
    x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
    -DSYSTEM_WGETRC="/win32dev/misc/wget/out64/etc/wgetrc"
    -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
    -I../lib -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
Link:
    x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    -L/win32dev/misc/wget/out64/lib -L/win32dev/misc/wget/out64/lib
    -lmetalink -lunistring -liconv -L/win32dev/misc/wget/out64/lib
    -lpcre2-8 -lidn2 -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
    -L/win32dev/misc/wget/out64/lib -lz -L/win32dev/misc/wget/out64/lib
    -lpsl -lws2_32 -lunistring -lws2_32 -lole32 -lcrypt32 -lexpat
    -L/win32dev/misc/wget/out64/lib -lgpgme -lassuan -lws2_32
    -lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
    ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
    /win32dev/misc/wget/out64/lib/libiconv.a
    /win32dev/misc/wget/out64/lib/libunistring.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.


When I tried the EOSDIS incantation it gave a prompt for the password, but didn't appear to take the input so gave html for the EOSDIS registration page.   To avoid entering the password, create a file "eosdis.wgetrc" with the user name and password:
user=<EOSDIS user>
password=<EOSDIS password>


Then, omitting the user and password from the previous command line and setting WGETRC the download succeeds:
C:UsersseadasDesktop>set WGETRC=C:Usersseadaseosdis.wgetrc
C:UsersseadasDesktop>wget --load-cookies C:Usersseadas.urs_cookies --save-cookies C:Usersseadas.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
--2019-05-26 06:07:05--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw [following]
--2019-05-26 06:07:05--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/restrict?code=390b6a1a3881dc2a0c3574e0eff6307ad92e71dd4d3d99efc35cabb90c00adec
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /hico/getfile/H2014252205906.L2_ISS_OC.nc [following]
--2019-05-26 06:07:06--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014252205906.L2_ISS_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 50612245 (48M) [application/octet-stream]
Saving to: 'H2014252205906.L2_ISS_OC.nc'

H2014252205906.L2_ISS_OC.nc   100%[=================================================>]  48.27M  1.82MB/s    in 11s

2019-05-26 06:08:19 (4.59 MB/s) - 'H2014252205906.L2_ISS_OC.nc' saved [50612245/50612245]


This version of wget "worked for me" on both Windows 7 and Windows 10.