wget only downloading partial file

[ermill99]

I'm using WGET to download L2 files to my server.  The files are showing up but seem to be capped at 13 kb.  Any idea why?

WGET Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=my-user--password='my-pw' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result

--2020-02-17 02:51:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA [following]
--2020-02-17 02:51:01--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'

     0K .......... ..                                           381K=0.03s

2020-02-17 02:51:01 (381 KB/s) - '/home/my_key/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png'
saved [12671] <----This appears to be the total file size downloaded.

[OB ODPS - jgwilding]

This usually indicates that an HTML file was downloaded.  If you're on a Linux platform, you can try the 'file' command on the saved file.  If it is HTML, you should be able to view it with a text viewer or in a browser.  A lot of times this ends up being the Earthdata login page and might mean your login credentials are not being accepted.

john

[gnwiii]

Rather than a partial download you may getting an html document you can view in a browser.   Try adding the wget --adjust-extension option.   You may need the --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies options.

[ermill99]

So, like this?

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --user=user--password='password' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

OR

wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --adjust-extension --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Postscript: I able was to open the file and it is indeed the login page.  I'm beginning to feel that there is no way to simply download a PNG file with WGET.

[ermill99]

I've tried all suggestions and just cannot get the single file I would like, to actually download.   

Are there any other suggestions to how to get this file : A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png to download non-interactively?

All that I am getting is the login page html downloaded to my server.

[gnwiii]

Wget (as well as wget2) also fails for me, but downloads succeed using a browser or the following curl incantation (from Download Methods):

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png

[ermill99]

here are the results of both methods and there appears to be no difference in outcome.  Still only getting the HTML file

1
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='PW' --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:50:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:50:02--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2'

     0K .......... ..                                          41.1M=0s

2020-02-17 18:50:02 (41.1 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.2' saved [12859]

2
Command
wget -P ~/www/saltwx.com/public_html/MODIS/sst/ --user=user--password='pw' --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge=on "https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png"

Result
--2020-02-17 18:40:01--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-02-17 18:40:08--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&response_type=code&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1'

     0K .......... ..                                          56.3M=0s

2020-02-17 18:40:08 (56.3 MB/s) - '/home/u250-zd3apyn5m53e/www/saltwx.com/public_html/MODIS/sst/A2020046181000.L2_LAC.S3470.nc.Southeast.sst.png.1' saved [12859]

[ermill99]

OK, the cURL seemed to download but I think it is still only delivering the HTML page.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 12859    0 12859    0     0  38157      0 --:--:-- --:--:-- --:--:-- 38157

[ermill99]

Confirmed that each method, WGET and cURL, result in a 12.859 KB file download. 

This authentication process is ridiculous.

[OB.DAACx - SeanBailey]

The cURL method requires a .netrc file (wget can also use this file, but you will need the --auth-no-challenge=on option) that contains your login Earthdata credentials.
Verify you have one, and it is correct, or generate one ala:

echo "machine urs.earthdata.nasa.gov login USERNAME password PASSWD" > ~/.netrc
chmod  0600 ~/.netrc

https://oceancolor.gsfc.nasa.gov/data/download_methods/ describes the process
Sean