Recent MODIS SST files w/no content

dbotis · by **dbotis** » Tue Jan 21, 2020 4:31 pm America/New_York

Hi,

Beginning on 1/15/2020, my subscription files are empty. My guess is that this is related to the recent Earthdata login requirement. I did update my wget script and created ~/.urs_cookies and ~/.netrc files with my password as in the "Data Download Methods" page https://oceancolor.gsfc.nasa.gov/data/download_methods/. However, the downloaded files are all still around 13kb in size and contain no data. Perhaps I'm not implementing the wget call correctly.

Here is the old wget command (subscription 1930):
wget --post-data="subID=1930&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -

Updated wget command:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --post-data="subID=1930&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -

Any ideas are appreciated. Thanks,
Dan

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Tue Jan 21, 2020 6:27 pm America/New_York

Dan,

You have the cookies on the wrong plate :grin:

There is no requirement for the file_search API, but on the files it identifies for downloading. Try:

wget --post-data="subID=1930&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies  --keep-session-cookies  -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -

Sean

7yl4r · by **7yl4r** » Wed Jan 22, 2020 11:10 am America/New_York

I've just tried using this updated wget command and am seeing the same problem as Dan.
The search works but the files that get downloaded are Earthdata Login html pages.


```
tylar@tylar-Latitude-XT3:~/test$ head AQUA_MODIS.20200119T183001.L2.SST.NRT.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Earthdata Login</title>
    <meta name="description" content="Earthdata Login">
```

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Wed Jan 22, 2020 11:41 am America/New_York

Do you have a valid .netrc file with the proper file permission? If not, the authentication will fail and you'll get the html page.

7yl4r · by **7yl4r** » Wed Jan 22, 2020 12:31 pm America/New_York

My .netrc looks good. Below I have redacted my user and password :


tylar@tylar-Latitude-XT3:/media/tylar/filestore/git-repos/imars/wv2-processing$ cat ~/.netrc
machine urs.earthdata.nasa.gov login ****** password ********

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Wed Jan 22, 2020 1:30 pm America/New_York

What about file permissions? It needs to be readable only by the user (-rw-------)

7yl4r · by **7yl4r** » Wed Jan 22, 2020 1:45 pm America/New_York

Yes permissions were set to 600 as specified in the instructions.


tylar@tylar-Latitude-XT3:~$ ls -lh .netrc 
-rw------- 1 tylar tylar 64 Jan 22 11:02 .netrc

7yl4r · by **7yl4r** » Wed Jan 22, 2020 5:37 pm America/New_York

I think the problem here may be that wget only looks for the netrc file after recieving a 401 and earthdata returns a 200 with login html instead.

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Wed Jan 22, 2020 6:05 pm America/New_York

Try deleting your ~/.urs_cookies file.

7yl4r · by **7yl4r** » Thu Jan 23, 2020 8:32 am America/New_York

Same problem when using no `.urs_cookies` and with and empty `.urs_cookies`.

Here is a section of the output showing the request coming back with a 302 and then a 200:


--2020-01-23 08:30:16--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020019183000.L2_LAC_OC.nc
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020019183000.L2_LAC_OC.nc [following]
--2020-01-23 08:30:17--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020019183000.L2_LAC_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-01-23 08:30:17--  https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: â€˜./A2020019183000.L2_LAC_OC.nc.1â€™

A2020019183000.L2_LAC_OC.nc.1            [ <=>                                                                ]  12.44K  --.-KB/s    in 0s      

2020-01-23 08:30:17 (72.3 MB/s) - â€˜./A2020019183000.L2_LAC_OC.nc.1â€™ saved [12736]