Page 1 of 2
Recent MODIS SST files w/no content
Posted: Tue Jan 21, 2020 4:31 pm America/New_York
by dbotis
Hi,
Beginning on 1/15/2020, my subscription files are empty. My guess is that this is related to the recent Earthdata login requirement. I did update my wget script and created ~/.urs_cookies and ~/.netrc files with my password as in the "Data Download Methods" page
https://oceancolor.gsfc.nasa.gov/data/download_methods/. However, the downloaded files are all still around 13kb in size and contain no data. Perhaps I'm not implementing the wget call correctly.
Here is the old wget command (subscription 1930):
wget --post-data="subID=1930&addurl=1&results_as_file=1" -O -
https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -
Updated wget command:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --post-data="subID=1930&addurl=1&results_as_file=1" -O -
https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -
Any ideas are appreciated. Thanks,
Dan
Recent MODIS SST files w/no content
Posted: Tue Jan 21, 2020 6:27 pm America/New_York
by OB.DAACx - SeanBailey
Dan,
You have the cookies on the wrong plate :grin:
There is no requirement for the file_search API, but on the files it identifies for downloading. Try:
wget --post-data="subID=1930&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search | wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies -P /home1/scratch/epa/satellite/modis/GOM/L2_MODA_sub/ -i -
Sean
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 11:10 am America/New_York
by 7yl4r
I've just tried using this updated wget command and am seeing the same problem as Dan.
The search works but the files that get downloaded are Earthdata Login html pages.
```
tylar@tylar-Latitude-XT3:~/test$ head AQUA_MODIS.20200119T183001.L2.SST.NRT.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Earthdata Login</title>
<meta name="description" content="Earthdata Login">
```
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 11:41 am America/New_York
by OB.DAACx - SeanBailey
Do you have a valid .netrc file with the proper file permission? If not, the authentication will fail and you'll get the html page.
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 12:31 pm America/New_York
by 7yl4r
My .netrc looks good. Below I have redacted my user and password :
tylar@tylar-Latitude-XT3:/media/tylar/filestore/git-repos/imars/wv2-processing$ cat ~/.netrc
machine urs.earthdata.nasa.gov login ****** password ********
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 1:30 pm America/New_York
by OB.DAACx - SeanBailey
What about file permissions? It needs to be readable only by the user (-rw-------)
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 1:45 pm America/New_York
by 7yl4r
Yes permissions were set to 600 as specified in the instructions.
tylar@tylar-Latitude-XT3:~$ ls -lh .netrc
-rw------- 1 tylar tylar 64 Jan 22 11:02 .netrc
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 5:37 pm America/New_York
by 7yl4r
I think the problem here may be that wget only looks for the netrc file after recieving a 401 and earthdata returns a 200 with login html instead.
Recent MODIS SST files w/no content
Posted: Wed Jan 22, 2020 6:05 pm America/New_York
by OB.DAACx - SeanBailey
Try deleting your ~/.urs_cookies file.
Recent MODIS SST files w/no content
Posted: Thu Jan 23, 2020 8:32 am America/New_York
by 7yl4r
Same problem when using no `.urs_cookies` and with and empty `.urs_cookies`.
Here is a section of the output showing the request coming back with a 302 and then a 200:
--2020-01-23 08:30:16-- https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2020019183000.L2_LAC_OC.nc
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/A2020019183000.L2_LAC_OC.nc [following]
--2020-01-23 08:30:17-- https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2020019183000.L2_LAC_OC.nc
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country [following]
--2020-01-23 08:30:17-- https://urs.earthdata.nasa.gov/oauth/authorize?response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fob%2Fgetfile%2Frestrict&client_id=Z0u-MdLNypXBjiDREZ3roA&required_scope=study_area,country
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘./A2020019183000.L2_LAC_OC.nc.1’
A2020019183000.L2_LAC_OC.nc.1 [ <=> ] 12.44K --.-KB/s in 0s
2020-01-23 08:30:17 (72.3 MB/s) - ‘./A2020019183000.L2_LAC_OC.nc.1’ saved [12736]