Earthdata Forum

[this perhaps belongs here: https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=55068, but I was hesitant to post to a month old thread]Regarding the error:curl: (47) Maximum (50) redirects followedWe've been wrestling with this for a month or so, whereby our downloads fail after a seemingly random number of file acquisitions. This is the command we generally use:> curl -O -b ~/.urs_cookies3 -c ~/.urs_cookies3 -L -n --interface 2607:fe50:0:6330::112 --retry 5 --retry-delay 2 --max-time 0 --remote-name-all https://oceandata.sci.gsfc.nasa.gov (/ob/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /path/to/filelist.txt)}Where filelist.txt is simply a list of L1A/L2/anc files. Regardless of how long the list is, if a file eventually returns the error, all subsequent files will fail (see snipped of curl stdout, below).I watched this behavior today and noticed that all of the failures happen on the hour from the initial call. Of the ones I timed today, ~12 such downloads ran for exactly 1 hour, ~3 ran for exactly 2 hours, and 1 ran for 3 hours. I don't know if this helps diagnose the issue, but I hope so...-brian...[137/1029]: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218213500.L1A_LAC.bz2 --> A2011218213500.L1A_LAC.bz2--_curl_--https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218213500.L1A_LAC.bz2100  194M  100  194M    0     0  3895k      0  0:00:51  0:00:51 --:--:-- 3629k[138/1029]: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218214000.L1A_LAC.bz2 --> A2011218214000.L1A_LAC.bz2--_curl_--https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218214000.L1A_LAC.bz2100  214M  100  214M    0     0  3962k      0  0:00:55  0:00:55 --:--:-- 4004k[139/1029]: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218214500.L1A_LAC.bz2 --> A2011218214500.L1A_LAC.bz2--_curl_--https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218214500.L1A_LAC.bz2  0  214M    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0curl: (47) Maximum (50) redirects followed[140/1029]: https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218215000.L1A_LAC.bz2 --> A2011218215000.L1A_LAC.bz2--_curl_--https://oceandata.sci.gsfc.nasa.gov/ob/getfile/A2011218215000.L1A_LAC.bz2  0  214M    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0curl: (47) Maximum (50) redirects followed...

In our investigation of this problem last month, the solution provided by the Earthdata Login folks was to delete the cookies.  The cookies have an expiration date, and if your process passes an expired cookie, the authentication server says "try again"...so the client tries again with an expired cookie and the server says "try again"...eventually the client says "enough! taking my ball and going home"
...so, try deleting your cookies.

Sean

Got it, thanks.

Yes, we heeded that advice then - these batch curl commands were/are immediately preceded by a command to delete the cookie file (in the example above, > rm ~/.urs_cookies3, with that specific file only used once for downloads during any given time). As such, the cookie file is created during the curl call, and the 'expiration date' is exactly 1, 2, or 3 hours from when it was first created.?

On our end, we can work around this by having more numerous (and much smaller) batch download calls. This means more individual connections to NASA... We're ok with that if you are...
-brian

Brian,

I *think* the cookie lifetime is an hour, can't explain the 2 and 3 hour ones....maybe they're moving faster than light?
You can certainly run smaller batches.  Shouldn't be a problem.  We are working on a "app key" approach, which will eliminate the need to re-authenticate so frequently.  When that's in place you should be able to go back to the single firehose scenario.

Sean