Page 1 of 2
File Search broken ?
Posted: Tue Jun 14, 2022 4:09 am America/New_York
by dem1
Hi,
Since yesterday 2022-06-13 04:00:00PM UTC the file search API
https://oceandata.sci.gsfc.nasa.gov/api/file_search/ seems broken depending on search parameters:
for example when selecting Start=2022-05-06 End=2022-05-20 DataType=Level2 Mission=MODIS Aqua the search never ends, while with Start=2022-05-15 it succeed quickly. Note that the issue seems varying because 5 minutes ago Start=2022-05-07 was also quick, but now it blocks too.
Thanks,
Julien
Re: File Search broken ?
Posted: Tue Jun 14, 2022 6:36 am America/New_York
by dana.r.wilson
We are looking into the issue.
Re: File Search broken ?
Posted: Tue Jun 14, 2022 2:14 pm America/New_York
by mcoyne
If it helps, it seems to consistently break at result 1680 for me (or 131072 bytes). I haven't tested it, but it may work correctly for smaller searches. Each time I try a search I get the same error:
"Connection closed at byte 131072"
Re: File Search broken ?
Posted: Tue Jun 14, 2022 2:20 pm America/New_York
by dana.r.wilson
The issue has been resolved.
Re: File Search broken ?
Posted: Tue Jun 14, 2022 3:02 pm America/New_York
by mcoyne
File search is still breaking for me after receiving 131072 bytes.
Re: File Search broken ?
Posted: Tue Jun 14, 2022 3:09 pm America/New_York
by OB.DAACx - amscott
Hi,
Would you mind walking us through your steps so we can try to recreate your issue? Are you using the web interface or the command line implementation? When I try the web interface using any of the dates and information you supplied, I receive expected data. If command line, please share your command.
Re: File Search broken ?
Posted: Tue Jun 14, 2022 3:29 pm America/New_York
by mcoyne
Sure. I am using the command line implementation. Here's an example of code that was working last Friday, but does not work today:
wget --post-data="dtype=L3m&addurl=1&results_as_file=1&search=*DAY_*CHL_chlor_a_4km.nc&cksum=1" -O -
https://oceandata.sci.gsfc.nasa.gov/api/file_search
Sorry, that should be all one line, but I can't seem to get BBCode to turn "on" to format the code.
Re: File Search broken ?
Posted: Fri Jun 17, 2022 4:14 pm America/New_York
by dana.r.wilson
The issue has been resolved.
Re: File Search broken ?
Posted: Mon Jun 27, 2022 5:12 am America/New_York
by dem1
Hi,
The same problem came back since 25 June 10:00pm UTC, the search timeouts after 5 minutes depending on period (and the issue seems varying from one request to another).
Thanks,
Julien
Re: File Search broken ?
Posted: Mon Jun 27, 2022 6:01 am America/New_York
by dem1
Example of request which always fails:
Code: Select all
2022-06-27 11:41:32,495 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'aqua', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'A*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='oceandata.sci.gsfc.nasa.gov', port=443): Read timed out. (read timeout=300)
I tried to reproduce the timeout on the same request using the file search API webpage but I have no timeout (last time I were able to reproduce it).
On the other side the same request for Terra works well without timeout:
Code: Select all
2022-06-27 11:40:02,771 [INFO] session.request('POST', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search', data={'sensor': 'terra', 'sdate': '2021-12-09', 'edate': '2021-12-31', 'psdate': '2021-12-09', 'pedate': None, 'dtype': 'L2', 'search': 'T*.L2_LAC_OC.nc', 'std_only': 1, 'cksum': 1, 'results_as_file': '1', 'format': 'txt'})