Earthdata Forum

The file_search API seems very slow with larger queries, should I expect this? For example, searching for "S*L3m_MO_CHL_chlor_a_9km.nc" (all monthly 9km L3m SeaWiFS chl) yesterday took 96s and returns 150ish file matches. In contrast, the same query conducted year-by-year, looping over all years 1997-2010 took 11.2s in total.

Is this expected? I note that the file search page suggests that queries of the form "S200312[0-9]*MLAC" may "take a while to process", but it's not clear what "form" is being referred to there. Is it the regexp digit match that is slow? Or the mid-string glob? In either case, the mid-string glob in my SeaWiFS example doesn't seem to be problematic when only a single year is being searched at a time, so I don't think this is the root issue.

Thanks for any pointers.

Query times can vary for a number of reasons, some of which are out of our control.  In your case however, you can do something to get a better response.  Be more specific.

Here's you original query:

$ curl -d "addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14190  100 14129    0    61    131      0  0:01:47  0:01:47 --:--:--  2877

Yep, kinda slow.  But likely you'll only ever need to do that one once, so not too bad in the grand scheme of things :razz:

But what if you limited the search to only Level 3 mapped data?
$ curl -d "dtype=L3m&addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14200  100 14129  100    71   6040     30  0:00:02  0:00:02 --:--:--  6068

Hey! 2 seconds, not too shabby!

But what if we go crazy and limit it to only SeaWiFS L3 mapped data:
$ curl -d "sensor=seawifs&dtype=L3m&addurl=1&results_as_file=1&search=S*L3m_MO_CHL_chlor_a_9km.nc" https://oceandata.sci.gsfc.nasa.gov/api/file_search -o test.lst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14215  100 14129  100    86  30125    183 --:--:-- --:--:-- --:--:-- 30309

If you blinked, you missed it...

Sean

benraymond
We are looking into the issue, a few pointers would be choosing a specific mission and or type in your case selecting the level3 smi option from the web interface would help or if you are using command line dtype=L3m and sensor=Seawifs.

Perfecto. Thanks!!

@dana - no need to look further on my behalf, specifying the extra details is perfectly feasible and looks like it will solve my issue. Ta.