Page 1 of 1

Search with wget won't work

Posted: Mon Oct 24, 2022 8:38 pm America/New_York
by oceanwatchcp
Tried the example for command line interface provided on the "Search and Download" page:

wget -q --post-data="sensor=octs&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&addurl=1&results_as_file=1&search=*DAY_CHL*" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Got an error message:
<h1>An error has occurred</h1>
Filename pattern invalid

Can I get help on how to modify the search? Thank you!

Re: Search with wget won't work

Posted: Tue Oct 25, 2022 8:57 am America/New_York
by OB.DAAC-EDL - amscott
Try: "O*DAY_CHL*"

wget -q --post-data="results_as_file=1&std_only1&subType1&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&search=O*DAY_CHL*&sensor_id=9" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Re: Search with wget won't work

Posted: Tue Oct 25, 2022 5:50 pm America/New_York
by oceanwatchcp
Thanks! This works!

How should I modify the search command to be able make it work in general? My goal is to get a list of files for daily and monthly MODIS Aqua chlorophyll-a data files, and check for new files to download (so that the data is up to date locally).

I tried to search the MODIS Aqua chlorophyll-a data, this is the command generated on the page after a manual search on the API. But it didn't work on my computer.

wget -q --post-data="results_as_file=1&std_only1&subType1&search=A2019*.L3m_MO_CHL_chlor_a_4km.nc" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Would you help me figure out how to modify this one and alike? Thank you very much!
Daisy

Re: Search with wget won't work

Posted: Wed Oct 26, 2022 9:25 pm America/New_York
by OB.DAAC-EDL - amscott
Hi Daisy,

Your wget command works for me. In general, there are instructions here: https://oceancolor.gsfc.nasa.gov/data/download_methods/
Screen Shot 2022-10-26 at 9.19.51 PM.png
Screen Shot 2022-10-26 at 9.19.51 PM.png (141.1 KiB) Not viewed yet
We will look into what went wrong with the API search. A new version with updated search capabilities is being developed so please stay tuned.

Re: Search with wget won't work

Posted: Thu Oct 27, 2022 3:56 pm America/New_York
by oceanwatchcp
Hi, thanks for helping me!

I used a python script with the call function for the wget command:
call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType1&search=A2019*.L3m_MO_CHL_chlor_a_4km.nc"','-O', '-', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

But I got the below error message in the output html page:
<h1>An error has occurred</h1>
Filename pattern invalid

It is strange that the same command won't work in the python script.

The earlier search after I modified it according to your suggestion worked with the same script:
call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType1&sdate=1996-11-01&edate=1997-01-01&dtype=L3b&search=O*DAY_CHL*&sensor_id=9"','-O','-','https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

Do you have an idea why and how I can fix it?

Thanks!
Daisy

Re: Search with wget won't work

Posted: Mon Oct 31, 2022 11:05 am America/New_York
by OB.DAAC-EDL - amscott
try: call(['wget','-q', '--post-data="results_as_file=1&std_only1&subType=1&sdate=1999-01-01&edate=1999-12-31&search=A*MO_CHL_*4km*"','-O', '-', 'https://oceandata.sci.gsfc.nasa.gov/api/file_search'])

Re: Search with wget won't work

Posted: Tue Nov 01, 2022 3:05 pm America/New_York
by oceanwatchcp
Thanks again!

I tried the command suggested, but I still got the same error message:

<h1>An error has occurred</h1>
Filename pattern invalid

I modified the search here and there, e.g., "subType1", "search=*MO_CHL*4km*"', but kept getting the same error message.

Do you have other suggestions how I can modify the commend?

Thank you,
Daisy

Re: Search with wget won't work

Posted: Tue Nov 01, 2022 3:19 pm America/New_York
by OB.DAAC-EDL - amscott
Hi Daisy,

The search just uses regex and you can try patterns until something works. There are still some differences between your search and mine. I would recommend matching the beginning pattern of the file type you are looking for (aqua, snpp) in addition to the product suite (chl, rrs) and, if desired, resolution (4km, 9km). You can see a listing of files in the direct data access area of our site and our filenaming conventions to become more familiar with valid options to request.

Alicia