Daymet OPENDAP

[rmcd]

I am accessing Daymet through the OPENDAP endpoint https://thredds.daac.ornl.gov/thredds/dodsC/daymet-v4-agg/na.ncml. I'm experiencing very slow download speeds. On the USGS VPN I'm getting ~2Kb/s and from home without VPN I'm getting ~5Mb/s. In my experience this is quite slow. I am interpolating daymet variables to Alaska's HUC12, and generally I have I request the data by year and by quarter to stay within the allowable size of the request. This has been fine in the past but now in addition to the slow download speeds and maybe because of it I'm also getting curl timeout errors. I changed the download request to monthly chunks and that is better but eventually I run into the same timeout errors. Just checking and asking to see if there is something on the server side that might be slowing the downloads down? For more context I'm using python and xarray to open and process the data via xarray using the OPENDAP endpoint linked above. Any help would be appreciated.

[ORNLx - mthornton]

Hi @rmcd ,
Thanks for your message. We'll check into this and provide input ASAP.
Michele

[ORNL - yaxingwei]

Hi @rmcd ,

Thanks for your message! To help better diagnose into this issue, could you please run a few download tests (see instruction below) and let us know how the speed will be in different cases?

Instruction: Please download (using curl or wget) the following two data files twice, first time with your VPN turned on and second time with your VPN turned off.
- https://daac.ornl.gov/daacdata/daymet/Daymet_Daily_V4R1/data/daymet_v4_daily_na_tmax_2023.nc
- https://data.ornldaac.earthdata.nasa.gov/protected/daymet/Daymet_Daily_V4R1/data/daymet_v4_daily_na_tmax_2023.nc

Note that downloading data from the above two URLs require Earthdata login. Please refer to the ORNL DAAC's guide (https://daac.ornl.gov/help/download/) on how to do that.

Feel free to let us know if you have any questions.

[rmcd]

Thank you very much for your help. Here are the responses. The first 2 are with VPN on and the second 2 are with VPN off. I exited the third as you can see. Results are in screenshot attachment.

[ORNL - yaxingwei]

Thank you @rmcd!

These are interesting results.

To give you some context info, URLs starting with https://daac.ornl.gov/ point to our on-prem data pool at ORNL and URLs starting with https://data.ornldaac.earthdata.nasa.gov point to our data pool on the AWS cloud. Also, accessing data from these URLs was directly file download. There was no service, like THREDDS, involved.

Seems there is some issue with the network connection between your computer and the ORNL server. It's weird to see the downloading speed from ORNL data pool was even slower when your USGS VPN was turned off. I see download speed from ORNL data pool was much slower than what you got from the THREDDS data server yesterday. This makes me feel the bottleneck is unlikely in the THREDDS data server, but in the network.

I'll share this information with our system expert and diagnose into it more. I'll keep you updated.

But at the same time, I'd like to suggest you try our OPeNDAP service newly hosted on AWS Cloud. The AWS OPeNDAP service for Daymet data follow the URL pattern below. You will need to change the spatial region (na, pr, or hi), variable name, and year to switch to another data file.
https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_na_tmax_2023.nc.dmr

Our on-prem THREDDS data server will be turned off in the next 12 months. We expect our current THREDDS users will switch to the AWS Cloud OPeNDAP (as described above) in the future. Currently, you will have to use index ranges (rather than a bounding box or time range) to subset data through the AWS Cloud OPeNDAP service and you have to switch from file to file (instead of an aggregated data like what you got from THREDDS). But we expect to improve its usability progressively.

Hope this helps and apologize for the inconvenience.

Best,
Yaxing

[rmcd]

Thank you Yaxing! I appreciate the help, I'm still a bit surprised by the speeds from my home network for the on-prem endpoint, while off our VPN. I re-ran the on-prem case this morning without the VPN and it was much faster, but still quite slow. It's good to know about the upcoming change to the OPENDAP and I can switch my workflow to use the AWS endpoint, which will greatly improve the processing time :). Thanks again for taking the time!

[rmcd]

Hello, I'm still having issues accessing daymet via OpenDAP. Currently I'm working on the Puerto Rico ___domain, but I think it's true for all the domains. I'm using xarray to open and load the data using the OpenDAP URL. I'm working with both of these endpoints.

"https://thredds.daac.ornl.gov/thredds/dodsC/daymet-v4-agg/puertorico.ncml"
and
"https://opendap.earthdata.nasa.gov/collections/C2532426483-ORNL_CLOUD/granules/Daymet_Daily_V4R1.daymet_v4_daily_{region}_{var}_{year}.nc"

the latter gives me better behavior but still gives me intermittent but often, errors, making it difficult to finish my data processing workflow. I've used the former URL quite a bit over the last 2-3 years with no issues, except for the last month or 2. I've worked on both USGS systems and at home, and there is no difference in this behavior, so I don't think it's a USGS network issue. I would very much like to avoid downloading all the data locally if possible.

I can provide a working example, including a ipynb file, data file, and conda env file, if that would help.

[ORNLx - mthornton]

Thank you for your message. Our TDS manager will be addressing your question soon.

[ORNL - yaxingwei]

Hi @rmcd , apologize for the delayed reply. I would appreciate you sharing with me a working example. I'd like to do further tests from my home network. If the files are small, you can email them directly to weiy@ornl.gov. Otherwise, please share them with me via Google Drive or a similar approach.

Best,
Yaxing

[rmcd]

Hello Yaxing, Thank you. I sent an email to you with a link to my google drive containing a zip folder with everything you need to run my daymet processing workflow. There is a README.md file with directions. Thanks again, Rich