Earthdata Forum

Hi,

I've developed some R code that accesses the hourly NLDAS data rods (ASCII) via url by generating links specifying the time frame and NLDAS cells needed for various data elements. I query these URLs in parallel to speed up the process, as it can be quite lengthy over a large time frame. One thing that I've struggled with in accessing the API is how to best structure my calls to the server to avoid timeouts - I find that at varying geographic scales (tens to hundreds of cells), time chunks of around 1,000 hours work well for the most part, but still time out occasionally. Today, I've encountered an issue I was previously unaware of, the NLDAS server said I had too many connections from my IP address. This was puzzling as my code parallelizes over only 7 cores, as I'm using my home computer and not a larger computing cluster. In general I can get around failed calls to the server with error handling in R, but figured I'd ask users here if there are any best practices to make this process as smooth as possible.

Thanks!

Hi Rishi,

With the exception of data stored in the cloud, we have a limit of 15 concurrent connections from the same IP. Try slowing down your requests or avoid launching too many requests at the same time.

There are tools you can use to handle timeouts and failed connections. For example, when using the command line utility wget, you can specify a rate limit with the wait, waitretry, and limit_rate arguments, and you can use the no-clobber flag to ensure that files that have already been downloaded are not re-downloaded. You may be able to implement something similar with your R code.

If you need further assistance, please let us know.