Earthdata Forum

I'm needing to access data on shortwave radiation from a specific site in the years 2022-2024, but the GRIB-1 files stopped being updated in August 2024. Previously, I used an R package to download the data needed for that site within those dates, but now I'm trying to fill in the last couple months of 2024 with the new netCDF data.

I visited this site (https://disc.gsfc.nasa.gov/datasets/NLDAS_FORA0125_H_2.0/summary) and following a tutorial I clicked the Subset/Get Data button, used the GES DISC subsetter download method, specified the very small lat/long region coordinates, and selected the "Shortwave radiation flux downwards" variable. However, I'm having issues with 1. An extremely long list of download links, which I haven't tried using wget with yet, and 2. I tested downloading a couple files and opening them, but they only appear to contain metadata and no numbers I can actually use. How can I download data on shortwave radiation at a single site, and parse it into a useable dataframe in R?

Thank you in advance!

Hi Janette,

I will try to address each of your questions.

1. Long list of download links
Which tutorial are you following? There is a "Download Instructions" button that appears upon getting the list of URLs from the subsetting tool. The next step is to use this list of URLs to download all the data files in one step using wget. See this how-to for syntax: https://disc.gsfc.nasa.gov/information/howto?keywords=wget&title=How%20to%20Access%20GES%20DISC%20Data%20Using%20wget%20and%20curl

2. Files appear to only contain metadata
What program are you using to open the downloaded files? Since the files are in NetCDF format, viewing their contents requires specialized software. Panoply is free and easy to use for basic visualization, and programming languages like R and Python have packages/libraries for reading NetCDF files.

Panoply download link: https://www.giss.nasa.gov/tools/panoply/

As far as parsing with R, I would recommend using the ncdf4 and terra libraries. There is a good tutorial here: https://pjbartlein.github.io/REarthSysSci/netCDF.html

Hope this helps!

Jacob

Hi Jacob!

Thank you, that was extremely helpful! I worked through the tutorial and finally got the wget command to run. However, I don't think it ran correctly, as it instead downloaded each of the ~2,000 files individually into the root directory on my computer. I was expecting a single .nc file that I could read into R using the ncdf4 package. Instead, the file names are along the lines of "HTTP_services.cgi?FILENAME=%2Fdata..." with no extension at the end.

I'm following this tutorial: https://disc.gsfc.nasa.gov/information/howto?keywords=wget&title=How%20to%20Access%20GES%20DISC%20Data%20Using%20wget%20and%20curl
and using the command to download multiple data files at once when using subsetting services (step 4b) on a Mac.

Do you know where I might be going wrong?

I am not sure why you are getting ~2000 files. If you are trying to download three years of hourly data, that means there should be 3*365*24 = 26280 individual NetCDF files, each with one timestep. Of course, downloading and processing ~26000 files to get a timeseries at one ___location is not very efficient. I am sorry I did not think of this earlier, but we have a special tool for just this case - the Hydrology Data Rods web service. This allows you to download a time series of NLDAS data for a particular ___location as a text file. You can find this web service and instructions/examples at https://disc.gsfc.nasa.gov/information/tools?title=Hydrology%20Data%20Rods

You can download the data as a text file and then read them into R, or you can use an R function to directly access the data from R. Here is a function I wrote to do that (requires httr package):

get_NLDASv20_datarod <- function(start_date, end_date, lat, lon, var)
{
base_url <- "https://hydro1.gesdisc.eosdis.nasa.gov/daac-bin/access/timeseries.cgi"

full_url <- paste0(base_url,
"?variable=NLDAS2:NLDAS_FORA0125_H_v2.0:", var,
"&startDate=", start_date, "T00&",
"endDate=", end_date, "T23&",
"___location=GEOM:POINT(", lon, ",%20", lat, ")",
"&type=asc2"
)
print(full_url)

x <- GET(full_url)

# Parse response
z <- content(x, as = "text")
z1 <- strsplit(z, "\n")[[1]] # separate data by line

# Get dates and values. First 13 lines are metadata.
z1 <- z1[14:length(z1)]

# Split each entry by the tab separator into datetime and value
split_entries <- strsplit(z1, "\t")

# Extract datetime and value columns
thedatetime <- sapply(split_entries, function(x) x[1]) # First part is datetime
thevalue <- sapply(split_entries, function(x) as.numeric(x[2])) # Second part is value (convert to numeric)

# Create the data frame
df <- data.frame(datetime = as.POSIXct(thedatetime, format = "%Y-%m-%dT%H:%M:%S", tz = "UTC"), value = thevalue)

return(df)
}

Incredible, thank you so much!! That function and website were exactly what I needed for accessing the data between a specific timeframe and at a single lat-lon site. I can't thank you enough for your guidance and help!