Earthdata Forum

I'm pretty sure there is an issue with the level 2a GPM DPR radar data when downloaded from GES DISC and ArthurHou. I am attempting to convert the HDF5 data to GeoTiff using a simple gdal command

gdal_translate -of GTiff HDF5:"input_file.HDF5"://FS/CSF/flagShallowRain output_file.tif

When I convert the GPM DPR data that is downloaded from JSimpson, the output file looks correct. When I convert the GPM DPR data that is downloaded from GES DISC or ArthurHou it is clearly wrong. Does anyone know why this is happening? I’ve also attempted this conversion using python’s rasterio module and got it to work with JSimpson as well but not ArthurHou or GES DISC… I’ve attached an image of the correct file (Jsimpson) and one of the incorrect data file (ArthurHou) and also here are the links I’m using for the data


JSimpson:https://jsimpsonhttps.pps.eosdis.nasa.gov/radar/DprL2/2A.GPM.DPR.V920240130.20241120-S174100-E181100.V07C.RT-H5

ArthurHou:https://arthurhouhttps.pps.eosdis.nasa.gov/gpmdata/2024/11/01/radar/2A.GPM.DPR.V9-20240130.20241101-S223703-E001016.060637.V07C.HDF5

GES DISC:https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L2/GPM_2ADPR.07/2024/324/2A.GPM.DPR.V9-20240130.20241119-S210218-E223531.060914.V07C.HDF5

We see no problems with
"GES
DISC:https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L2/GPM_2ADPR.07/2024/324/2A.GPM.DPR.V9-20240130.20241119-S210218-E223531.060914.V07C.HDF5"

and variable "//FS/CSF/flagShallowRain" - This variable converts to a raster without issues in Panoply and IDL. The problem is likely with GDAL.

Please be aware that the DPR data on jsimpson is from the GPM near-real time processing stream, whereas the DPR data at GDAAC is from arthurhou.

Ok I did get it to work in Panoply. Thank you for the help. Do you know of a way to programmatically convert this to GeoTIFF format?



GES DISC - ezamkoff wrote:
> We see no problems with
> "GES
>
> DISC:https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L2/GPM_2ADPR.07/2024/324/2A.GPM.DPR.V9-20240130.20241119-S210218-E223531.060914.V07C.HDF5"
>
> and variable "//FS/CSF/flagShallowRain" - This variable converts
> to a raster without issues in Panoply and IDL. The problem is likely with
> GDAL.
>
> Please be aware that the DPR data on jsimpson is from the GPM near-real
> time processing stream, whereas the DPR data at GDAAC is from arthurhou.

We have a FAQ for this,

https://disc.gsfc.nasa.gov/information/faqs?keywords=geotiff&title=How%20do%20I%20convert%20a%20NetCDF%20or%20HDF%20to%20GeoTIFF%3F

This can be done in IDL, MatLab, and Python, if the commercial ArcPRO is not available.

I've tried several different ways and keep getting this same incorrect output. I've followed the FAQ too with no luck. Do you have a direct example that uses this data to programmatically convert the hdf5 to tif?

We were able to reproduce your issues and found a solution to get what you need. Please follow these steps:

1) Go to the data product’s OPeNDAP site where you can subset the data file and download it as a NetCDF4 (.nc4) file:

https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L2/GPM_2ADPR.07/contents.html

Navigate through the dated directories until you reach the file you want to plot, then click on the HDF5 file to open the subsetting page. Your example would be:

2024 --> 324 --> 2A.GPM.DPR.V9-20240130.20241119-S210218-E223531.060914.V07C.HDF5

2) The subsetting page will have all the variables listed. You will want to select: “FS_CSF_flagShallowRain”, “FS_Longitude”, “FS_Latitude”, and “FS_sunLocalTime”.

3) Now you can click the “Get as NetCDF4” button at the top of the subsetting page, and it will prompt you to login with your Earthdata Login credentials, then to a page where your file will automatically be downloaded and saved to your local files (it will ask you where you want to save it).
4) Open QGIS and add the netcdf4 layer to your view by going to Layer  Add Layer  Add Raster Layer. Follow the prompts to point to your file, add flagShallowRain and select your CRS as EPSG:4326 (WGS84).

5) You’ll notice that it still isn’t displaying correctly. You will want to open the GDAL tool from the processing toolbox called “Warp (reproject)” and run it on the layer you just added to your map view.

6) Your inputs to GDAL warp should be:
Input Layer: the layer you just added to the map view
Target CRS: EPSG:4326 – WGS84
Resampling method: Nearest Neighbor
Nodata value: -9999
Output file resolution: 0.05
Output data type: Float32
CRS of the target raster extent: EPSG:4326 – WGS84
Save the reprojected output to a file, where you can select your output type as a TIFF (.tiff) file.

7) Click Run. It should work and be added to your map view automatically. If not, add it from where it was saved. It should now show up correctly georeferenced (attached image 1).

8) The GDAL tool run from the processing toolbox that you run can also be accessed as a Python command. Just navigate to your processing toolbox, open Processing History (clock icon), and right click on the successful run, and click “Copy as Python Command”. The tool itself at the bottom also includes a command line call if needed. These should be able to be run in batch in a python environment (or the Python console in QGIS) if you have multiple files.

Caveats:

-Level 2 data is not inherently regularly gridded. It is “swath” data recorded as satellite point observations, which can be interpolated into a raster. So, there is some “data creation” to fill in values at unknown locations as a natural part of the interpolation method used to create a raster image. GDAL Warp in this example is using the Nearest Neighbor (default) method to do this.

-We noticed errors with previous versions of GDAL installed on QGIS, where some data artifacts were displaying, making the resulting layer incorrect. Please be aware of these artifacts (attached image 2). The versions of QGIS, GDAL and other plugins used where this method worked were:

QGIS version 3.34.13-Prizren
QGIS code revision222b5995
Qt version 5.15.13
Python version 3.12.7
GDAL/OGR version 3.9.3
PROJ version 9.5.0
EPSG Registry database version v11.016 (2024-08-31)
GEOS version 3.13.0-CAPI-1.19.0
SQLite version 3.46.1
PDAL version 2.8.1
PostgreSQL client version 16.2
SpatiaLite version 5.1.0
QWT version 6.3.0
QScintilla2 version 2.14.1
OS version Windows 11 Version 2009

Active Python plugins
Crayfish 3.6.0
ImageServerConnector 2.1.1
quick_map_services 0.19.36
timemanager 3.6
db_manager 0.1.20
grassprovider 2.12.99
MetaSearch 0.3.6
Processing 2.12.99