How to open MISR L3 aerosol product in Python?

[staochun]

Hi NASA staffs.

I have already downloaded MISR L3 product: MI3DAENF.002 (https://asdc.larc.nasa.gov/data/MISR/MI3DAENF.002/). I don't know how to open that as I failed when I used xarray/netCDF4 in python (because it's in netcdf format, as shown in the figure below where the dataset seems empty when trying to use xarray). Also there's little on this aspect in Q&A or forum posts of NASA webs.

I found a python script in NASA github but it seems in hdf5 format (old version?). So it doesn't help.

Sincerely appreciate if I could be given a reply on how to address this issue! Also welcome to reach out via email: stc20@mails.tsinghua.edu.cn for insightful discussions.

[ASDC - cheyenne.e.land]

Hello,

Thank you for your interest in MISR data.

The reason why the dataset seems empty when trying to use xarray is because xarray can only be used if variables are at the root level. This dataset (netCDF) organizes its variables, dimensions, and attributes in hierarchical groups, however, you can open it using xarray if you know the specific groups/path.

If not, to get the list of groups you this code:

Code: Select all

import netCDF4 as nc

fn = 'MISR_AM1_CGAS_FIRSTLOOK_OCT_12_2022_F15_0032.nc'
dataset = nc.Dataset(fn)
group_dataset = dataset.groups.keys()
print(group_dataset)

Output:

Code: Select all

dict_keys(['Source_file', 'Aerosol_Parameter_Average', 'HDFEOS INFORMATION', 'Time of Observations Aerosol_Parameter_Average'])

Once you have the list of groups you can use xarray like this:

Code: Select all

import xarray as xr

Aerosol_Parameter_Average_xarray = xr.open_dataset(fn, group='Aerosol_Parameter_Average')

Output:

Code: Select all

<xarray.Dataset>
Dimensions:                                                (Longitude: 720, Latitude: 360, Band: 4, Optical_Depth_Range: 9, Coefficient: 3, Algorithm_Type: 3, Retrieval_Success_Type: 2)
Coordinates:
  * Longitude                                              (Longitude) float64 ...
  * Latitude                                               (Latitude) float64 ...
  * Band                                                   (Band) object 'blu...
  * Optical_Depth_Range                                    (Optical_Depth_Range) object ...
  * Coefficient                                            (Coefficient) object ...
  * Algorithm_Type                                         (Algorithm_Type) object ...
  * Retrieval_Success_Type                                 (Retrieval_Success_Type) object ...
Data variables: (12/28)
    crs                                                    int32 ...
    Absorbing_Optical_Depth                                (Latitude, Longitude, Optical_Depth_Range) float32 ...
    Absorbing_Optical_Depth_Count                          (Latitude, Longitude, Optical_Depth_Range) float64 ...
    Absorbing_Optical_Depth_Standard_Deviation             (Latitude, Longitude, Optical_Depth_Range) float32 ...
    Aerosol_Optical_Depth                                  (Latitude, Longitude, Optical_Depth_Range) float32 ...
    Aerosol_Optical_Depth_Count                            (Latitude, Longitude, Optical_Depth_Range) float64 ...
                                                    ...
    Aerosol_Optical_Depth_Per_Band                         (Latitude, Longitude, Optical_Depth_Range, Band) float32 ...
    Aerosol_Optical_Depth_Per_Band_Count                   (Latitude, Longitude, Optical_Depth_Range, Band) float64 ...
    Absorbing_Aerosol_Optical_Depth_Per_Band               (Latitude, Longitude, Optical_Depth_Range, Band) float32 ...
    Absorbing_Aerosol_Optical_Depth_Per_Band_Count         (Latitude, Longitude, Optical_Depth_Range, Band) float64 ...
    Algorithm_Type_Count                                   (Latitude, Longitude, Algorithm_Type, Retrieval_Success_Type) float64 ...
    Average_Fill_Flag                                      (Latitude, Longitude) float32 ...
Attributes:
    resolution_in_degrees:  0.5

I also want to add that another user friendly way to view data is Panoply. Panolpy plots geo-referenced and other arrays from netCDF, HDF and other datasets.

Here is the link for more information and where to Download:
https://www.giss.nasa.gov/tools/panoply/
https://www.giss.nasa.gov/tools/panoply/download/

Hope this helps.

Regards,
ASDC User Services

[staochun]

Hi ASDC staffs,

Big thanks for telling me how to use xarray to open this product and introduce another useful library. Indeed the group concept is very useful in tackling this kind of issue as you mentioned in the post.

Xarray is a widely used library that appears normally in articles published by Nature portfolios and alike. Therefore I think your reply post will be a boost to spread and usage of this product. Your solution to this issue makes my post worthwhile!

Maybe a systematic tutorial that illustrates how to use those library (especially newly released in the Python community) can help researchers and industrial users unleash the potential of those products to contribute more to the earth data science community!!

Best,
Taochun

[vasu_007]

I am having 22 years of MISR level 2 data version 23. These are swath data. I am trying to combine these multiple files and to make a mean value dataset for the world.I also want to regridd the data for my purpose. I tried using Xarray - open_mfdataset and open_dataset with for loop but I failed. I want to combine this file in such a way that variables over India can be extracted and interpreted after plotting. Please help me if you can.
Thanks
Vasu

[ASDC - cheyenne.e.land]

Hello @staochun

We agree, a tutorial would be very helpful and would be beneficial to those within science community. We are looking to create one in the near future. Thanks again for your interest in using MISR data.

Regards,
ASDC User Services

[ASDC - cheyenne.e.land]

Hello @vasu_007,

Thank you for your interest in MISR data.

We will look in to this and get back to you soon. Also, could you give the name of the Level 2 MISR collection that you are using for this? Thank you.

Regards,
ASDC User Services

[vasu_007]

MIL2ASAE_3
I have marked in the image the data I am using

[staochun]

Hello @ASDC - cheyenne.e.land

Thanks for agreeing with my idea! Question raised by @vasu_007 is also interesting!

Hope I will have opportunies with you all to help create this kind of useful tutorial which may coutribute to the xarray (or other projects) in terms of compatibility between satellite products and code design in Github (e.g. post an issue)!

It will be a big step forward. I suppose there's emerging scientific ecosystem of libraries on how to analyze those products (e.g., nctools, ESMPy, etc) and some comprehensive tutorials for those integrations are urgently needed, especially if we consider CUDA-based acceleration-integrated extensions (e.g., CuPy based).

Best,
Taochun

[vasu_007]

MIL2ASAE_3
I have marked in the image the data I am using

[vasu_007]

Please, can you give me solution I have mentioned in above posts