Earthdata Forum

I want to obtain MODIS rhos products, but the processing speed using the l2gen module is very slow. Here is my Python code：
import subprocess
import os

# Assume that L1B files and GEO files correspond to each other and follow the same naming convention,
# but are located in different directories.
l1b_folder = "/data/L1B/"
geo_folder = "/data/GEO/"

# Get lists of L1B and GEO files, ensuring their filenames match.
l1b_files = [f for f in os.listdir(l1b_folder) if f.endswith('.L1B_LAC')] # Retrieve L1B files.
geo_files = [f for f in os.listdir(geo_folder) if f.endswith('.GEO.hdf')] # Retrieve GEO files.

# Ensure the number of files is consistent and matches one-to-one.
if len(l1b_files) != len(geo_files):
print("The number of files does not match. Please check the L1B and GEO files.")
else:
for l1b_file, geo_file in zip(l1b_files, geo_files):
# Construct the full file paths.
l1b_path = os.path.join(l1b_folder, l1b_file)
geo_path = os.path.join(geo_folder, geo_file)

# Specify the output file path (can be dynamically generated based on the filename).
output_file = f"/data/L2gen/{l1b_file.split('.')[0]}.L2.OC.nc"

# Set the command and parameters.
command = [
"l2gen",
f"ifile={l1b_path}", # Input L1B file.
f"geofile={geo_path}", # Input GEO file.
f"ofile={output_file}", # Output file path.
"aer_opt=-99", # Skip aerosol correction (Rayleigh only).
# Required rhos bands (can be adjusted as needed). Adding l2_flags for quality flag checking.
"l2prod=rhos_469,rhos_555,rhos_645,rhos_859,rhos_1240,l2_flags",
"oformat=netCDF4" # 645 red, 555 green, 469 blue, 859 NIR, 1240 SWIR.
]

# Execute the command.
subprocess.run(command)
--------------------------------------------------------------
This is my command line output:
Opening: /data/L2gen/A2003001044000.L2.OC.nc


The following products will be included in /data/L2gen/A2003001044000.L2.OC.nc.
0 rhos_469
1 rhos_555
2 rhos_645
3 rhos_859
4 rhos_1240
5 l2_flags



Begin l2gen Version 9.8.0-V2024.8 Processing
Sensor is MODISA
Sensor ID is 7
Sensor has 16 reflective bands
Sensor has 8 emissive bands
Number of along-track detectors per band is 10
Number of input pixels per scan is 1124
Processing pixels 1 to 1124 by 1
Processing scans 1 to 840 by 1
Ocean processing enabled
Land processing enabled
Atmospheric correction enabled

Begin MSl12 processing at 2024359154226000

Allocated 1918732 bytes in L1 record.
Allocated 1918732 bytes in L1 record.
Allocated 1918732 bytes in L1 record.
Loading radiance to brightness temperature from /root/ocssw/share/modis/aqua/cal/bt_modisa.hdf
Loading land mask information from /root/ocssw/share/common/gebco_ocssw_v2020.nc
Loading DEM information from /root/ocssw/share/common/gebco_ocssw_v2020.nc
Loading ice mask file from /root/ocssw/share/common/ice_climatology.hdf
Loaded monthly NSIDC ice climatology HDF file.
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_412.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_443.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_469.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_488.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_531.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_547.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_555.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_645.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_667.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_678.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_748.hdf
Loading XCAL rvs and polarization sensitivities from /root/ocssw/var/modisa/xcal/OPER/xcal_modisa_axc_oc_v4.0d_869.hdf
Loading DEM info from /root/ocssw/share/common/gebco_ocssw_v2020.nc
Loading climatology file /root/ocssw/share/common/sst_climatology.hdf
Loading SSS reference from Climatology file: /root/ocssw/share/common/sss_climatology_woa2009.hdf


Opening meteorological files.
met1 = /root/ocssw/share/common/met_climatology_v2014.hdf
met2 =
met3 =
ozone1 = /root/ocssw/share/common/ozone_climatology_v2014.hdf
ozone2 =
ozone3 =
no2 = /root/ocssw/share/common/no2_climatology_v2013.hdf


Opening ozone file /root/ocssw/share/common/ozone_climatology_v2014.hdf


Opening NO2 file /root/ocssw/share/common/no2_climatology_v2013.hdf


Opening NO2 frac file /root/ocssw/share/common/trop_f_no2_200m.hdf


Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_412_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_443_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_469_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_488_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_531_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_547_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_555_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_645_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_667_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_678_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_748_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_859_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_869_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_1240_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_1640_iqu.hdf
Loading Rayleigh LUT /root/ocssw/share/modis/aqua/rayleigh/rayleigh_modisa_2130_iqu.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_412.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_443.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_469.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_488.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_531.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_547.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_555.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_645.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_667.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_678.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_748.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_859.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_869.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_1240.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_1640.hdf
Loading polarization file /root/ocssw/share/modis/aqua/cal/polcor_modisa_2010b_2130.hdf

Using 869.0 nm channel for cloud flagging over water.
Using 412.0 nm channel for cloud flagging over land.

Processing scan # 0 (1 of 840) after 4 seconds
Aerosol selection bands 748 and 869

Applying ocean BRDF including:
Reflection/refraction for upwelling radiance.
Reflection/refraction for downwelling radiance.
Morel f/Q
chl_hu: using 443.00 547.00 667.00
rh_ndims=5 rh_dimids=4 3 2 1 0
morel f/q file dimensions n_a=13 n_n=17 n_c=6 n_s=6 n_w=7

Reading foq file /root/ocssw/share/common/morel_fq.nc ndims=5 nvars=6 sds_id=5 var=foq


Closing foq file /root/ocssw/share/common/morel_fq.nc


Morel f/Q table from file /root/ocssw/share/common/morel_fq.nc

Compute Raman scattering correction #2.

Loading Raman coefficients from: /root/ocssw/share/modis/aqua/raman.nc.
Processing scan # 50 (51 of 840) after 56 seconds
Processing scan # 100 (101 of 840) after 108 seconds
Processing scan # 150 (151 of 840) after 168 seconds
Processing scan # 200 (201 of 840) after 223 seconds
Processing scan # 250 (251 of 840) after 276 seconds
Processing scan # 300 (301 of 840) after 328 seconds
Processing scan # 350 (351 of 840) after 379 seconds
Processing scan # 400 (401 of 840) after 432 seconds
Processing scan # 450 (451 of 840) after 483 seconds
Processing scan # 500 (501 of 840) after 534 seconds
Processing scan # 550 (551 of 840) after 584 seconds
Processing scan # 600 (601 of 840) after 634 seconds
Processing scan # 650 (651 of 840) after 687 seconds
Processing scan # 700 (701 of 840) after 736 seconds
Processing scan # 750 (751 of 840) after 789 seconds
Processing scan # 800 (801 of 840) after 842 seconds

Percentage of pixels flagged:
Flag # 1: ATMFAIL 0 0.0000
Flag # 2: LAND 91878 9.7312
Flag # 3: PRODWARN 0 0.0000
Flag # 4: HIGLINT 0 0.0000
Flag # 5: HILT 732487 77.5808
Flag # 6: HISATZEN 37943 4.0187
Flag # 7: COASTZ 21104 2.2352
Flag # 8: SPARE 0 0.0000
Flag # 9: STRAYLIGHT 161533 17.1086
Flag #10: CLDICE 848678 89.8871
Flag #11: COCCOLITH 63 0.0067
Flag #12: TURBIDW 75324 7.9779
Flag #13: HISOLZEN 0 0.0000
Flag #14: SPARE 0 0.0000
Flag #15: LOWLW 0 0.0000
Flag #16: CHLFAIL 0 0.0000
Flag #17: NAVWARN 0 0.0000
Flag #18: ABSAER 0 0.0000
Flag #19: SPARE 0 0.0000
Flag #20: MAXAERITER 0 0.0000
Flag #21: MODGLINT 10229 1.0834
Flag #22: CHLWARN 0 0.0000
Flag #23: ATMWARN 0 0.0000
Flag #24: SPARE 9 0.0010
Flag #25: SEAICE 0 0.0000
Flag #26: NAVFAIL 0 0.0000
Flag #27: FILTER 0 0.0000
Flag #28: SPARE 0 0.0000
Flag #29: BOWTIEDEL 0 0.0000
Flag #30: HIPOL 221 0.0234
Flag #31: PRODFAIL 0 0.0000
Flag #32: SPARE 0 0.0000

End MSl12 processing at 2024359155706000
Processing Rate = 0.954545 scans/sec

There are numerous reasons why processing may be slower than expected, most of which are related to I/O.
One thing you can do is to run with atmocor=0, this will disable a lot of unnecessary processing that setting aer_opt=-99 doesn't do. Although it does disable aerosol selection and removal, it doesn't stop the rest of the atmospheric correction processing. Setting atmocor=0 will. Rayleigh is still computed, so rhos can be calculated.

Regards,
Sean