Running l2gen on OLCI through cmd line

[slsmit34]

Hello,
I am having difficulty with running l2gen on an S3A OLCI scene due to errors with locating the parameter files.

I've set the OCDATAROOT and OCVARROOT with the following:

Code: Select all

os.environ['OCDATAROOT'] = '/.../l2gen/ocssw'
os.environ['OCVARROOT'] = '/.../l2gen/ocssw/var'

And I am using subprocess to run the following command:

Code: Select all

cmd = f'{l2gen_path} ifile={input_radiance_file_path} ofile={output_file_path} par={param_file_path}'
result = subprocess.run(cmd, shell=True, encoding='ascii', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("l2gen output:\n", result.stdout)

The default installation of ocssw put the common folder inside share, but regardless of what I put for the param_file_path, I get this error:

Code: Select all

-E- clo_readFile: Can't open parameter file - /.../l2gen/ocssw/common/msl12_defaults.par

I have confirmed that msl12_defaults.par exists in the common folder. I also currently have the olci folder, which has its own msl12_defaults.par, into the /ocssw folder.

ocssw_folder.PNG (17.33 KiB) Not viewed yet

share_folder.PNG (6.92 KiB) Not viewed yet

Interestingly, I can sometimes get the code to work when I shuffle the folders around mid-run (i.e. If I put both olci and common folders in the ocssw folder, change the param_file_path accordingly, then run the code, it gets frozen. If I move the common folder back into share folder while the run is frozen, it will un-freeze and work. But this is the only way I have gotten it to work - the folder must be manually moved mid-run.)

Best,
Sami

[OB ODPS - towens]

Try adding share to the OCDATAROOT

os.environ['OCDATAROOT'] = '/.../l2gen/ocssw/share'

and there should be an olci directory under share

Tommy

[slsmit34]

Hi Tommy,
Thanks for reaching out! I appreciate your help. I've moved /common and /olci back under /share, and changed OCDATAROOT accordingly:

Code: Select all

os.environ['OCDATAROOT'] = '/panfs/ccds02/nobackup/people/slsmit34/Scripts/2023_2024/Jun2024_modernScripts/l2gen/ocssw/share'

share_folder_updated.PNG (7.73 KiB) Not viewed yet

Now when I run the code, It gets freezes at:
Running command: /.../l2gen/ocssw/bin/l2gen ifile=/.../Satellite/Sentinel-3_OLCI/L1B/2021/S3A_OL_1_EFR____20210401T145256_20210401T145556_20210402T185622_0179_070_139_2340_LN1_O_NT_002.SEN3/Oa14_radiance.nc ofile=/.../Satellite/Sentinel-3_OLCI/l2gen/2021/S3A_OLCI_EFRNT.20210401T145313.L2gen_Oa14.nc par=/.../l2gen/ocssw/share/common/msl12_defaults.par.

As per some googling, I've tried running it with subprocess.Popen instead:

Code: Select all

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

which then runs until the "Compute Raman scattering correction #2.", but this step has been taking very long to do, ~15 seconds for each 50 scans/4091 required:

raman_process.PNG (21.32 KiB) Not viewed yet

Best,
Sami

[OB ODPS - towens]

The Raman Correction was computed already, that output is l2gen processing the file. These are chunky L1B files that take a while to process.

Tommy

[slsmit34]

Hi Tommy,
Ok, thank you! I have a bit of a confusing question regarding processing the L1B files. Since I am processing OLCI S3A/S3B, the L1B file is split into 21 separate .nc files for radiance (Oa01_radiance, Oa02_radiance,etc).

L1b_files_list.PNG (7.05 KiB) Not viewed yet

Is it recommended to run l2gen 21 times, once on each radiance file? Or Is there a way to run it once on the entire image, like how the MODIS L1 files are processed? (I.e. combining all radiance .nc files into a single .nc file and processing it this way.)

Thank you so much for all of your support, I really appreciate it!

Best,
Sami

[OB ODPS - towens]

l2gen only needs the first file:
ifile=Oa01_radiance.nc
geofile=geo_coordinates.nc

It processes all from there.

Tommy