Creation of level 3 Kd_490 files

[jcfischer]

Dear all,
I would be very happy about some help regarding the creation of level 3 files of the diffuse attenuation coefficient at 490 nm. Those respective files can be directly downloaded in 4 and 9 km resolution but for my turbidity study I need higher resolution data. Therefore, I tried to create those files by myself using the SeaDAS processing.
Here is what I did:
- download of L1A.LAC files
- processing with first modis_GEO.py processor and second modis_L1B.py processor
- processing of created L1B_LAC file with l2gen processor with Kd (Kd_490) as the only selected product (under Products in the GUI) and the resolution set to 250 m (under Miscellaneous). This process takes one hour to complete and a L2_LAC_OC file is created. By trying to load this file into SeaDAS I get this error message: execution exception: java.lang.OutOfMemoryError: Java heap space. It should be noted that I run SeaDAS for this processing in a virtual machine with Linux on my Windows system.

Can anybody tell me how to create the files I need? Might the mulitlevel processor be useful? Can I also process the L2A_LAC_OC.nc files which are also available for direct donwloading?
Thank you very much! Your help is highly appreciated! Greetings, JC

[OB SeaDAS - dshea]

a 250m file is going to be pretty big.  Processing it with l2gen on the command line should not be a problem, but loading the file into SeaDAS might fill up your memory.  How much RAM did you allocate to your virtual machine?

don

[gnwiii]

Are you using extracted data to minimize the number of pixels in the calculation?   For a coastal region you may need to extract multiple small boxes (using the order system or l1aextract_modis using lonlat2pixline to provide pixel coordinates).  If you have a lot of files this could become tedious using the GUI, but can be done efficiently using shell scripts. 

Running SeaDAS in a Windows VM is far from ideal for a large project due mainly to disk I/O overhead.  There are ways to optimize disk I/O for a VM, but the effort needed to get it right is hard to justify for a short-term project.  For this sort of work, a relatively low-end processor with a disk array can outperform more expensive systems with high-end processors and a single large disk.

If the Windows system has lots of RAM you can allocate more for the VM and tweak the limits for the SeaDAS gui in the seadas.vmoptions file.
The multilevel processor can simplify running a complex workflow, but won't speed up processing or reduce memory requirements.

The L2_LAC files don't give 250m resolution, but maybe they are adequate for your purposes.  The current binning software is likely to give Moire patterns when you ask for higher resolutions, but we have been promised new binning software in the next release that mitigates the problem.  You can also use the GPT mosiac operator to map level-2 data.

[jcfischer]

Thank you both very much for answering!
The RAM available in the virtual machine was set to 4 GB. The use of the order data function helped. The processing of (spatially) smaller files takes way less time and I can load the created level 2 files in the SeaDAS GUI. This also works with only 2 GB allocated to the virtual machine.
My area of interest is a 70 km coastline segment of the western side of La Réunion, France. Therefore, I would like to use 250 m resolution data instead of 1 km.

In order to produce the wanted Kd_490 level 3 files (I need those for front detection analysis with ArcMap) I use "l2bin" with the L2_LAC_OC.x.hdf file as input file. The oformat is set to "netCDF4". A L3b_DAY_OC.x.hdf file is created. Why is it not a netCDF4 or nc file? Loading this L3b_DAY_OC.x.hdf file in SeaDAS is not possible: no appropriate reader found. How can I solve this problem?
Thanks in advance!

[gnwiii]

The L3b files have data structures designed to facilitate further binning, but need processing to generate mapped grids (e.g., using l3mapgen).  At your scale, fronts can be tricky since they move with tides.   You may get nice results at 250m for a single pass on a clear day but if you have to bin several days to get a cloud-free composite, frontal features will be broken up across cloud edges and smeared in areas that were free of clouds in multiple passes.   You may have to select L2 images carefully to group mostly clear-sky passes with similar tidal phases, using mosaic to map each L2 pass then selecting files to build up a nice picture of the fronts.  You can use GPT mosaic or l3bin+l3mapgen to make composites.   Comparing the two approaches may be helpful -- mosaic tends to overweight lower quality pass edge pixels so you tend to get images with fewer missing data areas but some dubious values that produce high gradient values and may confuse edge detection tools.

[jcfischer]

Thank you very much for replying and all the useful input!
I thought a lot about the fact of (turbidity) fronts moving with the tides, grouped images representing the same tide status and tried to create representative composites. In fact, I realized that the horizontal movement of the (surface) water masses around La Réunion does not reach a magnitude of 1 km at all. And the MODIS data I am using has a resolution of 1 km. Therefore, I came to realize that this effect can be neglected. What do you think?
Thanks in advance!

[gnwiii]

There are sub-pixel edge detection methods (Breder et al Sub-Pixel Accuracy Edge Fitting by Means of  B-Spline (2009)).   Another option is a sensor with much higher resolution, using a drone, aircraft, or geostationary satellite as in Hu et al , Diurnal Variability of Turbidity Fronts Observed by Geostationary Satellite Ocean Color Remote Sensing (2016).  Turbidity fronts tend to have covariates such as SST and may allow you to use multiple sensors, as in Miller, Composite front maps for improved visibility of dynamic sea-surface features on cloudy SeaWiFS and AVHRR data (2009).  Some fronts are associated with lots of biological activity (marine mammals, seabirds, fishers) which might allow collection of GPS data.