Earthdata Forum

I'm trying to extract spectral radiance, sensor zenith/azimuth, and solar zenith/azimuth for unsaturated cloud pixels from HICO data I downloaded from OBDAAC Level 1 & 2 browser (https://oceancolor.gsfc.nasa.gov/cgi/browse.pl?sen=amod). I am having a few issues.

First, I can't seem to find the actual spectral radiance ('Lt') data from the L1B files. My issue is similar to the one reported here:
https://forum.step.esa.int/t/lt-data-in-nc-format-downloaded-from-nasa-could-not-be-displayed-entirely/30593
only I can't view the data in either SNAP or SeaDAS. I'm using these viewers to browse, but I'm also using the netCDF4 python library to navigate and pull data from these files, and they also aren't showing anything about TOA radiance data aside from the metadata (e.g. wavelengths, band fwhm, units, scaling factors). Given how large these files are, it seems the data must be there. I've tried two different L1B files from two different months.

Second, I'm having problems pulling azimuthal angle data, the same as the user in this post:
viewtopic.php?f=7&t=1609
I think the answer provided here - that the angle data is outside the bounds of the valid bounds (-180 to 180, whereas actual azimuth is stored 0 to 360) is correct. The user said they resolved it on a different computer but I'm sort of wondering if they just pulled a file that, by luck, didn't have azimuthal angles above 180. This isn't a huge deal to me because I, too, can find such data, but it is sort of annoying.

Finally, the Ocean Color "Sensor and Data Characteristics" page (https://oceancolor.gsfc.nasa.gov/hico/instrument/dataset-characteristics/) has a section on flags that suggests L1B files have flags for things like clouds, which is exactly what I need to create a mask for, but I can't seem to locate this either. The page in question is for hdf5 files, not netCFD4, so maybe that's the difference.

I can find spectral radiance data in L2 files but these seem to lack sensor/solar angle information (haven't looked in python yet, only SeaDAS).

Any insight is appreciated, thanks!

For your first question, there is a bug in SeaDAS 8.1 that it does not show Lt band in HICO L1B files. It's fixed, and the fix will be in the next release 8.2.

Thank you! Good to know.

As far as the netCDF4 python library goes, I think I was just unfamiliar with using it. I was able to find what I think are radiance values; the attached figure shows my terminal output for pixel (136,361). 128 values corresponding to radiances for each of the 128 measured bands, I think. Pixel indexing doesn't seem to be consistent between L1B and L2 files though - I verified in SeaDAS using the sensor_zenith data that the (0,0) pixel is the upper left corner, then loaded the L2 file and queried pixel radiance values using python and it doesn't seem to match (might be flipped or rotated 180?).

Probably more unfortunately for me, it looks like all pixels flagged as clouds appear to be NaN? At least, everything I've looked at between two files and several bands. An instrument designed to look at seawater would have exposure/gain set such that it saturates looking at most clouds, I guess. Bummer. Still, the quasi-color images in the browser preview seem to show pretty good definition in a lot of clouds which would suggest it is collecting info in at least some bands.

In case anyone was interested, I was ultimately successful in getting the data I wanted. Maybe I was just unlucky browsing in SeaDAS when it came to finding unsaturated cloud pixels, but a more systematic approach with my own Python script revealed plenty of usable data. I have a pretty rudimentary filter for clouds and bad pixels, here's the Python code I used for the images:

import numpy as np
from netCDF4 import Dataset
from PIL import Image

# import netCDF4 file and extract radiance data
nc = Dataset('H2010032203758.L1B_ISS.nc','r')
lt = np.array(nc.groups['products']['Lt'])

# remove a strange 16-bit pixel that's in there for some reason
lt[lt == 65535] = 0

# crop spectral data between 450nm and 800nm (short and long bands seemed messier)
cropped_lt = lt[:,:,17:80]

# find maximum value in spectral data and call that 'saturation'
saturation = max(np.ndarray.flatten(cropped_lt))

im = np.zeros((2000,512))

for i in np.arange(0,2000):
for j in np.arange(0,512):
if max(cropped_lt[i][j]) < 0.8 * saturation and sum(cropped_lt[i][j])>5000 and 0 not in cropped_lt[i][j]:
im[i][j] = sum(cropped_lt[i][j])


im = im / max(np.ndarray.flatten(im)) * 65535
im = im.astype(np.uint16)
im = Image.fromarray(im)
im.save('H2010032203758.L1B_ISS.png')

While you seem to have resolved the issue raised in this thread, there were a couple of questions asked that I'll answer here:

Pixel indexing doesn't seem to be consistent between L1B and L2 files though - I verified in SeaDAS using the sensor_zenith data that the (0,0) pixel is the upper left corner, then loaded the L2 file and queried pixel radiance values using python and it doesn't seem to match (might be flipped or rotated 180?)

The SeaDAS product reader attempts to orient the scene with north at the top. The L2 metadata includes information that helps with this ( the global attributes startDirection and endDirection). The L1B metadata does not contain the information in the same easily usable form, so it is possible - in fact likely - that the orientation will differ. The underlying L2 data do match the L1B, but yes, the visualization flips the data.

Finally, the Ocean Color "Sensor and Data Characteristics" page (https://oceancolor.gsfc.nasa.gov/hico/instrument/dataset-characteristics/) has a section on flags that suggests L1B files have flags for things like clouds, which is exactly what I need to create a mask for, but I can't seem to locate this either. The page in question is for hdf5 files, not netCFD4, so maybe that's the difference.

Yes, the information on that page is for an older format of the HICO L1B data. Thanks for alerting us to this discrepancy, we will update the information on that page.

Regards
Sean