OPeNDAP access VIIRS L1B data giving inconsistent responses

[tadejong]

I am trying to access VIIRS L1B data using OPeNDAP from the LAADS DAAC, as being able to subset the data would greatly reduce the amount of data we need to download for our workflow. (Currently we download full granules using API-V2)

Running Python3.10, pydap 3.4.0, xarray 2023.6.9, on a linux machine, I can use pydap.client.open_url to open a dataset (example: https://ladsweb.modaps.eosdis.nasa.gov/opendap/RemoteResources/laads/allData/5200/VJ202MOD/2024/016/VJ202MOD.A2024016.0112.002.2024016062617.nc.html).

However, when I then try to access the data in the arrays of the dataset, the server gives a response with an empty body (and otherwise equal headers) up to half of the time, while correctly returning the data the rest of the time.
The empty body results in a UnboundLocalError in pydap if the logger is not enabled, but ultimately leads back to an empty response body.

Although it might be a red herring, it seems this happens more often when accessing a previously unaccessed field or when subsetting the requested data using strides.

These empty response bodies occur often enough that opening the dataset using xarray always fails.

I tried the same code with a dap4 url from this PACE example: https://pydap.github.io/pydap/notebooks/PACE.html and that works.

The fact that the same request sometimes works and sometimes gives an empty response, makes me suspect something weird might be happening serverside?

==== MWE code example ====

import requests
from pydap.client import open_url
import xarray as xr
import os

# Get token.
with open(os.path.join(os.path.expanduser('~'),
'edl_token_auth.hdr')) as f:
lines, = f.readlines()
if not lines[-1] == '\n':
print("Please make sure '~/edl_token_auth.hdr' ends with a newline")
TOKEN = lines[:-1]

my_session = requests.Session()
my_session.headers={"Authorization": TOKEN.split(': ')[1]}


url_DAP4 = 'https://ladsweb.modaps.eosdis.nasa.gov/opendap/RemoteResources/laads/allData/5200/VJ202MOD/2024/016/VJ202MOD.A2024016.0112.002.2024016062617.nc'



ds_full = open_url(url_DAP4, session=my_session, protocol='dap4')

print(ds_full)

ds_full['/observation_data/M11'][:,:] # This fails sometimes due to empty response body.

ds_full['/observation_data/M03'][::8,::8] # This relatively often due to empty response body.


# The following and other ways to open a opendap dataset using xarray
# fail consistently
store = xr.backends.PydapDataStore.open(url_DAP4,
session=my_session)
xr.open_dataset(store)

[LAADSx_UserServices_M]

We are still investigating the issue and will get back to you as soon as we have a resolution.

[tadejong]

Thanks for letting me know and thanks for looking into this! Please let me know if there is anything I can do from my side.

[LAADSx_UserServices_M]

Our Systems team has installed an updated version of hyrax that appears to solve the issue. We cannot recreate the empty body/crash with the urls you sent. So, try again and let us know if you still see issue.

[tadejong]

Thank you! It seems this has indeed resolved the issue of empty responses.

Opening the dataset using xarray turned out to be a different issue related to the groups in the netcdf file and my used version of xarray. For the record: I am now able to open the dataset using updated versions of `pydap` and `xarray.open_datatree`.

[LAADSx_UserServices_M]

Great! thanks very much for the confirmation.