PACE empty L2_AOS datasets

[astrotheus]

Hello PACE community,

I am master student from Gent University and I am working on validating backscattering based PIC algorithms with PACE L2_AOS data. I have following issue: After downloading the data for my research area with earthaccess (through python), all files seem to be empty, or more precisely: they show 0 dimensions and 0 variables. I still can open the files and see metadata, but no variables and no dimensions. this happens with all the downloaded files, and I dont know what I did wrong. I posted my code (the download code and the opening code) below, Thanks for your help in advance. You can also find a example output from XR.open_dataset(path).

Looking forward to your answers,

Tim


###downloadscript###
import earthaccess
from pathlib import Path

auth = earthaccess.login(persist = True)

results = earthaccess.search_datasets(instrument = "oci")

for item in results:
summary = item.summary()
print(summary["short-name"])

min_lon = -32
min_lat = 45
max_lon = 8
max_lat = 70

tspan = ("2024-04-01", "2024-08-01")
bbox = (min_lon, min_lat, max_lon, max_lat)


results = earthaccess.search_data(
short_name="PACE_OCI_L2_AOP",
temporal=tspan,
bounding_box=bbox)


directory = Path('/Volumes/Extreme SSD/Master Thesis /Input/L2AOP')
directory.mkdir(parents=True, exist_ok=True)

paths = earthaccess.download(results, directory)

###openingscript###
from data import openfilecsv, openfilenetCDF
import xarray as xr
import os



PACEdirectory = '/Volumes/Extreme SSD/Master Thesis /Input/L2AOP'
PACEonefile = '/Volumes/Extreme SSD/Master Thesis /Input/L2AOP/PACE_OCI.20240426T074015.L2.OC_AOP.V3_0.nc'
SensorGPS_Signal = '/Users/timfugl/Desktop/Thesis/input/PIC sensor with matched GPS.csv'
SensorSampleMatchup = '/Users/timfugl/Desktop/Thesis/input/PIC sensor-sample matchup.csv'

PACE_testfile = openfilenetCDF(PACEonefile)
SensorPICGPS = openfilecsv(SensorGPS_Signal)
SensorSampleMatchup= openfilecsv(SensorSampleMatchup)


lst = []
lst2 = []
lst3 = []

#creates fullpathnames and then adds them to list 'lst'
for filename in os.listdir(PACEdirectory):
lst.append(os.path.join(PACEdirectory, filename))

#MAC creates weird _. metafiles for every file stored in a external drive. These cannot be opened. this short loop (for strings inlst) trys and sorts in lst2 and lst3
for path in lst:
try:
xr.open_dataset(path)
print(f" OK: {path}")
lst2.append(path)
except Exception as e:
print(f" ERROR in file: {path}")
print(e)
lst3.append(path)


dataset = xr.open_dataset(lst2[0])
print(dataset)

empty_files = []
not_empty_files = []

for file in lst2:
file_size = os.path.getsize(file)
if file_size == 0:
print(f'{file} is empty.')
empty_files.append(file)
else:
print(f'{file} is not empty.')
not_empty_files.append(file)

print(f'there are {len(empty_files)} empty files and {len(not_empty_files)} not empty files')

###output for opening a file (same for all files)###

there are 0 empty files and 1984 not empty files

<xarray.Dataset> Size: 0B
Dimensions: ()
Data variables:
*empty*
Attributes: (12/45)
title: OCI Level-2 Data AOP
product_name: PACE_OCI.20240731T090641.L2.OC_AOP.V3_...
processing_version: 3.0
history: l2gen par=/data15/sdpsoper/vdc/vpu14/w...
instrument: OCI
platform: PACE
... ...
geospatial_lon_max: 78.64095
geospatial_lon_min: -4.3571634
startDirection: Ascending
endDirection: Ascending
day_night_flag: Day
earth_sun_distance_correction: 0.9705536961555481

[OB ScienceSW - jlindo1]

Hi astrotheus,

Are you able to open the files with ncdump -h? As in, `ncdump -h PACE_OCI.20240426T074015.L2.OC_AOP.V3_0.nc`. If that's possible, that may help. What I am hoping to see there is for the metadata you're looking for to exist there, and I think that would point to a bug in your Python. If it's not there in the ncdump, then the files probably weren't downloaded properly.

Could you please also include the output of putting `file <filename>` on the command line, where <filename> is any one of the files you downloaded? This will also help me find out whether the file was downloaded properly; it should come back with something referring either to NetCDF or HDF.

[astrotheus]

Dear Jilindo,

Thanks a lot for your answer. I actually had a different mistake: I only worked with L3 data before, where the data is stored in netCDF - files without subgroups. After having a conversation with a colleague we discovered that the RRS data is stored in the subgroup 'geophysical_data' in the L2 files, and I managed to extract what I need in CSV files. So the problem is solved!

I really appreciate your effort for putting the answer!

Thanks again,

astrotheus :)