Python script to download VIIRS data doesn't work after Jan 2023

[tahersima]

Hello,

I have a python code (the second code from https://urs.earthdata.nasa.gov/documentation/for_users/data_access/python) to configure a connection and download data from Earthdata. In 2022 I could download thousands of URLs using that scripts. But starting January 2023 the code hasn't been working for me and I cannot figure out what is wrong.

Once I execute the code, it runs and downloads empty files - they are just 20KB and do not contain imagery data. If I copy the URL in browser, it donwloads the imagery and the files are ~150MB.
An example URL is: "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc"

Something is wrong and I can't figure out what it is. Any help would be much appreciated.

[OB ODPS - towens]

Run the file command on the short files.
They are likely HTML files. If so, open one in a browser and there will likely be a message about an authentication error when using the script.

Tommy

[tahersima]

You are absolutely right. Although they have .nc at the end of file name, they are just html files telling me to "Please copy and save the token below for use with any supported LAADS download utilities or scripts."

I know how to generate tokens from NRT, but don't know how to use them in a Python script.
The python examples I've found on earthdata use username/password. Like the ones here:
https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python

So far, we know this is an authentication issue that is happening to python codes after January 2023.

[LAADSx_UserServices_M]

Python script example for LAADS DAAC data download can be found at:
https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/#python

[tahersima]

Thank you for referring me to the python script example for LAADS DAAC data.

I configured the arguments with the URL and a token. But it throws the following error: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte.
This happens at this line 'return fh.read().decode('utf-8')'. I have tried several approaches that I found on the Internet, but nothing seems to work for this particular case.

Any ideas how to decode this correctly?
I am running the code in Python 3 and Windows.

[LAADSx_UserServices_M]

See if the suggestions made for another user at viewtopic.php?t=3979 resolve your issue.

[tahersima]

Unfortunately, it didn't resolve the issue. The error remains exactly the same:

in geturl
return fh.read().decode('utf-8')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

[LAADSx_UserServices_M]

@tahersima, It would help to know what URL you are trying to download. We have not seen that error in all our testing, but you might be downloading different types of files than we have tested. Also it would help to know if you are using Python2 or Python3.

[tahersima]

I also thought it could be because of file type but couldn't figure out what.
Example of the URLs that I am trying to download is:
https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc
The file contains NetCDF4 scenes captured by S-NPP VIIRS.

I am using Python 3 in Windows. I am also using Anaconda for environment management and run my codes in Spyder.

[LAADSx_UserServices_M]

We think it is because you are trying to use the script to download a single data file. The script is meant for downloading directory trees by getting a directory listing in either json or csv format first. Since you are asking for a netCDF formatted data file, the script doesn’t know what to do with it or how to parse it for directory information. We can modify the python script to be smarter and handle the single file case, but that will take some time to implement the change and test it. You can download just that single file with cURL, like

curl --fail --header "Authorization: Bearer YOUR_DOWNLOAD_TOKEN" -sS -L -b session --get https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5200/VNP02MOD/2022/365/VNP02MOD.A2022365.2142.002.2023001063801.nc > VNP02MOD.A2022365.2142.002.2023001063801.nc

or use wget. The LAADSWeb archive pages have examples of how to use wget.