ASDC Response: 401/Unauthorized when using python3

[edwardgrys]

I am trying to download CALIPSO data from ASDC using a python. scripts based on the examples on this forum. My example code is very simple

import requests
session = requests.Session()
session.auth = (<my username>, <my password>)

url = 'https://asdc.larc.nasa.gov/data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf'
r1 = session.get(url)

This returns a 401 response (unauthorised). I have my username and password correct, as I am able to download a MODIS file using the same code

url = 'https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/61/MYD08_D3/2020/001/MYD08_D3.A2020001.061.2020003014337.hdf'
r1 = session.request('get', url)

Is a status: 200 (success). I can download this file using the wget instructions, so this appears to be a problem with the python code, but given this is a basic example, I am not sure what the issue is. The library debug output is below (with some bit removed)

Many thanks
Ed

----

In[9]: session.get(url)
DEBUG:urllib3.connectionpool:Resetting dropped connection: asdc.larc.nasa.gov
send: b'GET /data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf HTTP/1.1\r\nHost: asdc.larc.nasa.gov\r\nUser-Agent: python-requests/2.27.1\r\nAccept-Encoding: gzip, deflate, br\r\nAccept: */*\r\nConnection: keep-alive\r\nCookie: ----------------\r\nAuthorization: Basic -------------------\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date: Thu, 17 Mar 2022 22:55:32 GMT
header: Server: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips
header: Content-Length: 381
header: Content-Type: text/html; charset=iso-8859-1
header: Strict-Transport-Security: max-age=31536000;includeSubDomains;preload
DEBUG:urllib3.connectionpool:https://asdc.larc.nasa.gov:443 "GET /data/CALIPSO/LID_L1-Standard-V4-10/2007/01/CAL_LID_L1-Standard-V4-10.2007-01-01T00-22-49ZN.hdf HTTP/1.1" 401 381
Out[9]: <Response [401]>

[ASDC - cheyenne.e.land]

Hello,

Just from looking at your code, you might have a typo. You say that the MODIS file works using the same code but for CALIPSO, it says that r1 = session.get(url) and for MODIS it says r1 = session.request('get', url). We have also notified a subject matter expert on your inquiry. Thank you.

Regards,
ASDC User Services

[edwardgrys]

Thanks for the quick reply! That was from me doing some experimenting, but unfortunately it doesn't fix the issue (I think one method is actually just calling the other)

[ASDC - cheyenne.e.land]

Hello,

Just wanted to make sure that wasn't the issue and again, a subject matter expert has been notified about your inquiry and will get back with you soon.

Regards,
ASDC User Services

[edwardgrys]

For anyone else who finds this question, it appears that a solution is to make the call to 'get' without supplying any authentication information.

session = Session()
response = session.get(url)
with open(outputfilename, 'wb') as f:
f.write(response.content)

appears to work for me when downloading CALIPSO data. I assume only unauthenticated access is not the intention though, so this might be changed in the future.

[njester]

For anyone else who finds this question, it appears that a solution is to make the call to 'get' without supplying any authentication information.

session = Session()
response = session.get(url)
with open(outputfilename, 'wb') as f:
f.write(response.content)

appears to work for me when downloading CALIPSO data. I assume only unauthenticated access is not the intention though, so this might be changed in the future.

You may want to verify that it downloaded what you think it downloaded. If you download without passing credentials, you probably downloaded a redirect page asking you to log in. Recently, systems were updated to require a token for login instead of credentials and I'm still in the process of testing and updating example scripts. You can find an updated Python example that can be used to download part a campaign here:
viewtopic.php?t=2330&sid=51ba8fe08ac1b2fc40bec7a8eee35a16
The script was just recently udpated, so if you have any issues with the script or forum post, let me know.

[edwardgrys]

Thanks for the quick response - I had originally tried using an Earthdata token (I have a working script for downloading MODIS data from LAADS), but it didn't seem to work for downloading the CALIPSO data.

Good point about the download - I checked though and the files contain what looks like the correct lidar data. I am happy to send over the actual requests made if that is useful.