Failed to download OPERA_L2_RTC-S1

[gabor]

Hi there, I am a newbie in this forum and won't be probably able to apply the correct protocol how to ask here. Pls be patient.

I am trying to download via Python VV bands from product Opera_S1_RTC. The login credentials seem to be ok, but I get following error:
Failed to download OPERA_L2_RTC-S1_T042-088916-IW3_20231114T140829Z_20231116T163134Z_S1A_30_v1.0_VV.tif: No connection adapters were found for '/vsicurl/https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T042-088916-IW3_20231114T140829Z_20231116T163134Z_S1A_30_v1.0_VV.tif'

Here the code (jupyter notebook) with the response:

import asf_search
import os
from getpass import getpass
import requests

# Define search options
options = {
'intersectsWith': 'POLYGON((-122.5306 37.6278,-122.1372 37.6207,-122.1372 37.9566,-122.5287 37.9554,-122.5306 37.6278))',
'dataset': 'OPERA-S1',
'start': '2023-11-14T14:00:00Z',
'end': '2023-11-14T15:00:00Z',
'flightDirection': 'DESCENDING',
'processingLevel': 'RTC',
'maxResults': '1000'
}

# Perform the search
results = asf_search.search(**options)
print(f'Found {len(results)} products that match your search criteria')

# Filter results by polarization
polarization = 'VV'
urls = []
for result in results:
for url in result.umm['RelatedUrls']:
if url['URL'].startswith('https://') and url['URL'].endswith(f'{polarization}.tif'):
urls.append('/vsicurl/' + url['URL'])

# Set up environment variables for GDAL
os.environ['GDAL_HTTP_COOKIEFILE'] = '/tmp/cookies.txt'
os.environ['GDAL_HTTP_COOKIEJAR'] = '/tmp/cookies.txt'
os.environ['GDAL_HTTP_NETRC'] = 'YES'
os.environ['GDAL_HTTP_NETRC_FILE'] = 'temp_netrc'

# Get NASA Earthdata Login credentials
username = getpass('NASA Earthdata Login username: ')
password = getpass('NASA Earthdata Login password: ')

# Write credentials to the netrc file
with open(os.environ['GDAL_HTTP_NETRC_FILE'], 'w') as f:
f.write(f'machine urs.earthdata.nasa.gov login {username} password {password}')

with open(os.environ['GDAL_HTTP_NETRC_FILE'], 'r') as f:
content = f.read()
if username in content and password in content:
print("Credentials correctly written to netrc file.")
else:
print("Credentials not found in netrc file.")

# Function to download a file
def download_file(url, local_filename):
with requests.Session() as session:
session.auth = (username, password)
response = session.get(url, stream=True)
response.raise_for_status()
with open(local_filename, 'wb') as f:
for chunk in response.iter_content(chunk_size=8192):
f.write(chunk)

# Download the files
for url in urls:
# Extract the filename from the URL
filename = url.split('/')[-1]
print(f'Downloading {filename}...')
try:
download_file(url, filename)
print(f'Successfully downloaded {filename}')
except requests.exceptions.RequestException as e:
print(f'Failed to download {filename}: {e}')

Response:
Found 3 products that match your search criteria
NASA Earthdata Login username: ········
NASA Earthdata Login password: ········
Credentials correctly written to netrc file.
Downloading OPERA_L2_RTC-S1_T042-088917-IW3_20231114T140831Z_20231116T163134Z_S1A_30_v1.0_VV.tif...
Failed to download OPERA_L2_RTC-S1_T042-088917-IW3_20231114T140831Z_20231116T163134Z_S1A_30_v1.0_VV.tif: No connection adapters were found for '/vsicurl/https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T042-088917-IW3_20231114T140831Z_20231116T163134Z_S1A_30_v1.0_VV.tif'
Downloading OPERA_L2_RTC-S1_T042-088916-IW3_20231114T140829Z_20231116T163134Z_S1A_30_v1.0_VV.tif...
Failed to download OPERA_L2_RTC-S1_T042-088916-IW3_20231114T140829Z_20231116T163134Z_S1A_30_v1.0_VV.tif: No connection adapters were found for '/vsicurl/https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T042-088916-IW3_20231114T140829Z_20231116T163134Z_S1A_30_v1.0_VV.tif'
Downloading OPERA_L2_RTC-S1_T042-088915-IW3_20231114T140826Z_20231116T163134Z_S1A_30_v1.0_VV.tif...
Failed to download OPERA_L2_RTC-S1_T042-088915-IW3_20231114T140826Z_20231116T163134Z_S1A_30_v1.0_VV.tif: No connection adapters were found for '/vsicurl/https://datapool.asf.alaska.edu/RTC/OPERA-S1/OPERA_L2_RTC-S1_T042-088915-IW3_20231114T140826Z_20231116T163134Z_S1A_30_v1.0_VV.tif'

Here some additional parameters:
Download attempt: 2025-02-11
OS: macOS 15.1.1
Python version: 12.9

Help is very much appreciated! Thanks in advance!

[bbuechle]

Gabor

Sorry for the delayed response, I'll start with the summary: Dropping the prepending of "/vsicurl/" will likely resolve your immediate issue.

vsicurl is a virtual file system scheme that is used by tools like gdal to facilitate filesystem-like seekable IO on a networked (non-local) resource. This can be useful for avoiding downloading files or sparsely accessing within an iteration. However, as your error suggests, Python Requests does not support vsicurl.

If downloading the file, and not iterating on it as a filesystem object over the network, is your primary objective, asf_search has several built in tools do simplify managing the downloads for you.

Brian

[gabor]

Hi Brian,

thanks so much! You are right, I just had to remove the vsicurl prefix:

urls.append('/vsicurl/' + url['URL']) > urls.append(url['URL']) # Remove '/vsicurl/' prefix

Best,
gabor

[cole_speed]

Hi Brian,

I have a related question to the above, except I am seeking to stream the ASF-hosted OPERA products rather than download them. I am quite familiar with asf_search; however, I don't think it will provide me exactly what I am hoping to achieve here, as I am not wanting to download locally. Specifically, I am interested in programmatically streaming the OPERA data (specifically the CSLC/RTC .h5 products) using the ASF data pool URL and /vsicurl via rasterio. When running the below, I am receiving a 403 Forbidden error:

from osgeo import gdal
import rasterio as rio

gdal.SetConfigOption("GDAL_HTTP_COOKIEFILE", "cookies.txt")
gdal.SetConfigOption("GDAL_HTTP_COOKIEJAR", "cookies.txt")
gdal.SetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN", "TRUE")
gdal.SetConfigOption("CPL_VSIL_CURL_USE_HEAD", "NO")
gdal.SetConfigOption("CPL_VSIL_CURL_AUTHORIZATION_HEADER_ALLOWED_IF_REDIRECT", "YES")

ds = rio.open("https://datapool.asf.alaska.edu/CSLC/OPERA-S1/OPERA_L2_CSLC-S1_T173-370301-IW1_20160702T134318Z_20240611T083135Z_S1A_VV_v1.1.h5")

RasterioIOError: HTTP response code: 403

I do not encounter this issue when streaming other OPERA products hosted by PO.DAAC for example:

ds = rasterio.open("https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/OPERA_L3_DSWX-HLS_PROVISIONAL_V1/OPERA_L3_DSWx-HLS_T11SPA_20230409T181446Z_20230411T155818Z_L8_30_v1.0_B01_WTR.tif")

from osgeo import gdal
import rasterio as rio

gdal.SetConfigOption("GDAL_HTTP_COOKIEFILE", "cookies.txt")
gdal.SetConfigOption("GDAL_HTTP_COOKIEJAR", "cookies.txt")
gdal.SetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN", "TRUE")
gdal.SetConfigOption("CPL_VSIL_CURL_USE_HEAD", "NO")
gdal.SetConfigOption("CPL_VSIL_CURL_AUTHORIZATION_HEADER_ALLOWED_IF_REDIRECT", "YES")

ds = rio.open("https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/OPERA_L3_DSWX-HLS_PROVISIONAL_V1/OPERA_L3_DSWx-HLS_T11SPA_20230409T181446Z_20230411T155818Z_L8_30_v1.0_B01_WTR.tif")

This streams the data into memory successfully.

I am wondering if the authorization requirements are different for ASF-hosted data and if there is any way to stream the ASF data using a method similar my approach above. I have my .netrc file set up locally and all appears to be in working order. Are you aware of a way to stream RTC/CSLC with rasterio/xarray/h5py that circumvents this 403 Forbidden error for ASF datasets?

Thanks in advance for any insights.

-Cole

[bbuechle]

This is an interesting one!

I'm not actually able to reproduce the 403 error. It may be possible that 403 indicates you haven't approved the ASF data access EULA in EDL. Be sure that you can manually download the data using your provided credentials.

When I try to download using your provided code, I get a 401 error. Honestly, I'm not at all familiar with rasterio, but after a bunch of troubleshooting with `gdal.SetConfigOption("CPL_CURL_VERBOSE", "YES")`, this appears to be, best as I can tell, an issue with rasterio's gdal interface.

When you access OPERA data from ASF, this is the flow:

Code: Select all

1. datapool.asf.alaska.edu ( data ___location service) -> 
2.   cumulus.asf.alaska.edu (cloud distribution) -> 
3.     urs.earthdata.nasa.gov (OAuth2 Service) -> 
4.       cumulus.asf.alaska.edu ( cloud distribution) -> 
5.        AWS cloudfront ( S3 out-of-region Distribution )

When I try to download, no matter what I do, I can't get rasterio+gdal to provide my auth to urs.earthdata.nasa.gov.

If I short circuit the process a bit and start my request at step 3,

Code: Select all

url = 'https://urs.earthdata.nasa.gov/oauth/authorize?client_id=BO_n7nTIlMljdvU6kRRB3g&response_type=code&redirect_uri=https://cumulus.asf.alaska.edu/login&state=%2FCSLC%2FOPERA-S1%2FOPERA_L2_CSLC-S1%2FOPERA_L2_CSLC-S1_T173-370301-IW1_20160702T134318Z_20240611T083135Z_S1A_VV_v1.1%2FOPERA_L2_CSLC-S1_T173-370301-IW1_20160702T134318Z_20240611T083135Z_S1A_VV_v1.1.h5
ds = rio.open(url)

Its also worth noting, that this doesn't seem to affect Sentinel-1 data, which follows a similar path, exchanging cumulus.asf.alaska.edu for sentinel1.asf.alaska.edu. I was able to pull (but obviously not read a EDL/URS restricted sentinel product file no problem.

This isn't a workable solution. But its indicative a deeper problem somewhere in the tool chain, one I don't understand very well.

If you can confirm that you are able to manually download the data, and still getting a 403 error using those credentials, I'll try to escalate the issue to the team that manages cumulus.asf.alaska.edu. Given the different behavior between Sentinel products and OPERA products, there may be thread for them to pull.