Good afternoon,
I am currently using Dask distributed to query and process HLS data from CloudSTAC with pystac_client and Open Data Cube. I use client.map() to create percentile composites for 7 bands across several timeframes in parallel. However, it usually takes 5-10 tries to start the processing and fails prematurely 90% of the time with the error shown below, failing with a different file almost every time:
rasterio.errors.RasterioIOError: '/vsicurl/https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T10TER.2022137T185536.v2.0/HLS.L30.T10TER.2022137T185536.v2.0.Fmask.tif' not recognized as a supported file format.
My script does run nearly all the way through if I'm lucky. I have followed several HLS tutorials to ensure the correct GDAL environmental variables are set, I am using the rasterio environment, and my Earthdata credentials are accessible to the worker nodes.
I understand the sheer amount of data I'm processing is likely to result in more errors, but in the end, I need to be able to reliably generate these percentile composites for several AOIs in parallel with few interruptions.
Is there any unexpected behavior or are there any things I should be aware of when using HLS data from CloudSTAC with Dask and pystac_client? Do they interfere with each other in some way?
Is there anything I can do on my end to reduce the number of retries required to kick off the processing and keep the processing running more consistently?
Thank you!
Tips for using HLS with Dask distributed and pystac_client?
-
LP DAACx - dgolon
- User Services
- Posts: 422
- Joined: Mon Sep 30, 2019 10:00 am America/New_York
- Has thanked: 31 times
- Been thanked: 8 times
- Contact:
Re: Tips for using HLS with Dask distributed and pystac_client?
Hi @ioyler We have passed your question along to the developers. We will follow up when we have more information. Thanks -- Danielle
Subscribe to the LP DAAC listserv by sending a blank email to lpdaac-join@lists.nasa.gov.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Sign up for the Landsat listserv to receive the most up to date information about Landsat data: https://public.govdelivery.com/accounts/USDOIGS/subscriber/new#tab1.
Re: Tips for using HLS with Dask distributed and pystac_client?
Good morning,
To follow up on this discussion, I've made some advancements that might provide some more insight. I managed to stabilize one version of the image compositing processing by adjusting some of the GDAL settings, specifically by adding GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY to the gdal_config that is passed to the rio_env. I suspect the issue stems from one worker trying to read the file while another one is still using it, triggering the error message. With this version of the processing, I pass a list of 16 unique timeframes to generate composites for and they are generated consecutively in a 'for' loop.
However, I would much rather take advantage of Dask (client.map() in particular) to process all 16 of those time frames concurrently. The processing is still unacceptably unstable with client.map() and results in the same error message even with the two new GDAL settings added to rio_env and even with the GDAL_HTTP_MAX_RETRY set to something ridiculously high. I think there might be another setting that needs to be included or adjusted here to stabilize it. Any thoughts or ideas?
Developers, I hope this helps!
To follow up on this discussion, I've made some advancements that might provide some more insight. I managed to stabilize one version of the image compositing processing by adjusting some of the GDAL settings, specifically by adding GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY to the gdal_config that is passed to the rio_env. I suspect the issue stems from one worker trying to read the file while another one is still using it, triggering the error message. With this version of the processing, I pass a list of 16 unique timeframes to generate composites for and they are generated consecutively in a 'for' loop.
However, I would much rather take advantage of Dask (client.map() in particular) to process all 16 of those time frames concurrently. The processing is still unacceptably unstable with client.map() and results in the same error message even with the two new GDAL settings added to rio_env and even with the GDAL_HTTP_MAX_RETRY set to something ridiculously high. I think there might be another setting that needs to be included or adjusted here to stabilize it. Any thoughts or ideas?
Developers, I hope this helps!
-
kristianbodolai
- Posts: 12
- Joined: Mon May 27, 2024 7:00 am America/New_York
Re: Tips for using HLS with Dask distributed and pystac_client?
Hi!
In my experience - some assets are corrupted or hold no data - you can verify this by download that specific asset and try to open it with gdal, or loading it into QGIS.
An option is to find these assets and remove the items from your search (and probably notifying the data provider so they can reingest it!)
If you use`odc-stac`, you can set the fail_on_error parameter to False, although this can lead to unexpected behaviours down the line.
Hope it helps!
K.
In my experience - some assets are corrupted or hold no data - you can verify this by download that specific asset and try to open it with gdal, or loading it into QGIS.
An option is to find these assets and remove the items from your search (and probably notifying the data provider so they can reingest it!)
If you use`odc-stac`, you can set the fail_on_error parameter to False, although this can lead to unexpected behaviours down the line.
Hope it helps!
K.