Yes Earthdata Granules can be downloaded in parallel, but concurrent connections must be limited to no more than three.
You may experience a "Concurrent Connection Limit Reached" error if you exceed three concurrent connections.
When using curl, the following flags will only use a maximum of three concurrent connections: --parallel --parallel-max 3
When using wget, the 'xargs' tool must be used with the following flags: -n 1 -P 3
When using Python, use the concurrent.futures library to limit the amount of workers to 3 when calling downloads with requests:
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
executor.map(download_granule_function, input_urls_list)
FAQ: Can I Download Earthdata Granules In Parallel?
-
- User Services
- Posts: 23
- Joined: Wed Dec 04, 2024 4:05 pm America/New_York
- Has endorsed: 1 time
- Endorsed: 1 time