Earthdata Forum

How frequently and at what times does the ASF database update with new Sentinel-1 data? I'm running a global monitoring program which queries the ASF for new Sentinel-1 data daily, when I developed the data pipeline I found the the database seemed to update all at once at the same time every day so I start running my queries at 0100 PST daily which seemed to grab all the data from the previous day, I've been running my program at this scale for just over a year with few issues. Recently in the past couple months, I've found that my queries are often coming up empty. Given that my monitoring project is global in scale I don't think this is likely to be due to no data being acquired in those regions. I'm wondering if there was a change to how/when new data is being made available on the platform.

Sentinel-1 products are ingested at ASF as they are received from ESA. These then become available to download immediately. This process is continuous unless there is a problem with our data "pipeline."

We recently experienced one of these problems, where the volume of data received from ESA was reduced because of changes ESA made in the way they distribute data. This issue has been resolved, and we are now receiving data as before. We're working through a backlog of products that resulted from the slowdown but should get caught up over this weekend.

This could explain why some of your queries in the past couple of weeks have come up empty. But if you've been experiencing this for a couple of months, then something else may be going on. ASF has not made any changes to our search API or the asf_search module. Perhaps ESA has changed Sentinel-1 acquisition planning so that some areas that used to be covered no longer are? You can check this by using their acquisition planning webpage (https://sentinels.copernicus.eu/web/sentinel/copernicus/sentinel-1/acquisition-plans), which provides KML files showing past observation activity.

If this isn't the case, would you provide a copy of the query you've been using? You can do this through a private post if you prefer.

Thanks,
Bill Hauer

Hi Bill,
Thank you for the response. The issue does seem to have been resolved and my pipeline has begun working as expected again.

My download script is located here:
https://github.com/drewbranson/ShipDetection/blob/main/scripts/asf_download.py

It ingests a series of shapefiles to use for the bounds or each Area of Interest and reads a textfile containing the date when the last query ran for it's AOI. I'm guessing the problem is that it doesn't query specific granules but rather all images within the time window since it last ran. If the images were made available for download late then when the script runs it would not grab them and would update the search window so they are not downloaded when they are made available.

This is a fully automated pipeline and since there were no errors that would arise from this issue I would need to do some deep digging to find out exactly when this started. Judging by the final product, I began seeing gaps in my data about 60 days ago +-12 days but there have also been some other issues I've noticed within my own system that could have contributed to it (ie. power outage, reduced internet speeds both of which caused system breaking issues).

It's interesting that the data becomes available immediately, as I did some experimenting when I first developed my download script almost 2 years ago and found that everything seemed to be made available at the same time every day, but my testing was not extensive so I could have just been wrong, or things may have changed in the past 2 years.

Thank you, Drew! Glad it is working for you, it appears that all issues with downloading have been resolved. As always please don't hesitate to reach out if there is ever a question or issue.

Good day,

I just noticed that my system has run into the same download issue beginning on 2024-01-03 and still present this morning. I should probably program a check to deal with this in the future, but just an FYI.

Thanks,
Drew

Hello Drew,
A system issue that developed around the start of the new year affected the transfer of Sentinel-1 data from ESA. This was resolved, and the transfer resumed. However, the backlog of data that resulted took some time to clear, and I suspect this is the cause of the delay in finding new data you observed.

Thanks,
Bill