How to resolve error 500: An error occurred while communicating with Azure Storage errors from a durable function?
We were getting constant communication errors from our blob store linked to an Azure function app and from the exception message it seems to originate from the durable functions library which we use to persist data for function executions.
Previously we would occasionally get the exception, amounting to a couple of failed request every hour. Which for our data volumes is not really a big problem. However 2 days ago it then jumped up to nearly 100 per hour for the last 2 days before it dropped back down. Looking into it, our throughput volumes did not change. Nothing regarding the data that is being sent or the configuration of either resources changed.
Has anyone else experienced the same behavior with the blob storage?
Full exception message returned
Exception while executing function: Functions.FunctionName Result: Failure Exception: The operation failed with an unexpected status code: 500. Details: {"Message":"Something went wrong while processing your request","ExceptionMessage":"An error occurred while communicating with Azure Storage","ExceptionType":"DurableTask.AzureStorage.Storage.DurableTaskStorageException","StackTrace":" at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure[T] in //src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 46\r\n at DurableTask.AzureStorage.Storage.Blob.DownloadStreamingAsync(CancellationToken cancellationToken) in //src/DurableTask.AzureStorage/Storage/Blob.cs:line 119\r\n at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(Blob blob, CancellationToken cancellationToken) in //src/DurableTask.AzureStorage/MessageManager.cs:line 237\r\n at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.ConvertFromAsync(OrchestrationInstanceStatus orchestrationInstanceStatus, String instanceId, CancellationToken cancellationToken) in //src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 474\r\n at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.FetchInstanceStatusInternalAsync(String instanceId, Boolean fetchInput, CancellationToken cancellationToken) in //src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 417\r\n at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.GetStateAsync(String instanceId, Boolean allExecutions, Boolean fetchInput, CancellationToken cancellationToken)+MoveNext() in //src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 373\r\n at System.Linq.AsyncEnumerable.