Encrypt Azure Data Factory with customer-managed keys

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Azure Data Factory encrypts data at rest, including entity definitions and any data cached while runs are in progress. By default, data is encrypted with a randomly generated Microsoft-managed key that is uniquely assigned to your data factory. For extra security guarantees, you can now enable Bring Your Own Key (BYOK) with customer-managed keys feature in Azure Data Factory. When you specify a customer-managed key (CMK), Data Factory uses both the factory system key and the CMK to encrypt customer data. Missing either would result in Deny of Access to data and factory.

Azure Key Vault is required to store customer-managed keys. You can either create your own keys and store them in a key vault, or you can use the Azure Key Vault APIs to generate keys. Key vault and Data Factory must be in the same Microsoft Entra tenant and in the same region, but they may be in different subscriptions. For more information about Azure Key Vault, see What is Azure Key Vault?

About customer-managed keys

The following diagram shows how Data Factory uses Microsoft Entra ID and Azure Key Vault to make requests using the customer-managed key:

The following list explains the numbered steps in the diagram:

1. An Azure Key Vault admin grants permissions to encryption keys to the managed identity that's associated with the Data Factory
2. A Data Factory admin enables customer-managed key feature in the factory
3. Data Factory uses the managed identity that's associated with the factory to authenticate access to Azure Key Vault via Microsoft Entra ID
4. Data Factory wraps the factory encryption key with the customer key in Azure Key Vault
5. For read/write operations, Data Factory sends requests to Azure Key Vault to unwrap the account encryption key to perform encryption and decryption operations

There are two ways of adding Customer Managed Key encryption to data factories. One is during factory creation time in Azure portal, and the other is post factory creation, in Data Factory UI.

Prerequisites - configure Azure Key Vault and generate keys

Enable Soft Delete and Do Not Purge on Azure Key Vault

Using customer-managed keys with Data Factory requires two properties to be set on the Key Vault, Soft Delete and Do Not Purge. These properties can be enabled using either PowerShell or Azure CLI on a new or existing key vault. To learn how to enable these properties on an existing key vault, see Azure Key Vault recovery management with soft delete and purge protection

If you are creating a new Azure Key Vault through Azure portal, Soft Delete and Do Not Purge can be enabled as follows:

Grant Data Factory access to Azure Key Vault

Make sure Azure Key Vault and Azure Data Factory are in the same Microsoft Entra tenant and in the same region. You can use either access policies or access control permissions:

1. Access policy - In your key vault select Access policies -> Add access Policy -> search for your Azure Data Factory managed identity and grant Get, Unwrap Key, and Wrap Key permissions in the Secret permissions dropdown.

2. Access control - Your managed identity will need two roles in Access control: Key Vault Crypto Service Encryption User and Key Vault Secrets User. In your key vault select Access control (IAM) -> + Add -> Add role assignment. Select one of the roles, and then select Next. Under Members select Managed identity then Select members and search for your Azure Data Factory managed identity. Then select Review + assign. Repeat for the second role.

• If you want to add customer managed key encryption after factory creation in Data Factory UI, ensure data factory's managed service identity (MSI) has the correct permissions to Key Vault
• If you want to add customer managed key encryption during factory creation time in Azure portal, ensure the user-assigned managed identity (UA-MI) has the correct permissions to Key Vault

Generate or upload customer-managed key to Azure Key Vault

You can either create your own keys and store them in a key vault. Or you can use the Azure Key Vault APIs to generate keys. Only RSA keys are supported with Data Factory encryption. RSA-HSM is also supported. For more information, see About keys, secrets, and certificates.

Enable customer-managed keys

Post factory creation in Data Factory UI

This section walks through the process to add customer managed key encryption in Data Factory UI, after factory is created.

1. Make sure that data factory's Managed Service Identity (MSI) has Get, Unwrap Key and Wrap Key permissions to Key Vault.

2. Ensure the Data Factory is empty. The data factory can't contain any resources such as linked services, pipelines, and data flows. For now, deploying customer-managed key to a non-empty factory will result in an error.

3. To locate the key URI in the Azure portal, navigate to Azure Key Vault, and select the Keys setting. Select the wanted key, then select the key to view its versions. Select a key version to view the settings

4. Copy the value of the Key Identifier field, which provides the URI

5. Launch Azure Data Factory portal, and using the navigation bar on the left, jump to Data Factory Management Portal

6. Select the Customer managed key icon

7. Enter the URI for customer-managed key that you copied before

8. Select Save and customer-managed key encryption is enabled for Data Factory

During factory creation in Azure portal

This section walks through steps to add customer managed key encryption in Azure portal, during factory deployment.

To encrypt the factory, Data Factory needs to first retrieve customer-managed key from Key Vault. Since factory deployment is still in progress, Managed Service Identity (MSI) isn't available yet to authenticate with Key Vault. As such, to use this approach, customer needs to assign a user-assigned managed identity (UA-MI) to data factory. We will assume the roles defined in the UA-MI and authenticate with Key Vault.

To learn more about user-assigned managed identity, see Managed identity types and Role assignment for user assigned managed identity.

1. Make sure that User-assigned Managed Identity (UA-MI) has Get, Unwrap Key and Wrap Key permissions to Key Vault

2. Under Advanced tab, check the box for Enable encryption using a customer managed key

3. Provide the url for the customer managed key stored in Key Vault

   Tip

   If you don't pass the key version in the URL after the final '/' (for example: https://mykeyvault.vault.azure.net/keys/cmk/), the version will always default to the latest if the key is updated in the future.

   Currently this is only supported using the Azure portal.

4. Select an appropriate user assigned managed identity to authenticate with Azure Key Vault.

5. Continue with your factory deployment.

Update Key Version

When you create a new version of a key, update data factory to use the new version:

1. Locate the URI for the new key version through Azure Key Vault Portal:

   1. Navigate to Azure Key Vault, and select the Keys setting.
   2. Select the wanted key, then select the key to view its versions.
   3. Select a key version to view the settings.

2. Copy the value of the Key Identifier field, which provides the URI.

3. Launch Azure Data Factory portal, and using the navigation bar on the left, select the Data Factory Management Portal.

4. Select the Customer-managed key setting.

5. Enter the URI for customer-managed key that you copied before.

6. Select Save and Data Factory will now encrypt with the new key version.

Use a different key

To change key used for Data Factory encryption, you have to manually update the settings in Azure Data Factory:

1. Locate the URI for the new key version through Azure Key Vault Portal:

   1. Navigate to Azure Key Vault, and select the Keys setting.
   2. Select the wanted key, then select the key to view its versions.
   3. Select a key version to view the settings.

2. Copy the value of the Key Identifier field, which provides the URI.

3. Launch Azure Data Factory portal, and using the navigation bar on the left, select the Data Factory Management Portal.

4. Select the Customer-managed key setting.

5. Enter the URI for select that you copied before.

6. Select Save and Data Factory will now encrypt with the new key version.

Disable Customer-managed Keys

By design, once the select feature is enabled, you can't remove the extra security step. We will always expect a customer provided key to encrypt factory and data.

Customer managed key and continuous integration and continuous deployment

By default, CMK configuration is not included in the factory Azure Resource Manager (ARM) template. To include the customer managed key encryption settings in ARM template for continuous integration (CI/CD):

1. Ensure the factory is in Git mode
2. Navigate to management portal - customer managed key section
3. Check Include in ARM template option

The following settings will be added in ARM template. These properties can be parameterized in Continuous Integration and Delivery pipelines by editing the Azure Resource Manager parameters configuration

Related content

Go through the tutorials to learn about using Data Factory in more scenarios.