How to trigger an Azure Data Factory pipeline through API using praameters

Question

How to trigger an Azure Data Factory pipeline through API using praameters

Datta, Andolina 0

Hello All,

I have a use case where I want to trigger an Azure Data Factory pipeline through API. Right now I am calling the API in Databricks and using Service Principal(token based) to connect to ADF from Databricks.The ADF pipeline has some parameters which I want to pass via the API.

The issue is when I run the notebook, the ADF pipeline gets triggered, but the parameters get passed as empty values or ''. Below is the code for POST method to trigger the pipeline (some details have been masked for security reasons):

tenant_id = dbutils.secrets.get(scope="adb-dev-secret-scope",key="spn-databricks-dev-eastus2-001-tenantid")
client_id = dbutils.secrets.get(scope="adb-dev-secret-scope",key="spn-databricks-dev-eastus2-001-clientid")
client_secret = dbutils.secrets.get(scope="adb-dev-secret-scope",key="spn-databricks-dev-eastus2-001-secret")

subscription_id = "****"
resource_group = "****"
factory_name = "****"
pipeline_name = "Test Adf from databricks"
api_version = "2018-06-01"


# Optional: parameters to pass to pipeline
parameters = {
    'curr_working_user': 'xyz'
}

# === Get token using client secret auth ===
credential = ClientSecretCredential(tenant_id, client_id, client_secret)
token = credential.get_token("https://management.azure.com/.default").token


url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.DataFactory/factories/{factory_name}/pipelines/{pipeline_name}/createRun?api-version={api_version}"

headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json"
}


body = {
    'parameters': f'{parameters}'
}

print(json.dumps(body, indent=1))


response = requests.post(url, headers=headers, json=body)
print(response.status_code)
print(response.json())

I want the parameter "curr_working_user" to be passed as 'xyz' in ADF pipeline. The parameter in ADF is same: curr_working_user.

You help is greatly appreciated!!

Thanks,

Andolina

Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-15T18:46:47.44+00:00
@Datta, Andolina

To trigger an Azure Data Factory (ADF) pipeline through an API and pass parameters, you need to ensure that the parameters are formatted correctly in the request body. In your current code, the parameters are being converted to a string with f'{parameters}', which is likely causing them to be passed as empty values.

Instead, you should pass the parameters as a dictionary directly in the JSON body without converting it to a string. Here's how you can modify your body definition:

body = { 'parameters': parameters }

This ensures the API receives the parameters in the expected format.

Corrected Code Snippet:

body = { 'parameters': { 'curr_working_user': 'xyz' } }

After this change, the curr_working_user parameter should be passed correctly to your ADF pipeline.

For more details refer: Pipeline parameters and variables in Azure Data Factory and Azure Synapse Analytics

Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Datta, Andolina 0 Reputation points

2025-05-15T19:24:39.21+00:00

Hi Ganesh,

Thank you for replying. I have tried to pass the parameters both directly and indirectly and still facing the same issue. Right now, I have passed it exactly as you have mentioned.

I have put a Set Variable in ADF to check the parameter value and it is still getting passed as Null.

For the set variable I am passing: @pipeline().parameters.curr_working_user

Kindly let me know if I am missing something.

Thanks,

Andolina
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-16T17:25:21.2633333+00:00

@Datta, Andolina

Thank you for the update and for confirming that you're passing the parameters in the correct format.

From the screenshots you’ve shared, I can see that - The pipeline is being triggered successfully via the API. The parameter curr_working_user is present in the request payload. However, inside the ADF pipeline, the value appears as null.

This typically happens if the parameter is not explicitly defined in the pipeline.

Could you please confirm the following:

In your ADF pipeline, the parameter curr_working_user is defined under the “Parameters” tab (top pane) with the exact name and correct type (e.g., String).

If the parameter is not declared in the pipeline definition, it won't receive the value even if it is passed in the API call.

Reference: https://learn.microsoft.com/en-us/azure/data-factory/concepts-parameters-variables

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Datta, Andolina 0 Reputation points

2025-05-16T17:50:10.4766667+00:00

Hi Ganesh,

In my pipeline the parameter is declared exactly as "curr_working_user" and passed as string.

When the pipeline is triggered you can see it is passing the value as null. Below is the parameter value from the triggered pipeline.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-16T21:14:39.11+00:00
@Datta, Andolina

Thanks again for sharing the details.
Based on what I see, it looks like you're very close to the correct setup.

Just to confirm, Azure Data Factory’s REST API requires strict JSON formatting — meaning all keys and string values in the request body must use double quotes.

Please ensure your payload is constructed like this:

body = { "parameters": { "curr_working_user": "xyz" } }

Using single quotes (or converting a dictionary to a string using f'{parameters}') can result in ADF treating the parameters as invalid or empty, leading to null in the pipeline.

I hope this information helps.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-19T10:00:38.1333333+00:00

@Datta, Andolina

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-20T16:38:25.37+00:00

@Datta, Andolina

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Datta, Andolina 0 Reputation points

2025-05-20T17:11:31.0266667+00:00
Hi Ganesh,

Apologies for the delay in response. Yes I have used double quotes and single quotes for each keys and values and for both, I get the same result.

I have noticed that even though I pass something else as a key(eg. current_working_id) in databricks and use current-working_user as parameter in ADF, the adf pipeline doesn't throw an error. Due to this I suspect the request payload is not passing the "json" at all.

Also I am using the below for tokens. Is there an incompatibility with this method ""ClientSecretCredential" and parameter?

credential = ClientSecretCredential(tenant_id, client_id, client_secret) token = credential.get_token("https://management.azure.com/.default").token

Kindly let me know your thoughts.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-20T17:49:44.07+00:00

@Datta, Andolina

Thanks a lot for your detailed update and no worries at all about the delay.

You're absolutely right to question what's happening here. If you're passing a different parameter name like current_working_id and ADF doesn't throw an error, it likely means that the parameters block isn't actually being received by ADF. That would explain why you're seeing null even when everything looks correct.

It's important to use json=body in the requests.post() method that ensures it's sent as proper JSON. Using data= or converting it to a string might silently cause the parameters to be ignored.

The method you're using ClientSecretCredential is totally fine and fully supported. You're getting the token for the correct scope (https://management.azure.com/.default), so there's no issue there if you're getting a 200 OK and a runId.

I hope this information helps.
Datta, Andolina 0 Reputation points

2025-05-20T18:59:59.18+00:00

Hi Ganesh,

I have used json=body and also tested with passing the entire json instead of body. It is not making any difference. I used data=body and it gave me some error as probably it expected a string rather than json. Kindly help as I am really lost here.

Andolina

Mihir Saxena 0 Microsoft External Staff Moderator

Hi @Datta, Andolina
Your observations are correct.

If you're triggering an Azure Data Factory pipeline via REST API and see null for parameters in the ADF Monitor view, it's likely due to how the request body is constructed in your code.

This strongly indicates that the pipeline trigger succeeded, but the JSON body was never received by the ADF REST endpoint.

In your shared code you're using f'{parameters}'. This implicitly converts the dictionary into a string, resulting in a JSON payload like:


{

  "parameters": "{'curr_working_user': 'xyz'}"

}

This is not valid JSON. ADF expects the parameters key to contain a nested JSON object, not a stringified dictionary. So you need to replace it with:


body = {

  'parameters': parameters

}

So your full corrected snippet must look like:


import requests

from azure.identity import ClientSecretCredential

tenant_id = dbutils.secrets.get(scope="adb-dev-secret-scope", key="spn-databricks-dev-eastus2-001-tenantid")

client_id = dbutils.secrets.get(scope="adb-dev-secret-scope", key="spn-databricks-dev-eastus2-001-clientid")

client_secret = dbutils.secrets.get(scope="adb-dev-secret-scope", key="spn-databricks-dev-eastus2-001-secret")

subscription_id = "****"

resource_group = "****"

factory_name = "****"

pipeline_name = "Test Adf from databricks"

api_version = "2018-06-01"

parameters = {

  "curr_working_user": "xyz"

}

credential = ClientSecretCredential(tenant_id, client_id, client_secret)

token = credential.get_token("https://management.azure.com/.default").token

url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.DataFactory/factories/{factory_name}/pipelines/{pipeline_name}/createRun?api-version={api_version}"

headers = {

    "Authorization": f"Bearer {token}",

    "Content-Type": "application/json"

}

body = {

    "parameters": parameters

}

print("Sending payload:")

print(json.dumps(body, indent=2))  # Debug

response = requests.post(url, headers=headers, json=body)

print(response.status_code)

print(response.json())

Because no matter the parameter type in ADF (String, Int, etc.), the structure of the REST request must be valid JSON.

Check for reference - Pipelines - Create Run

Datta, Andolina 0 Reputation points

2025-05-22T14:37:35.92+00:00
Hi Mihir,

Thank you for your response. I believe my initial post created the confusion. I started with exactly how you have suggested and later tried various ways to create the json and pass it in the body. It is always passing as null even if I pass it the JSON as:

body = {

"parameters": parameters

}

parameters = {

"curr_working_user": "xyz"

}

If you see the previous discussions with Ganesh, I have tried all those options. I have even pass the entire body directly in the POST request. Looks like a bug in the ADF API.

FYI, I have tried to pass the same using Postman and still got null as parameter.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-23T06:24:27.6033333+00:00
Thanks for sharing the details @Datta, Andolina

If the issue still occurs even when using Postman, then the issue is might be with how the Azure Data Factory pipeline is interpreting or receiving the payload. Try below troubleshooting methods as well:

Kindly check the Name of the parameter (curr_working_user) once more as it is case-sensitive and make sure that there are no leading/trailing spaces.

Try triggering the pipeline via Azure CLI as well:

az login az datafactory pipeline create-run \ --resource-group <resource-group> \ --factory-name <factory-name> \ --name <pipeline-name> \ --parameters curr_working_user='xyz'
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-26T09:45:35.7966667+00:00

Hey @Datta, Andolina
We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-27T05:55:15.3766667+00:00

Hey @Datta, Andolina
We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

2 answers

Your answer

Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-15T18:46:47.44+00:00

@Datta, Andolina

To trigger an Azure Data Factory (ADF) pipeline through an API and pass parameters, you need to ensure that the parameters are formatted correctly in the request body. In your current code, the parameters are being converted to a string with f'{parameters}', which is likely causing them to be passed as empty values.

Instead, you should pass the parameters as a dictionary directly in the JSON body without converting it to a string. Here's how you can modify your body definition:

body = { 'parameters': parameters }

This ensures the API receives the parameters in the expected format.

Corrected Code Snippet:

body = { 'parameters': { 'curr_working_user': 'xyz' } }

After this change, the curr_working_user parameter should be passed correctly to your ADF pipeline.

For more details refer: Pipeline parameters and variables in Azure Data Factory and Azure Synapse Analytics

Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Datta, Andolina 0 Reputation points

2025-05-15T19:24:39.21+00:00

Hi Ganesh,

Thank you for replying. I have tried to pass the parameters both directly and indirectly and still facing the same issue. Right now, I have passed it exactly as you have mentioned.

I have put a Set Variable in ADF to check the parameter value and it is still getting passed as Null.

For the set variable I am passing: @pipeline().parameters.curr_working_user

Kindly let me know if I am missing something.

Thanks,

Andolina
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-16T17:25:21.2633333+00:00

@Datta, Andolina

Thank you for the update and for confirming that you're passing the parameters in the correct format.

From the screenshots you’ve shared, I can see that - The pipeline is being triggered successfully via the API. The parameter curr_working_user is present in the request payload. However, inside the ADF pipeline, the value appears as null.

This typically happens if the parameter is not explicitly defined in the pipeline.

Could you please confirm the following:

In your ADF pipeline, the parameter curr_working_user is defined under the “Parameters” tab (top pane) with the exact name and correct type (e.g., String).

If the parameter is not declared in the pipeline definition, it won't receive the value even if it is passed in the API call.

Reference: https://learn.microsoft.com/en-us/azure/data-factory/concepts-parameters-variables

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Datta, Andolina 0 Reputation points

2025-05-16T17:50:10.4766667+00:00

Hi Ganesh,

In my pipeline the parameter is declared exactly as "curr_working_user" and passed as string.

When the pipeline is triggered you can see it is passing the value as null. Below is the parameter value from the triggered pipeline.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-16T21:14:39.11+00:00

@Datta, Andolina

Thanks again for sharing the details.
Based on what I see, it looks like you're very close to the correct setup.

Just to confirm, Azure Data Factory’s REST API requires strict JSON formatting — meaning all keys and string values in the request body must use double quotes.

Please ensure your payload is constructed like this:

body = { "parameters": { "curr_working_user": "xyz" } }

Using single quotes (or converting a dictionary to a string using f'{parameters}') can result in ADF treating the parameters as invalid or empty, leading to null in the pipeline.

I hope this information helps.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-19T10:00:38.1333333+00:00

@Datta, Andolina

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-20T16:38:25.37+00:00

@Datta, Andolina

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.
Datta, Andolina 0 Reputation points

2025-05-20T17:11:31.0266667+00:00

Hi Ganesh,

Apologies for the delay in response. Yes I have used double quotes and single quotes for each keys and values and for both, I get the same result.

I have noticed that even though I pass something else as a key(eg. current_working_id) in databricks and use current-working_user as parameter in ADF, the adf pipeline doesn't throw an error. Due to this I suspect the request payload is not passing the "json" at all.

Also I am using the below for tokens. Is there an incompatibility with this method ""ClientSecretCredential" and parameter?

credential = ClientSecretCredential(tenant_id, client_id, client_secret) token = credential.get_token("https://management.azure.com/.default").token

Kindly let me know your thoughts.
Ganesh Gurram 7,140 Reputation points Microsoft External Staff Moderator

2025-05-20T17:49:44.07+00:00

@Datta, Andolina

Thanks a lot for your detailed update and no worries at all about the delay.

You're absolutely right to question what's happening here. If you're passing a different parameter name like current_working_id and ADF doesn't throw an error, it likely means that the parameters block isn't actually being received by ADF. That would explain why you're seeing null even when everything looks correct.

It's important to use json=body in the requests.post() method that ensures it's sent as proper JSON. Using data= or converting it to a string might silently cause the parameters to be ignored.

The method you're using ClientSecretCredential is totally fine and fully supported. You're getting the token for the correct scope (https://management.azure.com/.default), so there's no issue there if you're getting a 200 OK and a runId.

I hope this information helps.
Datta, Andolina 0 Reputation points

2025-05-20T18:59:59.18+00:00

Hi Ganesh,

I have used json=body and also tested with passing the entire json instead of body. It is not making any difference. I used data=body and it gave me some error as probably it expected a string rather than json. Kindly help as I am really lost here.

Andolina
Datta, Andolina 0 Reputation points

2025-05-22T14:37:35.92+00:00

Hi Mihir,

Thank you for your response. I believe my initial post created the confusion. I started with exactly how you have suggested and later tried various ways to create the json and pass it in the body. It is always passing as null even if I pass it the JSON as:

body = {

"parameters": parameters

}

parameters = {

"curr_working_user": "xyz"

}

If you see the previous discussions with Ganesh, I have tried all those options. I have even pass the entire body directly in the POST request. Looks like a bug in the ADF API.

FYI, I have tried to pass the same using Postman and still got null as parameter.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-23T06:24:27.6033333+00:00

Thanks for sharing the details @Datta, Andolina

If the issue still occurs even when using Postman, then the issue is might be with how the Azure Data Factory pipeline is interpreting or receiving the payload. Try below troubleshooting methods as well:

Kindly check the Name of the parameter (curr_working_user) once more as it is case-sensitive and make sure that there are no leading/trailing spaces.

Try triggering the pipeline via Azure CLI as well:

az login az datafactory pipeline create-run \ --resource-group <resource-group> \ --factory-name <factory-name> \ --name <pipeline-name> \ --parameters curr_working_user='xyz'
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-26T09:45:35.7966667+00:00

Hey @Datta, Andolina
We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Mihir Saxena 0 Reputation points Microsoft External Staff Moderator

2025-05-27T05:55:15.3766667+00:00

Hey @Datta, Andolina
We haven’t heard from you on the last response and wanted to follow up to check if your issue has been resolved. In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Answer 1

ADF does not throw an error when an expected parameter is missing or mismatched it just ignores it or sets it as null, which can lead to silent failures.

Since the issue persists even when using Postman, it strongly suggests that the ADF pipeline is either not interpreting the request payload correctly or is not receiving it as expected.

Try below troubleshooting methods as well:

Parameter Name Validation: Ensure that the parameter name in the ADF pipeline definition is exactly curr_working_user:

Parameter names are case-sensitive.
Ensure there are no leading or trailing spaces.

Try triggering the pipeline via Azure CLI: If this succeeds and the parameter is received, the issue likely lies in how your Databricks/REST API code is crafting the payload.


az login

az datafactory pipeline create-run \

  --resource-group <resource-group> \

  --factory-name <factory-name> \

  --name <pipeline-name> \

  --parameters curr_working_user='xyz'

Enable diagnostics logging for ADF and route the logs to a Log Analytics workspace or Storage Account to inspect raw requests. This helps capture the exact request structure being received by ADF. Kindly refer - Diagnostics settings in Azure Monitor

Datta, Andolina 0 Reputation points

2025-05-29T13:00:27.05+00:00

Thank you Mihir. This is a good information that ADF just might not be interpreting the request payload. But I am putting the payload as is recommended in MS docs(which doesn't have much detail though) and other websites. May be the request is not accepted as it has been mentioned in the websites or may be we need to pass some additional keys and values to get it accepted. For now, I can confirm that the parameters in ADF and Databricks are in same case in both places with no leading/trailing spaces. Let me try the CLI commands and check. Meanwhile I have raised an MS ticket already and checking the issue with them. If I get any answers from them, I will surely provide the solution here.

Thanks,

Andolina

Answer 2

Hello All,

Thank you for all your suggestions on this thread.

We have raised the issue as a product bug to Microsoft. They did not update us as to why parameters do not work with http request. However they did demo to us that parameters work with python script.

https://learn.microsoft.com/en-us/rest/api/datafactory/pipelines/create-run?view=rest-datafactory-2018-06-01&tabs=Python

This resolved our problem and I have tested by passing multiple parameters through databricks to ADF and it was taking the parameters correctly.

The document doesn't mention how to write the parameters; you have to write it inside the response. Eg:

##code******************

from azure.identity import DefaultAzureCredential

from azure.mgmt.datafactory import DataFactoryManagementClient

print("Running ADF Pipeline....")

tenant_id = ***

client_id = ***

client_secret = ***

parameters = {

'curr_working_user': f'{current_user}',

'ServerName': f'{servername}',

'SharedFolder': f'{shared_folder}',

'Folder': f'{folder}'

}

def main():

client = DataFactoryManagementClient(

    credential=ClientSecretCredential(tenant_id, client_id, client_secret),

    subscription_id = "***",

)

response = client.pipelines.create_run(

    resource_group_name = "***",

    factory_name = "***",

    pipeline_name = "Test Adf from databricks",

    parameters = parameters or {}

)

print(response)

if name == "main":

main()

Share via

How to trigger an Azure Data Factory pipeline through API using praameters

2 answers

Your answer