The ADF Databricks job Activity doesn't seem to pass parameters

Question

The ADF Databricks job Activity doesn't seem to pass parameters

Debie, Hanneke 35

I have created an ADF pipeline that uses the databricks job activity. The linked service works well. When I run the pipeline, the databricks job fails - the ADF component fails, and I see a failed job run in databricks too.

The problem seems to be the parameters. When I look in the ADF log in the input from the databricks job task, I see that the parameters have been passed correctly.

When I look at the databricks job run, the parameter fields are empty. This is also evident by the error when it failed - where the parameters are supposed to be used, the value was empty.

Below are the adf task settings and the databricks run details from my debug run.

setup adf

task run details

J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-15T08:23:36.8466667+00:00

Just checking in to see if the below answer provided by @Amira Bedhiafi helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Debie, Hanneke 35 Reputation points

2025-06-11T14:36:40.2066667+00:00

@SSingh-MSFT

Thank you very much for your reply! It contained the information I needed to pinpoint the problem: My selfhosted IR is of version 5.48.9106.2.

Currently, my work was not impeded by this issue I had, since I had implemented a different way to reach my goal. I hope that this interaction can help other users struggling with the issue in the future.

I consider the issue resolved on my end now.

My thanks to everyone who weighed in on this thread!

Accepted answer

1 additional answer

Your answer

J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-15T08:23:36.8466667+00:00

Just checking in to see if the below answer provided by @Amira Bedhiafi helped.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Debie, Hanneke 35 Reputation points

2025-06-11T14:36:40.2066667+00:00

@SSingh-MSFT

Thank you very much for your reply! It contained the information I needed to pinpoint the problem: My selfhosted IR is of version 5.48.9106.2.

Currently, my work was not impeded by this issue I had, since I had implemented a different way to reach my goal. I hope that this interaction can help other users struggling with the issue in the future.

I consider the issue resolved on my end now.

My thanks to everyone who weighed in on this thread!

Answer 1

Hi @Debie, Hanneke

We are sorry about the issue you are facing in parameter fetch.

When passing parameters between jobs & pipelines, we can pass parameters to jobs using jobParameters property in Databricks activity.

Note:

Job parameters are only supported in Self-hosted IR version 5.52.0.0 or greater.

As per the official document as of today https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-job, the Azure Databricks Jobs activity is currently in preview. This information relates to a pre-release product that may be substantially modified before it's released. Microsoft makes no warranties, expressed or implied, with respect to the information provided here.

User's image

If you are using "Databricks Notebook Activity", you can pass parameters to notebooks using baseParameters property in databricks activity.

In certain cases, you might require to pass back certain values from notebook back to the service, which can be used for control flow (conditional checks) in the service or be consumed by downstream activities (size limit is 2 MB).

In your notebook, you can call dbutils.notebook.exit("returnValue") and corresponding "returnValue" will be returned to the service.
You can consume the output in the service by using expression such as @{activity('databricks notebook activity name').output.runOutput}.

https://learn.microsoft.com/en-us/azure/data-factory/transform-data-databricks-notebook

Please let us know if you have queries.

Thank you!

SSingh-MSFT 16,371 Reputation points Moderator

2025-06-13T04:39:55.1+00:00

Hi Debie, Hanneke •

Thank you so much for replying back and using Azure.

If this answers your query, do click Accept Answer and Mark Helpful for the same. And, if you have any further query do let us know.

Thanks!

Answer 2

Amira Bedhiafi 32,756 Volunteer Moderator

Hello Debbie !

Thank you for posting on Microsoft Learn.

The parameter names in ADF exactly match the Databricks job parameter names (please be aware that they are case-sensitive).

Try to check the case of both pipeline().parameters.* and the left-hand name field in ADF job parameters matches exactly including capitalization.

If you are running debug mode in ADF and did not set pipeline parameters in the debug pane, then:

@pipeline().parameters.Entity

If your notebook expects JSON or structured input, and you're using dynamic content from ADF, you must serialize it properly.

Your Databricks notebook should use the parameters like this:

dbutils.widgets.get("EntityName")

If the widget isn't defined in the notebook with:

dbutils.widgets.text("EntityName", "")

Debie, Hanneke 35

Hello Amira!

I checked if the parameters had been passed properly, and they did.

The activity details show this as input:

 {
    "jobId": "46909467545214",
    "jobParameters": {
        "EntityName": "scenariodefinitionliquidityevergreen",
        "EnvName": "ont",
        "MasterRunId": "03bfa19e-3b2f-4191-a98e-36e1c508e05c",
        "SourceName": "eucrisk"
    }
}

However, when I look in databricks under workflows -> job runs -> my job, the run happened, but the taskrun details in the side panel show that the parameters are empty.

task run details2

The first cell contains this:

dbutils.widgets.text("EnvName")
dbutils.widgets.text("WheelFileName", "platform_tools-0.1.0-py3-none-any.whl")
dbutils.widgets.text("BasePath", "/Volumes/platform_ont/bronze_landing/bronze_landing/")
dbutils.widgets.text("EntityName")
dbutils.widgets.text("MasterRunId")
dbutils.widgets.text("SourceName")
dbutils.widgets.dropdown("Debug", "False", ["True", "False"])

And a later cell has this:

wheel_file_name = dbutils.widgets.get("WheelFileName")
source_name = dbutils.widgets.get("SourceName")
entity_name = dbutils.widgets.get("EntityName")
base_path = dbutils.widgets.get("BasePath")
master_run_id = dbutils.widgets.get("MasterRunId")
debug = dbutils.widgets.get("Debug")
env_name = dbutils.widgets.get("EnvName")

J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-16T07:19:32.58+00:00
@Debie, Hanneke
Mismatch or misalignment in parameter names and widget declarations between Azure Data Factory and the Databricks notebook job is the most probable cause.

ADF passes parameters such as EntityName, SourceName, MasterRunId, and EnvName.

In the Databricks notebook, these must be declared as widgets with exact same names and casing before any code that uses them.

If widgets are not declared before execution starts, or their names/case don't match, the values won’t be assigned, and dbutils.widgets.get() will return empty strings.

The parameter name must match the name declared in the notebook using dbutils.widgets. The name is case-sensitive.

Ensure the parameter names in ADF match the widget names in Databricks exactly.

You must define widgets at the beginning of the notebook, before referencing them.

From Microsoft doc - expressions, ADF lets you reference pipeline parameters with:

@pipeline().parameters.EntityName

But Your using

@pipeline().parameters.Entity @pipeline().parameters.source

Use the correct names, exactly matching what's in the notebook:

@pipeline().parameters.EntityName @pipeline().parameters.SourceName

I hope this info helps,if you have any further query do let us know.
Debie, Hanneke 35 Reputation points

2025-05-16T14:06:55.7833333+00:00

I'm kind of confused.

In the job I add the parameters as key-value pairs. The key are spelled correctly and have the right capitalization. The ADF pipeline parameters are named differently, but that shouldn't matter for the outcome, right?

As a sidenote: I used a webactivity to start the job, and that one passes the parameters correctly.
J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-16T16:33:38.8133333+00:00

@Debie, Hanneke
The issue seems to be not with ADF parameter mapping, but with how Databricks handles parameters at runtime.
Please check the below options and confirm:
Define the widget at the beginning of the notebook using dbutils.widgets.text(...). If the widget is not defined, the parameter will not be received by the notebook.

If a parameter is missing or appears as empty, verify that the widget is created before trying to access its value.

https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-databricks-notebook#pass-parameters-to-notebook

I hope this info helps,if you have any further query do let us know. I am happy to assist further.
Viralkumar Patel 0 Reputation points

2025-05-19T23:03:25.4533333+00:00

Hello @Debie, Hanneke

I had similar issue. In my case all parameter value was correct. I have fixed the issue by switching databricks linked service from self-hosted IR to Microsoft Managed IR. Please check your linked service IR.

Thanks

Debie, Hanneke 35

@J N S S Kasyap

The widgets are defined in the first cell of the notebook. Not all of them require external input, but some do. It looks like this:

dbutils.widgets.text("EnvName", "local")
dbutils.widgets.text("WheelFileName", "platform_tools-0.1.0-py3-none-any.whl")
dbutils.widgets.text("BasePath", "/Volumes/bdp_ont/bronze_landing/bronze_landing/")
dbutils.widgets.text("EntityName", "instrumenttransaction")
dbutils.widgets.text("MasterRunId", "000000-0000-0000-0000")
dbutils.widgets.text("RunId", "1")
dbutils.widgets.text("SourceName", "prime")
dbutils.widgets.dropdown("Debug", "False", ["True", "False"])

In a later cell they are retrieved:

wheel_file_name = dbutils.widgets.get("WheelFileName")
source_name = dbutils.widgets.get("SourceName")
entity_name = dbutils.widgets.get("EntityName")
base_path = dbutils.widgets.get("BasePath")
master_run_id = dbutils.widgets.get("RunId")
debug = dbutils.widgets.get("Debug")
env_name = dbutils.widgets.get("EnvName")

Now I'm quite certain that this is not the issue, because the API call to the job passed the parameters just fine.

To be sure, this is the json of the pipeline with only the jobs activity:

{
    "name": "13_test",
    "properties": {
        "activities": [
            {
                "name": "Job1",
                "type": "DatabricksJob",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "jobId": "812604096617191",
                    "jobParameters": {
                        "EntityName": {
                            "value": "@pipeline().parameters.Entity",
                            "type": "Expression"
                        },
                        "EnvName": "ont",
                        "MasterRunId": {
                            "value": "@pipeline().RunId",
                            "type": "Expression"
                        },
                        "SourceName": {
                            "value": "@pipeline().parameters.source",
                            "type": "Expression"
                        }
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDatabricks1",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "parameters": {
            "source": {
                "type": "string",
                "defaultValue": "eucrisk"
            },
            "Entity": {
                "type": "string",
                "defaultValue": "targetlimit"
            },
            "DestinationFolderDB": {
                "type": "string",
                "defaultValue": "bronze_landing"
            }
        },
        "folder": {
            "name": "10_extract"
        },
        "annotations": []
    }
}

The IR is hosted on azure. I use the same IR for the API call. Here is an image with the settings. I removed the workspace url and workspace resource id.

User's image

Here is the code for the API call that does pass the parameters.

{
    "name": "Trigger Databricks Job run",
    "description": "fire & forget",
    "type": "WebActivity",
    "state": "Active",
    "onInactiveMarkAs": "Succeeded",
    "dependsOn": [],
    "policy": {
        "timeout": "0.12:00:00",
        "retry": 3,
        "retryIntervalInSeconds": 30,
        "secureOutput": false,
        "secureInput": false
    },
    "userProperties": [],
    "typeProperties": {
        "method": "POST",
        "url": {
            "value": "@concat(\n    pipeline().globalParameters.DatabricksWorkspaceURL,\n    '/api/2.1/jobs/run-now'\n)",
            "type": "Expression"
        },
        "connectVia": {
            "referenceName": "bng-cloud-ir",
            "type": "IntegrationRuntimeReference"
        },
        "body": {
            "value": "{\n  \"job_id\": \"@{pipeline().globalParameters.DatabricksJobId}\",\n  \"job_parameters\": {\n    \"EntityName\": \"@{pipeline().parameters.Entity}\",\n    \"EnvName\": \"@{pipeline().globalParameters.Tier}\",\n    \"MasterRunId\": \"@{pipeline().parameters.MasterRunId}\",\n    \"SourceName\": \"@{pipeline().parameters.source}\"\n  }\n}",
            "type": "Expression"
        },
        "authentication": {
            "type": "MSI",
            "resource": "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
        }
    }
}

J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-26T00:32:24.23+00:00
@Debie, Hanneke
Apologies for the delayed response I appreciate your patience. After reviewing everything, it looks like the issue comes down to how the parameters are being passed in the DatabricksJob activity. Specifically, ADF expects the parameters under baseParameters, not jobParameters.

You are absolutely right that the widgets are properly defined in the notebook, and you confirmed that parameters do get passed correctly when using the Web activity (with a run-now API call). However, the same parameters do not work when using the ADF DatabricksJob activity.

Parameters are not being passed from Azure Data Factory's DatabricksJob activity to the Databricks notebook, even though they are correctly defined and passed in the pipeline.

In the ADF pipeline, the user incorrectly uses the key "jobParameters" inside the typeProperties of the DatabricksJob activity.

However, ADF expects the key to be baseParameters in order to pass parameters into the Databricks job. The key "jobParameters" is ignored by ADF and does not get forwarded to the Databricks job run.

Modify your activity JSON from:

"typeProperties": { "jobId": "812604096617191", "jobParameters": { "EntityName": { "value": "@pipeline().parameters.Entity", "type": "Expression" },

To:

"typeProperties": { "jobId": "812604096617191", "baseParameters": { "EntityName": { "value": "@pipeline().parameters.Entity", "type": "Expression" },

I hope this info helps,if you have any further query do let us know. I am happy to assist further.
Valentin GAILLEZ 5 Reputation points

2025-05-27T21:29:33.0666667+00:00

I'm having exactly the same problem trying to test the new azure data factory activity, which lets you run a databricks job instead of a notebook. Impossible to pass parameters each time they are empty on the databricks side whereas adf seems to retrieve the right values. Modifying the json of the adf activity by replacing job parameters by based parameters has changed nothing, still empty.
J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator

2025-05-28T08:11:04.0966667+00:00

@Valentin GAILLEZ, Please create a new thread with detailed information, We are happy help you on your task.
Debie, Hanneke 35 Reputation points

2025-05-30T15:02:13.7033333+00:00

@J N S S Kasyap

I tried your suggestion. I edited the json, saved, and ran a debug run. However, the result is still the same. No parameters were received by the databricks job.
Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-03T10:42:06.16+00:00
Hello @Debie, Hanneke,

Can you try following the below steps:

In the activity settings:

Select your Databricks linked service.

Provide the notebook path.

Add base parameters (these will be passed to the notebook): { "EntityName": "scenariodefinitionliquidityevergreen", "EnvName": "ont", "MasterRunId": "03bfa19e-3b2f-4191-a98e-36e1c508e05c", "SourceName": "eucrisk" }

Note: The parameter names MUST match exactly in your notebook using dbutils.widgets.text(...)

In your Databricks notebook:

dbutils.widgets.text("EntityName", "") dbutils.widgets.text("EnvName", "") dbutils.widgets.text("MasterRunId", "") dbutils.widgets.text("SourceName", "") entity = dbutils.widgets.get("EntityName") env = dbutils.widgets.get("EnvName") runid = dbutils.widgets.get("MasterRunId") source = dbutils.widgets.get("SourceName") print(f"Entity: {entity}") print(f"Env: {env}") print(f"Run ID: {runid}") print(f"Source: {source}") import json result = { "EntityName": entity, "EnvName": env, "MasterRunId": runid, "SourceName": source } dbutils.notebook.exit(json.dumps(result))

Debie, Hanneke 35

Good day,

I made the changes as asked. However, the parameters still didn't get passed.

The json of part of my ADF pipeline:

{
    "name": "13_test",
    "properties": {
        "activities": [
            {
                "name": "Job1",
                "type": "DatabricksJob",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "jobId": "46909467545214",
                    "baseParameters": {
                        "EntityName": "scenariodefinitionliquidityevergreen",
                        "EnvName": "ont",
                        "MasterRunId": "03bfa19e-3b2f-4191-a98e-36e1c508e05c",
                        "SourceName": "eucrisk"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDatabricks1",
                    "type": "LinkedServiceReference"
                }
            }

And the notebook plus output

User's image

In the ADF run activity I can see this as the input from the databricks job activity

"jobId": "46909467545214",

"baseParameters": {

    "EntityName": "scenariodefinitionliquidityevergreen",

    "EnvName": "ont",

    "MasterRunId": "03bfa19e-3b2f-4191-a98e-36e1c508e05c",

    "SourceName": "eucrisk"

}
```On a sidenote, I also tried to run a notebook activity from ADF, and that one worked correctly. 

If it would help, I'm open to do a teams call some time.

Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-04T06:24:38.76+00:00
Hello Debie, Hanneke

Can you check the below if it helps you?

The issue appears to be that the expected parameter values are not being used during execution, which suggests the Databricks job task isn't referencing the parameters correctly.

To ensure that parameters passed from ADF are properly utilized in Databricks:

In Azure Data Factory (ADF):

Use the jobParameters property in your Databricks job activity to pass values like:

`{ "entity_name": "scenariodefinitionliquidityevergreen" }`

In Databricks:

Go to the job configuration, then edit the task within the job.

Under Base Parameters, make sure the value is set using:

`entity_name = {{job.parameters.entity_name}}` This tells the task to dynamically reference the job-level parameter passed at runtime, rather than using a hardcoded or default value.

Refer to the Databricks job parameters documentation for more details.
Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-05T11:36:26.1233333+00:00

Hello @Debie, Hanneke,We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Debie, Hanneke 35 Reputation points

2025-06-05T14:30:44.8333333+00:00

Hello @Dileep Raj Narayan Thumula This didn't work. I made sure the parameter matched everywhere, so in adf, job level en in referencing it in the task it was entity_name.

However, I got the following error:

However, in all those runs, the job parameters were empty as well. So it wasn't a case of job parameters not being pushed down to the task.
Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-06T10:53:29.6966667+00:00

@Debie, Hanneke, can you tr this set up Job Parameters at the Job Level
Go to the Jobs tab
Open your Job and click Edit.
In the “Parameters” section at the job level, add:

Name: entity_name

Default Value: (can be empty or a default like test)

Inside the task (click into the task in the job editor):

Under Base Parameters, enter:

entity_name = {{job.parameters.entity_name}}
Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-09T03:48:51.47+00:00

Hi @Debie, Hanneke We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dileep Raj Narayan Thumula 255 Reputation points Microsoft External Staff Moderator

2025-06-10T08:29:37.08+00:00

Hi @Debie, Hanneke We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Debie, Hanneke 35 Reputation points

2025-06-10T14:00:26.4366667+00:00

@Dileep Raj Narayan Thumula

The last reply you gave are basically the same instruction as the reply from before. The last time I ensured all the parameters matched in name, capitalization and so on.

Share via

The ADF Databricks job Activity doesn't seem to pass parameters

1 additional answer

Your answer