Tasks fail to detect GPU on some Pool nodes due to early startup race condition

Question

Tasks fail to detect GPU on some Pool nodes due to early startup race condition

JeffreyCMI 6

We’re running container tasks on a GPU-enabled Azure Batch pool (Docker-based, Standard_NC4as_T4_v3). All nodes are configured identically, and tasks are configured identically, running the same task in the same Docker image and entrypoint.

Despite this, the entire first round of tasks to run on certain pool nodes intermittently fail with:

failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

Investigation shows:

Only ~10% of new nodes are affected. The first round of Tasks on other (equivalent) nodes do find the GPU, logging Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3072 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0001:00:00.0, compute capability: 7.5
Affected tasks are the first tasks to run on the VM after provisioning. Later, Batch Tasks on the same node succeed and detect the GPU correctly.
nvidia-smi works on the host: GPU appears idle, lists no running processes using the GPU. Whereas when a Task has found the GPU, nvidia-smi shows GPU usage and lists running processes.
No node configuration drift: By the time I can SSH into the VM (some minutes after task start), docker info, nvidia-container-cli info, /etc/docker/daemon.json, and container runtime versions are consistent across nodes (good and bad).
No task configuration drift: same containerRunOptions (I am not using --gpus all), image name and tag, entrypoint.

Seems to me there's some race condition at node startup, where the GPU driver or NVIDIA container runtime stack is not fully initialized when the first task is scheduled. This feels very much within Azure's responsibility and out of my control.

We've encountered the bug intermittently, starting when we upgraded our Batch Pools image from

    publisher = "microsoft-azure-batch"
    offer     = "ubuntu-server-container-rdma"
    sku       = "20-04-lts"
    version   = "latest"

to

    publisher = "microsoft-dsvm"
    offer     = "ubuntu-hpc"
    sku       = "2204"
    version   = "latest"

JeffreyCMI 6 Reputation points

2025-06-18T20:39:20.5233333+00:00
I think this is a bug in Azure Batch.

Preferred resolutions (in order from most to least)

Azure Batch should not schedule GPU workloads until the node is fully GPU-ready. A platform fix that requires no action from me.

Or a reliable workaround that doesn’t involve, say, manually blocking in start tasks

Or a documented mechanism to delay task scheduling until GPU readiness
Anusree Nashetty 4,285 Reputation points Microsoft External Staff Moderator

2025-06-18T22:41:59.8633333+00:00
Hi JeffreyCMI,

The NVIDIA GPU driver stack or the NVIDIA container runtime is not always fully initialized by the time Azure Batch schedules the first task on a freshly provisioned node. This causes the first task to fail to detect the GPU, even though subsequent tasks on the same node succeed once the system is fully initialized.

Some containers do not automatically include GPU support. The NVIDIA Container Toolkit must be installed to enable GPU access inside containers. Ensure the Batch nodes have the NVIDIA Container Toolkit installed.

Run the following commands to ensure drivers are installed and GPUs are recognized

nvidia-smi lsmod | grep nvidia

If nvidia-smi fails, reinstall the drivers.
JeffreyCMI 6 Reputation points

2025-06-19T01:17:09.8433333+00:00
@Anusree Nashetty Thanks for confirming the race condition is known: that Azure Batch may schedule tasks before the NVIDIA GPU stack is fully initialized. This behavior is problematic for GPU workloads. Could you clarify:

Is this known issue on the Azure Batch roadmap for mitigation?

Is there a supported mechanism to delay task scheduling until the GPU runtime is fully ready?

Is there any internal health check that Azure uses to decide when a node is “Ready,” and can that be extended to GPU readiness?

To reiterate: this is a timing issue, not a driver or container config issue. I confirmed that after the first few minutes, drivers are installed, nvidia-smi works, etc.

Your answer

JeffreyCMI 6 Reputation points

2025-06-18T20:39:20.5233333+00:00

I think this is a bug in Azure Batch.

Preferred resolutions (in order from most to least)

Azure Batch should not schedule GPU workloads until the node is fully GPU-ready. A platform fix that requires no action from me.

Or a reliable workaround that doesn’t involve, say, manually blocking in start tasks

Or a documented mechanism to delay task scheduling until GPU readiness
Anusree Nashetty 4,285 Reputation points Microsoft External Staff Moderator

2025-06-18T22:41:59.8633333+00:00

Hi JeffreyCMI,

The NVIDIA GPU driver stack or the NVIDIA container runtime is not always fully initialized by the time Azure Batch schedules the first task on a freshly provisioned node. This causes the first task to fail to detect the GPU, even though subsequent tasks on the same node succeed once the system is fully initialized.

Some containers do not automatically include GPU support. The NVIDIA Container Toolkit must be installed to enable GPU access inside containers. Ensure the Batch nodes have the NVIDIA Container Toolkit installed.

Run the following commands to ensure drivers are installed and GPUs are recognized

nvidia-smi lsmod | grep nvidia

If nvidia-smi fails, reinstall the drivers.
JeffreyCMI 6 Reputation points

2025-06-19T01:17:09.8433333+00:00

@Anusree Nashetty Thanks for confirming the race condition is known: that Azure Batch may schedule tasks before the NVIDIA GPU stack is fully initialized. This behavior is problematic for GPU workloads. Could you clarify:

Is this known issue on the Azure Batch roadmap for mitigation?

Is there a supported mechanism to delay task scheduling until the GPU runtime is fully ready?

Is there any internal health check that Azure uses to decide when a node is “Ready,” and can that be extended to GPU readiness?

To reiterate: this is a timing issue, not a driver or container config issue. I confirmed that after the first few minutes, drivers are installed, nvidia-smi works, etc.

Share via

Tasks fail to detect GPU on some Pool nodes due to early startup race condition

Your answer