MMDetection pipeline component "Model Prediction" failed due to Docker error

Question

MMDetection pipeline component "Model Prediction" failed due to Docker error

jrud 15

Hi,

I am running the MMDetection pipeline mmdetection_image_objectdetection_instancesegmentation_pipeline invoked by a modified version of the mmdetection-fridgeobjects-object-detection.ipynb example. Until the model prediction step, the pipeline runs fine but the prediction component fails with the message "Failed to execute command group with error An unexpected error occurred while executing command due to: Failed to pull Docker image mcr.microsoft.com/azureml/curated/model-evaluation:45 due to: Some(DockerError("Docker stream error"))"

I've tried running an earlier version of the pipeline but then the same error occurs at an earlier stage.

Thanks!

Accepted answer

0 additional answers

Your answer

Answer 1

Azar 28,155 MVP

Hi there jrud

Thanks for using QandA platform

Guess the ML couldn’t pull the Docker image model-evaluation:45 from MCR.

Try Manually pulling it: docker pull mcr.microsoft.com/azureml/curated/model-evaluation:45 and Checking firewall or network restrictions. give it a restart if it doesnt work

if possible try with a different image version if possible.

If nothing helps i suggest you to raise a support req4

https://portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview

If this helps kindly accept the answer thanks much.

jrud 15 Reputation points

2025-05-07T06:32:19.1533333+00:00

Hi Azar, thanks for your help!

How can I pull the docker image manually, if it is part of the pipeline? Should I pull it on the VM? Same question regarding to the firewall and network restriction.

I'm only registering the pipeline from my local PC, the model training itself runs on a Azure Compute instance.

Thanks!
Saideep Anchuri 6,625 Reputation points Microsoft External Staff

2025-05-07T07:30:01.84+00:00

Hi jrud

To manually pull the Docker image as part of your pipeline, you would typically do this on the Azure Compute instance where your model training is running. Ensure that the path name to your container registry is correct and that you have the necessary credentials if you are pulling from a private registry.

Regarding firewall and network restrictions, you need to ensure that your Azure Compute instance has both inbound and outbound access to the public internet. If your container registry is behind a virtual network or using a private endpoint, you may need to configure it to allow public access or set up the necessary network rules to enable communication.

Kindly refer below link: docker-pull-issues

azureml

Thank You.
jrud 15 Reputation points

2025-05-07T10:27:59.89+00:00

Thanks for the quick answer!

I tried to pull the docker container on the GPU and found that it fails due to insufficient space on the device. I recreated the VM and then the pull succeeded.

After running the pipeline however, it seems that the docker pull also succeeded but now the space on the device runs out while pip installs package in the same component.

How can that be, considering I recreated the VM from scratch?

UPDATE:
I found out that the preinstalled conda environments take up a lot of space, so I removed them. Now the prediction component completed successfully!

Share via

MMDetection pipeline component "Model Prediction" failed due to Docker error

0 additional answers

Your answer