Intermittent Mount configuration error

Question

Intermittent Mount configuration error

I'm getting this error in my new Azure batch workloads. This error does not appear every time though. It is like 25% of the time my pool creates a new node this error happens. The only solution I have now is to re-image the node but that is not what I want to be doing.

NOTES:

I'm using a custom image from Ubuntu 24.04
The mount configuration is using blobfuse2 to mount a blob container.

Thanks,

Accepted answer

1 additional answer

Your answer

Answer 1

Hi Arthur Francisco Araujo Fernandes,

Good catch! Glad the issue is resolved for you finally. I will have this answer promoted by reposting it. As an Original Poster will not be able to accept your own answer. This is in the attempt to help others looking for a solution for a similar issue.

Use Ubuntu server 22.04 instead of Ubuntu serve 24.04, mounting a blob storage container which you need to have constant connection for a long job (hours) and have big data files (tenths to hundreds of GB each file).

User's image

Thanks again for sharing the solution here. Have a good day!

Answer 2

Markapuram Sudheer Reddy 1,670 Microsoft External Staff

Hi Arthur Francisco Araujo Fernandes,

Thank You for reaching out to Microsoft Q&A forum.

The error you are facing like out of memory can be due to high memory usage by running applications or insufficient memory allocation.

Ensure that the image you configured does not have any unnecessary services or applications running that could consume memory.

Ensure that the VM size you are using in your Azure Batch pool has sufficient memory and CPU resources.

Use the commands like top, htop, or free -m to check memory usage.

If you have any queries, please do let us know, we will help you.

Arthur Francisco Araujo Fernandes 20 Reputation points

2025-02-19T23:17:54.8066667+00:00
Thanks for the comment @Markapuram Sudheer Reddy . I will recreate my custom image and see if that solves the issue. But that issue happens even BEFORE my task starts, so I don't get why the OOM error. Also, what bothers me more is that it is intermittent. I.E. I have a test job that ran 80% of the time yesterday. The fails happened, most of the time when imaging the nodes. The test job is not supposed to cause any OOM issues. When I ran it locally it uses at most 50G and the SKU has 160G of memory.

Another piece that bothers me is that the error in the portal does NOT match the bfuse.log. This is what was inside the bfuse.log

2025-02-18T23:39:36,664296465+00:00 - INFO: Setting account name, key and SAS URL for /mnt/batch/tasks/fsmounts/bfuseconnect/dmuconnection.yaml 2025-02-18T23:39:36,666878498+00:00 - INFO: Downloading: https://packages.microsoft.com/keys/microsoft.asc as microsoft.asc 2025-02-18T23:39:36,721643129+00:00 - INFO: Downloading: https://packages.microsoft.com/config/ubuntu/24.04/prod.list as /etc/apt/sources.list.d/microsoft-prod.list gpg: no valid OpenPGP data found. gpg: dearmoring failed: File exists
Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-02-20T21:51:15.8733333+00:00

Hi Arthur Francisco Araujo Fernandes,

As you stated above that, even before the task starts, sometimes you are facing OOM issue, as it can also be due to Job preparation tasks or start task which runs on each node consuming memory during node setup. https://learn.microsoft.com/en-us/azure/batch/batch-job-prep-release#job-preparation-task

Check quotas and limits of batch service. https://docs.azure.cn/en-us/batch/batch-quota-limit

https://learn.microsoft.com/en-us/azure/batch/batch-quota-limit#pool-size-limits

As you are looking between the discrepancy in the error logs as the Azure portal often displays summarized error information, provides us a general overview of the issue whereas bfuse.log shows detailed logs generated by the bfuse process. The bfuse.log focuses mainly on bfuse's operations, such as setting up file system mounts or downloading packages.

The GPG error logs which suggests above states that the system is unable to validate the downloaded package lists. Ensure that your custom image is having the necessary GPG keys, and that the system is configured to properly validate packages.

If possible, try to recreate with a custom image or if it is convenient to use Azure Marketplace image.

If you have any queries, please do let us know, we will help you.

Arthur Francisco Araujo Fernandes 20

I don't have any job preparation task, the only preparation task that runs is from MS that installs blobfuse2. I didn't install/configure blobfuse2 since MS explicitly says to not install any Azure Extension when creating a custom image. https://learn.microsoft.com/en-us/azure/batch/batch-sig-images

Diving into the uploaded logs this is the full log error:

■■20250221T164509.170Z■00000000-0000-0000-0000-000000000000■ERROR■agent.mount.filesystems.basefilesystem■basefilesystem.py■run_cmd_persist_output_async■71■16719■MainThread■123136443162752■Mount command failed with exit code: 2, output: 2025-02-21T16:45:08,992119172+00:00 - INFO: Setting account name, key and SAS URL for /mnt/batch/tasks/fsmounts/bfuseconnect/dmuconnection.yaml
2025-02-21T16:45:08,994454408+00:00 - INFO: Downloading: https://packages.microsoft.com/keys/microsoft.asc as microsoft.asc
2025-02-21T16:45:09,086694242+00:00 - INFO: Downloading: https://packages.microsoft.com/config/ubuntu/24.04/prod.list as /etc/apt/sources.list.d/microsoft-prod.list
gpg: no valid OpenPGP data found.
gpg: dearmoring failed: File exists

■■20250221T164509.170Z■00000000-0000-0000-0000-000000000000■ERROR■agent.task.prep■prep.py■_perform_mounts_async■326■16719■MainThread■123136443162752■Error while performing mount□Traceback (most recent call last):□  File "agent/task/prep.py", line 324, in _perform_mounts_async□  File "agent/mount/mountmanager.py", line 71, in mount_async□  File "agent/mount/filesystems/blobfuse.py", line 55, in mount_async□  File "agent/mount/filesystems/blobfuse.py", line 88, in _install_and_mount_blobfuse_async□common.exceptions.MountConfigurationException: System error (out of memory, cannot fork, no more loop devices)

The error is with MS code when mounting the blob storage. However, it does not make sense to me that it fails from time to time but still runs correctly for some nodes. Seems like the node fails to automatically set the account and create the connection.yaml file. The OOM error happens when the agent attempts to trace back, which is probably a poor nested function call leading to the OOM error. But I don't have any control over Azure functions!

Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-02-21T20:33:28.67+00:00

Hi Arthur Francisco Araujo Fernandes,

I just wanted to check with you regarding the review of above response. If you have any queries, please share here, we will help you. If you are facing errors, please share the logs, to investigate further.
Arthur Francisco Araujo Fernandes 20 Reputation points

2025-02-21T20:36:45.9633333+00:00

@Markapuram Sudheer Reddy see my answer above.
Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-02-24T17:16:06.89+00:00

Hi Arthur Francisco Araujo Fernandes,

Based on above logs as unable to mount with Azure Batch pool node, as a mount configuration fails, can you please try once manually on the compute node which is failing with mount process by following the steps provided in the documentation.

https://learn.microsoft.com/en-us/azure/batch/virtual-file-mount?tabs=linux#troubleshoot-mount-issues

https://learn.microsoft.com/en-us/rest/api/batchservice/compute-node/get?view=rest-batchservice-2024-07-01&tabs=HTTP#computenodeerror

Please check network settings for a compute node -https://learn.microsoft.com/en-us/azure/batch/batch-virtual-network

If you have any queries, please do let us know, we will help you.
Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-02-25T22:05:01.8866667+00:00

Hi Arthur Francisco Araujo Fernandes,

I just wanted to follow up regarding above response to your query.

If you have any questions, please do let us know, we will help you.
Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-03-02T10:34:23.6033333+00:00

Hi Arthur Francisco Araujo Fernandes,

Thanks for your patience.

We have noticed that you have rated an answer as not helpful. We appreciate your feedback and are committed to improving your experience with the Q&A.

Sometimes the error Out of memory occurs in batch node VM can also be due to out of Virtual Memory. To overcome try to check with this recommendation by managing Azure Batch pools using autoscaling strategy.

Ensure swap space is configured on the nodes.

We recommend checking below community discussion that might provide insights on Out of memory issue on batch node. https://learn.microsoft.com/en-us/answers/questions/1161508/how-to-solve-azure-batch-node-out-of-memory?orderby=newest%2Cnewest&orderBy=Oldest

If the information is helpful, please click on "Accept Answer" and "Upvote"

If you have any queries, please do let us know, we will help you.
Markapuram Sudheer Reddy 1,670 Reputation points Microsoft External Staff

2025-03-03T21:30:30.9933333+00:00

Hi Arthur Francisco Araujo Fernandes,

I just wanted to check with you regarding the review of above response. If you have any queries, please share here, we will help you.
Arthur Francisco Araujo Fernandes 20 Reputation points

2025-03-03T22:09:24.69+00:00

@Markapuram Sudheer Reddy the link you send is saying to use autoscale, which I'm already using, thus, it was not helpful. in this case. Thank you so much for attempting in assist on this.

Arthur Francisco Araujo Fernandes 20

TLDR: Don't use Ubuntu 24.04 if you are mounting a blob storage container which you need to have constant connection for a long job (hours) and have big data files (tenths to hundreds of GB each file). Use Ubuntu server 22.04 instead.

Ans: After many backs and forth with people inside MS support I got an interesting request to use Ubuntu server 22.04 instead of Ubuntu serve 24.04. It worked!!! I don't know what the catch is, but apparently, MS start script has some conflicts with Ubuntu server 24.04 that is making the node lose the lease of the storage container. When the connection to the storage container is lost the blubfuse.py scripts go into a nested call that leads to OOM. I will run some more test runs but it was stable in my initial 10 test runs.Below is sample of the winning configuration in my specific case. But note that I didn't create it from Azure CLI but rather modified from the portal omitting the configurations that are only relevant to me (i.e. account key and stuff like that).

{
    "id": "your-pool",
    "displayName": null,
    "vmSize": "STANDARD_E20ds_V4",
    "virtualMachineConfiguration": {
        "imageReference": {
            "publisher": null,
            "offer": null,
            "sku": null,
            "version": null,
            "virtualMachineImageId": "/subscriptions/yoursubs/Microsoft.Compute/galleries/batchcomputegallery/images/yourcustom-ubuntu-22_04-lts-server/versions/1.0.0",
            "exactVersion": null
        },
        "nodeAgentSKUId": "batch.node.ubuntu 22.04",
        "securityProfile": {
            "securityType": "TrustedLaunch",
            "encryptionAtHost": null,
            "uefiSettings": {
                "secureBootEnabled": true,
                "vTpmEnabled": true
            }
        }
    },
    "enableAutoScale": true,
    "autoScaleFormula": "startingNumberOfVMs = 0; maxNumberofVMs = 10; pendingTaskSamplePercent = $PendingTasks.GetSamplePercent(180 * TimeInterval_Second); pendingTaskSamples = pendingTaskSamplePercent < 70 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second)); $TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples); $NodeDeallocationOption = taskcompletion",
    "autoScaleEvaluationInterval": "PT5M",
    "enableInterNodeCommunication": false,
    "networkConfiguration": {
        "subnetId": "/subscriptions/yoursubs/Microsoft.Network/virtualNetworks/net-batch-001/subnets/default",
        "dynamicVNetAssignmentScope": null,
        "publicIPAddressConfiguration": {
            "provision": "batchmanaged"
        },
        "enableAcceleratedNetworking": true
    },
    "applicationPackageReferences": [
        {
            "applicationId": "testApp",
            "version": null
        }
    ],
    "taskSlotsPerNode": 1,
    "taskSchedulingPolicy": {
        "nodeFillType": "spread"
    },
    "targetNodeCommunicationMode": "simplified",
	"mountConfiguration": [
        {
       		 "azureBlobFileSystemConfiguration": {
           		 "accountName": "yourStorageAcc",
           		 "accountKey": "yourKey",
           		 "blobfuseOptions": "--allow-other --log-level=LOG_DEBUG --read-only=false --disable-writeback-cache=false --file-cache-timeout=15000 --sync-to-flush=true -o attr_timeout=15000 -o entry_timeout=15000 -o negative_timeout=15000",
            "containerName": "yourContainer",
            "relativeMountPath": "yourContainer"
            }
        }
    ]
}

Share via

Intermittent Mount configuration error

1 additional answer

Your answer