Problem deploying Defender for Containers to Arc-enabled OpenShift Kubernetes cluster

Question

Problem deploying Defender for Containers to Arc-enabled OpenShift Kubernetes cluster

Andrew Mackintosh 0

After onboarding our on-prem OpenShift cluster to Azure Arc, I am having problems with deploying the Defender for Containers extension into the cluster.

I have successfully onboarded our non-production environment, but when attempting to recreate the deployment for the production cluster, I have the publisher pods stuck in a CrashLoopBackoffState.

The logs from the pods indicate the following error:

time="2025-04-09T01:08:38Z" level=info msg="Successfully registered a new certificate" componentName=Publisher
time="2025-04-09T01:08:38Z" level=warning msg="Failed to write certificate to file, error: open /var/microsoft/microsoft-defender-for-cloud/oms.crt: permission denied" componentName=Publisher
panic: Error encountered during client initialization open /var/microsoft/microsoft-defender-for-cloud/oms.crt: permission denied

I have checked on the host (where the volume mount is) and the directory /var/microsoft/microsoft-defender-for-cloud is owned by root and has no contents. The pod is deployed as a privileged container, so should be able to write as root to this path.

Does anybody have any suggestions as to the cause of the issue?

Andrew Mackintosh 0 Reputation points

2025-04-11T01:16:26.96+00:00
I managed to find a workaround to this issue, but it doesn't address the underlying problem.

In the DaemonSet for the microsoft-defender-publisher under the mdc namespace, I edited the resource YAML to change the cert-onboarding volume from a hostPath to an emptyDir and the pods were able to start successfully.

The changed values are below.

kind: DaemonSet apiVersion: apps/v1 metadata: namespace: mdc name: microsoft-defender-publisher spec: template: spec: volumes: - name: cert-onboarding emptyDir: {}

1 answer

Your answer

Andrew Mackintosh 0 Reputation points

2025-04-11T01:16:26.96+00:00

I managed to find a workaround to this issue, but it doesn't address the underlying problem.

In the DaemonSet for the microsoft-defender-publisher under the mdc namespace, I edited the resource YAML to change the cert-onboarding volume from a hostPath to an emptyDir and the pods were able to start successfully.

The changed values are below.

kind: DaemonSet apiVersion: apps/v1 metadata: namespace: mdc name: microsoft-defender-publisher spec: template: spec: volumes: - name: cert-onboarding emptyDir: {}

Answer 1

Alex Burlachenko 4,550

Hi Andrew,

Thank you for reaching out and sharing your question on the Q&A portal. I appreciate the detailed explanation of the issue you're facing with deploying Defender for Containers on your Arc-enabled OpenShift cluster.

From the error message you provided, it seems the Publisher pod is unable to write the certificate file due to a permission issue, even though the pod is running in privileged mode. This could be related to the directory permissions or a security context constraint (SCC) in OpenShift that might be restricting the pod’s access despite its privileged status.

Here are a few of my things you could check:

Ensure the directory /var/microsoft/microsoft-defender-for-cloud exists on the host and has the correct permissions (e.g., chmod 777 temporarily for testing).

Verify if OpenShift’s SCC (Security Context Constraints) for the pod’s service account allows the necessary permissions. You might need to adjust the SCC or assign a more permissive one.

Check if there are any admission controllers or policies in your production cluster that might be enforcing stricter rules compared to your non-production environment.

For further guidance, you might find these Microsoft articles helpful:

Deploy Defender for Containers on Kubernetes clusters

Troubleshooting Azure Arc-enabled Kubernetes

Let me know if you need further assistance or if you’ve already tried these steps. I’d be happy to help you dig deeper into the issue.

Best regards,
Alex

P.S. If my answer helps you, please Accept my answer.

Andrew Mackintosh 0 Reputation points

2025-04-11T01:21:53.6866667+00:00

Hi Alex,

Thanks for getting in touch. I had tried changing the permissions on the host directory (which I did confirm existed) but it didn't seem to make any difference to the permission error. I also created the oms.crt file that the container was attempting to write (just as a blank file) and it still was unable to write to that directory.

The SCC that the pod was running under was created by the Azure Arc installer and looks fit for purpose. We don't have any other admission controllers or policies beyond those that come out-of-the-box with OpenShift.

The only documentation I have been able to find is that it's not suggested to use hostPath mounts for OpenShift as the underlying Red Hat CoreOS has a read-only root filesystem, and it's recommended that only Machine Config writes to the /var/ directory.

As I mentioned in my answer below, changing this to an emptyDir volume type appears to have resolved the issue. I just wonder if this will be a problem for other Arc-enabled OpenShift clusters.
Naveena Patlolla 1,900 Reputation points Microsoft External Staff

2025-04-15T11:17:11.57+00:00
Hi Andrew Mackintosh
Please try these below steps:

1.Check whether the directory /var/microsoft/microsoft-defender-for-cloud/ and the file oms.crt have proper permissions using below commands

#Check permissions on the directory

ls -ld /var/microsoft/microsoft-defender-for-cloud/

# Check permissions on the file

ls -l /var/microsoft/microsoft-defender-for-cloud/oms.crt

Verify whether the user account running the Microsoft Defender for Cloud service has appropriate permissions using below command

ps aux | grep microsoft

Restart the MDC service to apply changes using below command

sudo systemctl status mdatp sudo systemctl restart mdatp

If the comment was helpful, please click "Upvote"
Naveena Patlolla 1,900 Reputation points Microsoft External Staff

2025-04-16T09:47:59.26+00:00

Hi Andrew Mackintosh

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"
Andrew Mackintosh 0 Reputation points

2025-04-16T23:15:26.1466667+00:00

Hi @Naveena Patlolla . The directory /var/microsoft/microsoft-defender-for-cloud/ does exist on the host, is owned by root and has the mask 755.

Unfortunately, this is a Red Hat CoreOS host as part of an OpenShift cluster so the Defender for Cloud deployment is controlled via a helm chart rather than systemctl.

Making the changes to the DaemonSet manifest I detailed in my answer seems to have resolved the issue enough to get the pods running. I do notice that it has reverted since then, but continues to work.
Madugula Jahnavi 410 Reputation points Microsoft External Staff

2025-04-23T16:14:58.51+00:00

Hello Andrew Mackintosh, Are you still facing the issue! And could you confirm that if you have verified fsGroup or securityContext in the Pod spec and nothing is missing!
Madugula Jahnavi 410 Reputation points Microsoft External Staff

2025-04-24T14:56:59.33+00:00

Andrew Mackintosh, We haven't heard any response from you. Could you provide the response to look into the issue further!

Share via

Problem deploying Defender for Containers to Arc-enabled OpenShift Kubernetes cluster

1 answer

Your answer