Limitations in ADO Service Connection Thumbprint field
Background:
We have some Self-Hosted agents based on GH-running-images. We need to use these as they are in-network to eliminate MSFT Hosted based network issues (i.e. accessing KV resources)/upkeep.
For Service Fabric Cluster deployments we had issues using Cert-based/common name. There was extensive back and forth with Microsoft. We eventually found that we had to use cert-based/thumbprints and include ALL the cluserCertificateThumbprints from each SF Cluster in each respective Service Fabric Service Connection.
This solved our issues temporarily; however, the number of thumbprints can fluctuate apparently. So when it went to 8 total, all the connections broke with "##[error]FABRIC_E_SERVER_AUTHENTICATION_FAILED: CertificateNotMatched"
The issue is the in the ADO UI for Service Fabric Service Connection thumbprint field is limited and can only take 7 thumbprints before it cuts off any other input. I found a variety of potential options recommended, but none are scalable as the Service Connection is shared out of one primary ADO Project and used by X number of other projects with a mixture of YAML and Classic pipelines.
The simplest solution is for the thumbprint max field input to be increased or allow an import of a file if the data set is over 7 thumbprints.
So, the question, is how can we get this request facilitated? Or is there another approach that is scalable for thumbprint-based usage?
Azure DevOps
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-14T09:40:13.9566667+00:00 Hello Chad Cummings,
The 7-thumbprint limit in the Azure DevOps Service Fabric Service Connection is causing your
FABRIC_E_SERVER_AUTHENTICATION_FAILED: CertificateNotMatched
error because your clusters need 8 thumbprints, but the UI caps at 7, messing up your shared connection across projects with YAML and Classic pipelines. You can fix this by using a script to update the Service Connection via the Azure DevOps REST API, which might bypass the UI limit. Since you’re using Key Vault, pull thumbprints from there. Below is PowerShell script to get you going:$thumbprints = "thumb1,thumb2,thumb3,thumb4,thumb5,thumb6,thumb7,thumb8" # Replace with Key Vault thumbprints $pat = "your_personal_access_token" # Your DevOps PAT $org = "your_organization" $project = "your_project" $endpointId = "your_service_connection_id" $body = @{ name = "SFConnection" type = "servicefabric" url = "https://your_cluster:19000" data = @{ ServerCertThumbprint = $thumbprints } } | ConvertTo-Json Invoke-RestMethod -Uri "https://dev.azure.com/$org/$project/_apis/serviceendpoint/endpoints/$endpointId?api-version=7.0" ` -Method Put -Headers @{Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat")))" } ` -ContentType "application/json" -Body $body
Update it with your details (like
your_personal_access_token
and thumbprints) and test in a safe environment first. To fetch thumbprints from Key Vault, try$cert = Get-AzKeyVaultCertificate -VaultName "your_vault" -Name "your_cert"; $thumbprint = $cert.Thumbprint
. Run the script after thumbprint changes, like renewals, to keep your pipelines working without manual updates. If you could split the Service Connection one per cluster, each under 7 thumbprints and adjust pipelines to use the right one, easier in YAML with variables.I hope this helps. Let me know if you need any further clarification, and I’ll be happy to assist you.
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-15T07:08:08.7733333+00:00 Hello Chad Cummings,
Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes, if you have any further query do let us know.
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-16T05:06:16.33+00:00 Hello Chad Cummings,
We haven’t heard from you on the last response and was just checking back to see could you please provide more details on your requirement so that I can assist you further.
-
Chad Cummings • 0 Reputation points
2025-04-21T15:32:47.9066667+00:00 Hi @Bodapati Harish - thank you for the input. Sorry for the delay, was away for the past week. I will review what you provided to see if it/or another approach incorporating the same concepts gets us past our current issue.
A side question, if you know, is there a reason the ADO UI would limit input for thumbprints to 7? It seems like those values could fluctuate and if values can be added with comma separators I'm not sure why it would be limited to so few?
Edited addition:
One thing we are trying is to create the Service Connection via the REST API to see if that allows us to get around the UI field limitations - https://dev.azure.com/{{organization}}/{{project}}/_apis/serviceendpoint/endpoints?api-version=7.1-preview. We have all the values the documentation says we need, but it keeps erroring with the below despite that being provided. The "endpoint" field is not in the documentation I see here https://learn.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints/create?view=azure-devops-rest-7.1&tabs=HTTP , but when that is removed we get the below error so I had to add it:
"message": "Value cannot be null.\r\nParameter name: endpoint",
Example of the body with values redacted:
{ "endpoint": { "data": {}, "name": "dev01-new-new-test", "type": "servicefabric", "url": "tcp://sfmc02-dev.eastus.cloudapp.azure.com:19000/index.html", "authorization": { "parameters": { "certificate": "encoded cert value", "certificatepassword": "cert password", "servercertthumbprints": [ list of thumbprints], "certLookup": "Thumbprint" }, "scheme": "Certificate" }, "isShared": false, "isReady": true, "serviceEndpointProjectReferences": [ { "projectReference": { "id": "project id value", "name": "project name" }, "name": "dev01-new-new-test"}] } }
Error:
{ "$id": "1", "innerException": null, "message": "At least one project reference required to create an endpoint.", "typeName": "System.ArgumentException, mscorlib", "typeKey": "ArgumentException", "errorCode": 0, "eventId": 0
-
James Smith 0 Reputation points
2025-04-21T15:47:15.2166667+00:00 Use the Azure DevOps REST API to update the connection with all required thumbprints dynamically. Recommend raising a support ticket to request lifting the UI limit or allowing file-based input.
-
Chad Cummings • 0 Reputation points
2025-04-21T17:27:23.99+00:00 Thanks James - I have been trying to get the update using the REST API via Postman to prove out that it will work, but can't seem to get past error for fields (endpoint, source, etc) that are not documented but erroring as required - https://learn.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints?view=azure-devops-rest-7.1 - and don't show up in the GET either. I'm not sure if it is a limitation with Service Fabric Service Connection updates or something else.
I did go ahead and open a ticket as well requesting if that field could be extended.
-
Chad Cummings • 0 Reputation points
2025-04-21T17:59:36.1733333+00:00 Looks like the null parameter errors (initially endpoint) were because it didn't like breaking the thumbprints out like "thumb1","thumb2", etc in the json ... it wants it as one string - "thumb1,thumb2" etc which does align with the way you enter in the UI. Unfortunately, even the REST call hits an issue with it exceeding 7 thumbprints :-(
{
"$id": "1", "innerException": null, **"message": "The String length exceeds maximum limit.\r\nParameter name: endpoint.Authorization.Parameters[servercertthumbprint]",** "typeName": "System.ArgumentException, mscorlib", "typeKey": "ArgumentException", "errorCode": 0, "eventId": 0 ```}
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-22T11:23:33.8433333+00:00 Hello Chad Cummings,
To work around the limit of 7 server certificate thumbprints per Azure DevOps Service Connection, you can create multiple Service Fabric Service Connections. Each connection will handle up to 7 thumbprints. The script below automates this process using the Azure DevOps REST API and Key Vault.
# Fetch thumbprints from Key Vault $vaultName = "your-vault-name" $certNames = @("cert1", "cert2", "cert3", "cert4", "cert5", "cert6", "cert7", "cert8") # Replace with your cert names $thumbprints = $certNames | ForEach-Object { (Get-AzKeyVaultCertificate -VaultName $vaultName -Name $_).Thumbprint } $thumbprintGroups = $thumbprints | Group-Object -Property { [math]::Floor([array]::IndexOf($thumbprints, $_) / 7) } | ForEach-Object { $_.Group -join "," } # Azure DevOps REST API details $pat = "your_personal_access_token" $org = "your_organization" $project = "your_project" $apiVersion = "7.1-preview.4" # Create Service Connections $index = 1 foreach ($thumbprintGroup in $thumbprintGroups) { $body = @{ name = "SFConnection-$index" type = "servicefabric" url = "tcp://your_cluster:19000" authorization = @{ parameters = @{ certlookup = "Thumbprint" servercertthumbprint = $thumbprintGroup certificate = "your_base64_encoded_client_cert" # Use: [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes("certificate.pfx")) certificatepassword = "your_cert_password" } scheme = "Certificate" } isShared = $true isReady = $true serviceEndpointProjectReferences = @( @{ projectReference = @{ id = "your_project_id" name = "your_project_name" } name = "SFConnection-$index" } ) } | ConvertTo-Json -Depth 10 $uri = "https://dev.azure.com/$org/$project/_apis/serviceendpoint/endpoints?api-version=$apiVersion" Invoke-RestMethod -Uri $uri -Method Post -Headers @{ Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat")))"; "Content-Type" = "application/json" } -Body $body $index++ }
- Replace placeholders such as
your-vault-name
,your_organization
,your_project
,your_project_id
, and certificate details with actual values. - Run the PowerShell script to create multiple Service Connections, each with up to 7 thumbprints.
- Use these connections in your Azure DevOps pipelines.
Below is YAML example
parameters: - name: clusterName type: string variables: serviceConnection: ${{ parameters.clusterName }}-SFConnection steps: - task: ServiceFabricDeploy@1 inputs: serviceConnectionName: $(serviceConnection)
- The script can be scheduled to run regularly or triggered by certificate rollover events to keep Service Connections updated.
- This method ensures each Service Connection respects the 7-thumbprint limit and automates the creation and maintenance of these connections please let me know If you need further assistance, please provide the comment.
- Replace placeholders such as
-
Chad Cummings • 0 Reputation points
2025-04-22T18:28:32.4866667+00:00 Hi Bodapati Harish - thanks for the continued support on possible options.
I am struggling to see how this approach solves the problem caused by the thumbprint limitation? As it is currently, if any SF Cluster has 8 (or more) thumbprints and the service connection is missing any of them the connection fails with a cert not matched error.
If multiple service connections are created for ones that have 7+ then both of them won't have the complete set needed to validation the connection as you can't use two service connections as a group for one deploy, can you?
A little more background on how ours are used
- We have nonProd SCs - sandbox, dev, test, uat
- We have Prod SCs
- We create all the SC's in a central ado Project and share to X number of projects that need them
- So, we have X number of projects with X number pipelines (YAML and Classic) with these connections coded or pre-selected (classic dropdown list)
So, a potential workaround would to:
- Pull the thumbprints dynamically from each cluster (X nonprod, X prod)
- Create or update the same a service connection of the same name in order to not break X number of pipelines using the connection(s)
Or given the limitation we are seeing in the Service Connection approach is there a mix in your approach(es) using key vault and some existing/combination of a ADO Task(s) to connect to a Service Fabric with a provided set of thumbprints in order to perform a deployment?
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-23T06:01:00.38+00:00 Hello Chad Cummings,
You're absolutely right about the core limitation: with the 7-thumbprint cap on the Service Fabric service connection, any cluster with 8 or more certs poses a problem, and there's no built-in support for grouping multiple service connections in a single deployment.
To work around this, we're proposing a dynamic update model, where a script or automation task periodically:
- Pulls the active thumbprints directly from each SF cluster (both non-prod and prod)
- Replaces the thumbprint list in the existing service connection (keeping the name and ID unchanged)
This way, we ensure that the service connection always reflects the current and valid set of certs without requiring pipeline changes or creating multiple service connections per cluster. Since you're using shared service connections across many pipelines and projects, keeping the SC identity intact is key to avoiding disruptions.
This approach does not technically increase the 7-thumbprint limit, but by keeping the SC updated with only the currently needed certs, we avoid mismatches and minimize the risk of failure. In clusters where more than 7 are active, you'd prioritize only those required for the specific deployment window (for example, certs that are active and not expired/soon-to-expire).
As an optional enhancement, we’re also evaluating use of Azure Key Vault for centralized cert tracking, or possibly leveraging CN-based authentication where suitable, to bypass the thumbprint limitation entirely.
Please let me know if you have any further question, I'll happy to assist you.
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-24T07:08:20.7366667+00:00 Hello Chad Cummings,
Following up to see if you have chance to check the previous response and help us with requested information to check and assist you further on this.
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-25T03:59:20.1733333+00:00 Hello Chad Cummings,
We haven’t heard from you on the last response and was just checking back to see could you please provide more details on your requirement so that I can assist you further.
-
Chad Cummings • 0 Reputation points
2025-04-25T19:25:32.97+00:00 Hi Bodapati - we are trying to pursue your suggestion on dynamic pulling of the thumbprints for the Service Fabric. While I did open a support to expand the UI field, it will still need automation if those thumbprints change periodically.
Right now, I am blocked internally with requirements to another team to create a custom role so my SP can talk to the Service Fabric cluster.
From the errors I am getting and researching around, it seems that the below actions are needed to pull the SF thumbprints. Reader, Contributor, etc don't have these abilities, nor do any others that I saw in our listings
"Actions": [
"Microsoft.ServiceFabric/clusters/read",
"Microsoft.ServiceFabric/clusters/applications/read",
"Microsoft.ServiceFabric/clusters/services/read"
]
Will advise once I get this role created.
I did manually test your point though about certs likely not all needed - some expired or soon to be - as I removed the first thumbprint in the list and included the next 7 and the service connection worked. I don't understand why Microsoft doesn't cleanup the ones they manage as there is no notice to our Cloud team when they add new thumbprints, but I'll ask that question on my open support ticket.
-
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-04-28T10:50:40.7133333+00:00 Hello Chad Cummings,
Thanks for the update! For the custom role, you’re on the right track with those permissions (
Microsoft.ServiceFabric/clusters/read
,Microsoft.ServiceFabric/clusters/applications/read
,Microsoft.ServiceFabric/clusters/services/read
). You might also needMicrosoft.ServiceFabric/clusters/list
to fully query cluster details. Here’s a quick role template to share with your team:{ "properties": { "roleName": "ServiceFabricThumbprintReader", "description": "Read Service Fabric cluster thumbprints", "assignableScopes": ["/subscriptions/your-subscription-id"], "permissions": [ { "actions": [ "Microsoft.ServiceFabric/clusters/read", "Microsoft.ServiceFabric/clusters/list", "Microsoft.ServiceFabric/clusters/applications/read", "Microsoft.ServiceFabric/clusters/services/read" ], "notActions": [], "dataActions": [], "notDataActions": [] } ] } }
Once the role is set, you can pull thumbprints from the cluster and update your Service Connections automatically. Here’s a short PowerShell script to do that, keeping the same Service Connection names so your pipelines don’t break:
$clusterEndpoint = "tcp://your-cluster:19000" $serviceConnectionName = "Sandbox-SC" $pat = "your_personal_access_token" $org = "your_organization" $project = "your_project" $clientCertPath = "path-to-client-cert.pfx" $certPassword = "your_cert_password" # Get thumbprints from cluster $cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($clientCertPath, $certPassword) $uri = "https://$($clusterEndpoint -replace 'tcp','https')/ClusterCertificates" $response = Invoke-RestMethod -Uri $uri -Method Get -Headers @{ "Authorization" = "Bearer $cert" } -Certificate $cert $thumbprints = $response.Certificates | Sort-Object NotAfter -Descending | Select-Object -First 7 -ExpandProperty Thumbprint | Join-String -Separator "," # Update Service Connection $uriGet = "https://dev.azure.com/$org/$project/_apis/serviceendpoint/endpoints?api-version=7.1-preview.4" $endpoints = Invoke-RestMethod -Uri $uriGet -Method Get -Headers @{ Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat")))"; "Content-Type" = "application/json" } $endpointId = ($endpoints.value | Where-Object { $_.name -eq $serviceConnectionName }).id $body = @{ name = $serviceConnectionName type = "servicefabric" url = $clusterEndpoint authorization = @{ parameters = @{ certlookup = "Thumbprint" servercertthumbprint = $thumbprints certificate = [System.Convert]::ToBase64String([System.IO.File]::ReadAllBytes($clientCertPath)) certificatepassword = $certPassword } scheme = "Certificate" } isShared = $true isReady = $true serviceEndpointProjectReferences = @( @{ projectReference = @{ id = "your_project_id" name = "your_project_name" } name = $serviceConnectionName } ) } | ConvertTo-Json -Depth 10 Invoke-RestMethod -Uri "https://dev.azure.com/$org/$project/_apis/serviceendpoint/endpoints/$endpointId?api-version=7.1-preview.4" -Method Put -Headers @{ Authorization = "Basic $([Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat")))"; "Content-Type" = "application/json" } -Body $body
This script grabs the 7 newest thumbprints from the cluster and updates the Service Connection without changing its name. Run it in a scheduled pipeline to keep things current.
If you have a cluster with 8+ active thumbprints that all need to be used, the Service Connection won’t work due to the 7-thumbprint limit. In that case, switch to the
ServiceFabricPowerShell@1
task to deploy directly, passing all thumbprints. Example for YAML:steps: - powershell: | $clusterEndpoint = "tcp://your-cluster:19000" $certPath = "path-to-client-cert.pfx" $certPassword = "your_cert_password" $cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($certPath, $certPassword) $uri = "https://$($clusterEndpoint -replace 'tcp','https')/ClusterCertificates" $response = Invoke-RestMethod -Uri $uri -Method Get -Headers @{ "Authorization" = "Bearer $cert" } -Certificate $cert $thumbprints = $response.Certificates | Select-Object -ExpandProperty Thumbprint | Join-String -Separator "," Write-Output "##vso[task.setvariable variable=Thumbprints]$thumbprints" displayName: 'Get Thumbprints' - task: ServiceFabricPowerShell@1 inputs: clusterConnectionParameters: | @{ ConnectionEndpoint = "tcp://your-cluster:19000" CertificateThumbprint = "$(Thumbprints)" ClientCertificate = @{ Thumbprint = "your_client_cert_thumbprint" StoreLocation = "LocalMachine" StoreName = "My" } } scriptType: 'Inline' inline: | Connect-ServiceFabricCluster @ClusterConnectionParameters Publish-NewServiceFabricApplication -ApplicationPackagePath '$(Build.ArtifactStagingDirectory)/pkg' -ApplicationName 'fabric:/MyApp' -OverwriteBehavior 'Always' overrideClusterConnectionParameters: true
For Classic pipelines, add a PowerShell task to set the thumbprints variable, then configure
ServiceFabricPowerShell@1
with the same parameters. -
Bodapati Harish • 400 Reputation points • Microsoft External Staff • Moderator
2025-05-02T08:09:20.5433333+00:00 Hello Chad Cummings,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, please respond with more details and we will try to help.
Sign in to comment