Connect to eventhub using Managed service identity (MSI) from Azure synapse analytics

Dheeraj Kumar Reddy Jogi 21 Reputation points Microsoft Employee
2024-07-24T00:29:21.13+00:00

I am trying to write the data to azure eventhubs from the azure synapse analytics notebook in scala using Managed service identity (MSI). Can you please help me with code for that.

Note that i was able to successfully connect using the SAS key.

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
711 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,318 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Amira Bedhiafi 31,416 Reputation points
    2024-07-26T20:37:58.49+00:00

    You need to add the Azure Event Hubs SDK and the Azure Identity SDK for Scala.

    Then use the EventHubsConf class to configure the connection to your Event Hub using MSI.

    Use the spark.writeStream method to write data to the Event Hub.

    
    // Import necessary libraries
    
    import com.microsoft.azure.eventhubs._
    
    import com.azure.identity._
    
    import org.apache.spark.eventhubs._
    
    import org.apache.spark.sql.functions._
    
    import org.apache.spark.sql.types._
    
    // Define Event Hub connection parameters
    
    val eventHubsNamespace = "<Your Event Hubs Namespace>"
    
    val eventHubName = "<Your Event Hub Name>"
    
    // Define the Managed Identity client
    
    val managedIdentityClientId = "<Your Managed Identity Client ID>" // Optional, if you want to specify a specific managed identity
    
    // Create a token credential using Managed Identity
    
    val credential = new DefaultAzureCredentialBuilder().build()
    
    // Build the EventHubsConf using Managed Identity
    
    val connectionString = new ConnectionStringBuilder(s"Endpoint=sb://${eventHubsNamespace}.servicebus.windows.net/;EntityPath=${eventHubName}")
    
    val eventHubsConf = EventHubsConf(connectionString.toString)
    
      .setTokenProvider(
    
        EventHubsUtils.buildTokenProvider(
    
          connectionString.getEndpoint.toString,
    
          connectionString.getEntityPath,
    
          credential
    
        )
    
      )
    
    // Sample data to write to Event Hubs
    
    val data = Seq(("Hello, Event Hubs!"))
    
    val df = data.toDF("message")
    
    // Write the data to Event Hubs
    
    val query = df
    
      .selectExpr("CAST(message AS STRING) AS body")
    
      .writeStream
    
      .format("eventhubs")
    
      .options(eventHubsConf.toMap)
    
      .option("checkpointLocation", "/mnt/checkpoints")
    
      .start()
    
    query.awaitTermination()
    
    
    

  2. Ravi Shankar Gupta 0 Reputation points Microsoft Employee
    2025-03-21T08:50:05.5133333+00:00

    Is there a way to generate MSI token in Synapse without using mssparkutils for services like ServiceBus and EventHub

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.