Unable to connect kafka topic as input in Azure Stream Analytics Job

Hi ,

Thanks for reaching out to Microsoft Q&A.

The error "Exception thrown while trying to retrieve Kafka partition count for <...>" in ASA usually indicates a connectivity or configuration issue between ASA and your Kafka cluster.

Here are the key areas you can check:

1. Kafka Configuration

Broker endpoint: Ensure the Kafka broker address is correct, including the port (default is 9092).

Security protocol: If your Kafka cluster requires SSL/SASL, you must provide the correct configuration in ASA.

For example, if using SASL over SSL, make sure to configure Authentication Mode, Username, and Password.

Zookeeper info not needed: ASA connects directly to the Kafka broker, not via Zookeeper.

2. Kafka Topic & Permissions

Topic exists: Confirm the topic name is correct and actually exists in the Kafka cluster.

Permissions: The service principal or authentication you are using must have permission to read metadata from the topic (especially the partition info).

3. Networking

Public vs. Private Kafka:

If the Kafka cluster is on a private network (e.g., hosted on a VM inside a VNet), ASA needs proper VNet integration or Private Endpoint setup.

Firewall/NSG Rules: Ensure no firewall or NSG is blocking communication on the Kafka port.

DNS Resolution: The Kafka broker DNS name must be resolvable from ASA.

4. TLS/SSL Issues

If SSL is enabled on your Kafka cluster:

ASA expects a trusted certificate. Self-signed certificates often cause failures.

• Use a Kafka cluster with a certificate signed by a trusted Certificate Authority (CA) or upload a custom trusted CA to ASA.

5. Partition Count Retrieval Error

This step is internal to ASA:

it tries to query the metadata from the broker. Failures can mean:

The broker is unreachable.

Incorrect auth credentials.

SSL handshake failure.

Diagnostic Steps

• Try to connect using a Kafka tool like kafkacat or kafka-console-consumer.sh from a VM in the same env as ASA.

Enable diagnostic logs in ASA to capture more details.

• Check Activity Logs in Azure for failures or configuration rejections.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.