azure stream analytics job issue.

Hi @Muralikrishna Chintada

Based on the information you provided, it seems like you are encountering several challenges related to cost management, autoscaling, data processing, and record handling in your Stream Analytics job. Here are some insights and recommendations to address the issues you mentioned:

Autoscaling and Data Reception Issue

When you enabled autoscaling for cost management, you experienced a stoppage in data reception in ADLS. This could be due to the autoscaling configuration impacting the job's processing capabilities. It's important to review the autoscaling settings and ensure they are optimized for your data processing requirements without causing disruptions.

Manual Scale vs. Autoscale

You mentioned that the job works with 2SU (Streaming Units) but not with 1SU, even though the data batch size is within the processing limits of 1SU. This discrepancy could be related to the specific workload characteristics and resource allocation. Consider adjusting the SU allocation based on the actual processing needs and performance observations.

Data Loss and Duplicates

The issue of records being dropped (750 out of 1000) and the generation of duplicate summary files after 2 hours indicates potential data processing or output handling issues. It's crucial to investigate the data flow, transformations, and output configurations to identify the root cause of data loss and duplication. Ensure that the data processing logic and output mechanisms are correctly implemented to prevent such issues.

Recommendations

• Review the autoscaling configuration and its impact on data reception in ADLS. Adjust the settings if necessary to ensure continuous data flow.
• Evaluate the SU allocation based on workload requirements and performance benchmarks to determine the optimal scaling for your job.
• Investigate the data loss and duplication issues by examining the data processing pipeline, transformations, and output generation logic. Make necessary adjustments to prevent data discrepancies.

By addressing these aspects and fine-tuning your Azure Stream Analytics job settings, you should be able to improve data processing efficiency, reduce data loss, and enhance the overall job performance.

If you need further assistance or have additional details to share, please feel free to provide more information for a more targeted resolution.

I hope this guidance helps in resolving your Azure Stream Analytics job issues. Let me know if you have any more questions or require further support.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Hello @Muralikrishna Chintada,

Could you provide some information

In the Autoscaling Behavior

What specific autoscaling rules are currently configured?

Are there any logs or metrics indicating why data stops flowing when autoscaling is enabled?

For Streaming Unit (SU) Allocation have you tested intermediate SU values like 1.5SU to see if performance improves?

And regarding the Data Loss Investigation are there any error logs or dropped event metrics that indicate why records are missing?

Hello @Muralikrishna Chintada ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.