Endpointing and latency issue with streaming azure STT

Hi @Hasan Ali

Currently, Azure Speech-to-Text supports various languages, but the documentation on semantic end pointing may not specify which ones are applicable. I recommend reviewing the latest Azure Speech Language Support documentation to confirm if your target languages are supported for semantic end pointing. To reduce latency, consider the following best practices: Ensure streaming is properly configured in your implementation to provide earlier access to interim results. If possible, batch multiple requests together to significantly improve performance. Avoid mixing different workloads on a single endpoint to prevent delays due to queuing. Check your SDK configuration settings to ensure they are optimized for low latency (such as setting the appropriate output format). Further details are available in the Performance and latency documentation

Short utterances can be problematic due to the inherent processing time needed to interpret and respond. Stream audio starting from the first received chunk to make the interaction feel more immediate. Consider using the Speech SDK’s capabilities for more efficient buffering and streaming of audio data, as described in the Lower speech synthesis latency documentation.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Hello Hasan Ali,

Thank you for posting your question in the Microsoft Q&A forum. 

1. Supported Languages for Semantic Endpointing - Azure Speech-to-Text's semantic endpointing feature, designed to intelligently detect natural speech pauses rather than relying solely on silence, currently supports a limited set of languages, primarily focusing on English (en-US) with some additional European languages including Spanish (es-ES), French (fr-FR), and German (de-DE).  If your application uses an unsupported language, consider: 

• Adjusting silence-based thresholds (initialSilenceTimeoutMs, endSilenceTimeoutMs) to reduce false positives. 
• Post-processing logic to filter out unnatural pauses. 
• Hybrid endpointing (combining silence detection with intent-based pauses via LUIS or OpenAI). 

2. Reducing Latency for Interim Results - To minimize latency in Azure Speech-to-Text's continuous recognition, prioritize low-latency network configurations by deploying resources in regions closest to users and enabling WebSocket compression. Adjust recognition parameters like SegmentationSilenceTimeout (300–500ms) to balance responsiveness and accuracy, while opting for the Detailed output format to receive partial results faster. For optimal performance, pre-warm sessions with silent audio and monitor real-time metrics using SpeechSDK logging to identify and address bottlenecks. 

To optimize real-time responsiveness: 

• Use Detailed output format instead of Simple to get faster partial results. 
• Lower SegmentationSilenceTimeout (e.g., 300ms) to trigger quicker interim updates. 
• Enable WebSocket compression (EnableCompression) to reduce data transfer delays. 
• Deploy in a region closest to users (e.g., westus2 for North America) to minimize network latency. 

3. Accelerating Single-Word Utterance Processing - To optimize single-word utterance processing in Azure Speech-to-Text, configure the PhraseDetectionTimeout to 0 to prevent unnecessary waiting periods and set a short MaxPhraseDuration (e.g., 1 second) to enforce faster finalization. Additionally, pre-warm the recognition session with a brief silent preamble to mitigate cold-start delays and consider batching requests if real-time constraints persist. These adjustments prioritize speed for short inputs while maintaining accuracy in quick-interaction scenarios. 

For short utterances: 

• Set PhraseDetectionTimeout to 0 to disable waiting for additional input. 
• Use MaxPhraseDuration (e.g., 1s) to force faster finalization. 
• Pre-warm the recognition session (send a silent audio preamble) to avoid cold-start delays. 
• Consider batch processing if real-time latency is unacceptable for single words. 

Few additional recommendations:

• Test with SpeechSDK Logging to diagnose bottlenecks. 
• Leverage gRPC streaming (if available) for lower-latency than REST. 
• File a feature request for expanded semantic endpointing language support. 

Some related Microsoft documentation that may help you learn more -  

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/  

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-identification?tabs=once&pivots=programming-language-csharp#use-speech-to-text  

If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue. Your contribution to the Microsoft Q&A community is highly appreciated.