Azure AI Video Indexer enabled by Arc

Azure AI Video Indexer enabled by Arc is an Azure Arc extension enabled service that runs video and audio analysis, and generative AI on edge devices. The solution is designed to run on Azure Arc enabled Kubernetes and supports many video formats, and assumes that one Video Indexer resource is mapped to one extension. It can transcribe spoken content in more than 35 source languages and supports translation to English. For a full list of supported languages, see Supported languages per scenario.

If you aren't already familiar with Azure AI Video Indexer, we recommend that you familiarize yourself with the cloud service first.

Additionally, before you start working with Azure AI Video Indexer enabled by Arc, review the transparency note to understand usage restrictions.

What is Azure Arc and Azure Arc-enabled Kubernetes?

Azure Arc simplifies governance and management of complex environments that extend across data centers, multiple clouds, and edge by delivering a consistent multicloud and on-premises management platform.

Azure Arc-enabled Kubernetes allows you to attach Kubernetes clusters running anywhere so that you can manage and configure them in Azure. By managing all of your Kubernetes resources in a single control plane, you can enable a more consistent development and operations experience to run cloud-native apps anywhere and on any Kubernetes platform.

When the Azure Arc agents get deployed to the cluster, an outbound connection to Azure is initiated, using industry-standard SSL to secure data in transit.

Once clusters are connected to Azure, they're represented as their own resources in Azure Resource Manager (ARM), and they can be organized using resource groups and tagging.

For more information about Azure Arc and Arc-enabled Kubernetes, see Azure Arc overview and What is Azure Arc-enabled Kubernetes?

What is an Azure Arc extension?

Virtual machine (VM) extensions are small applications that provide post-deployment configuration and automation tasks on Azure VMs. For example, if a virtual machine requires software installation, anti-virus protection, or to run a script in it, a VM extension can be used. For more information about extensions, see Virtual machine extension management with Azure Arc-enabled servers.

The Azure AI Video Indexer extension installs and deploys Azure AI Video indexer to the Kubernetes cluster.

Azure AI Video Indexer enabled by Arc only supports Azure Resource Manager accounts. Resource Manager operations are decoupled from video insight operations. This design allows you to perform analysis on your edge devices without the need to upload your media assets to Azure.

The extension is supported in direct connection mode scenarios only. While all processing is performed in the edge environment, control plane information is sent to the cloud for billing and monitoring purposes. New extension versions are downloaded from the cloud. No customer data, such as what videos were indexed or indexed insights, are sent from the edge location to the cloud.

Language models

The Phi language model is included and automatically connected with your VI extension. You can start using it immediately. For more information about using language models with VI, see:

• Use textual summarization
• Use Azure AI Video Indexer to create prompt content
• Azure AI Video Indexer Bring Your Own (BYO) AI model overview

See also the transparency note for textual summarization with VI enabled by Arc for hardware requirements, limitations, and known issues.

Use cases

• Data governance – You can bring the AI to the content instead of vice versa. Use Azure AI Video Indexer enabled by Arc when you can’t move indexed content from on-premises to the cloud due to:
  • Regulations.
  • Architecture decisions.
  • Data store being too large, making lift and shift a significant effort.
• On-premises workflow – Your indexing process is part of an on-premises workflow, and you want to lower the indexing duration latency affecting the flow.
• Pre-indexing – You want to index before uploading the content to the cloud. To create clarity, you can presort your on-premises video or audio archive, and then only upload it for standard or advanced indexing in the cloud.

Example deployment

The following diagram shows the Azure AI Video Indexer extension running on Azure Arc. There are three types:

1. Store type A uses both vision and audio presets.
2. Store type B uses only vision presets. It also has a custom model. For more information about using a custom model with Azure AI Video Indexer enabled by Arc, see Bring Your Own AI model.
3. Store C uses only audio presets.

The extension is stored on each edge device and each device is associated with a single Azure AI Video Indexer account that interfaces with Azure Arc and the cloud.

Supported AI presets

Azure AI Video Indexer enabled by Arc supports the following indexing presets:

Minimum hardware requirements

Video Indexer enabled by Arc is designed to run on any Arc enabled Kubernetes environment.

Minimum software requirements

Supported input formats and codecs

The following section lists the supported input formats and codecs for Azure AI Video Indexer enabled by Arc.

Video formats

• AVI (.avi)
• FLV (with H.264 and AAC codecs) (.flv)
• ISMV (.isma, .ismv)
• Matroska (.mkv)
• MP4 (.mp4, .m4a, .m4v)
• MXF (.mxf)
• MPEG2-TS
• QuickTime (.mov)
• WAVE/WAV (.wav)
• Windows Media Video (WMV)/ASF (.wmv, .asf)

Video codecs

• AVC 8-bit/10-bit, up to 4:2:2, including AVCIntra
• Digital video (DV) (in AVI files)
• DVCPro/DVCProHD (in MXF container)
• HEVC/H.265
• MPEG-1
• MPEG-2 (up to 422 Profile and High Level; including variants such as Sony XDCAM, Sony XDCAM HD, Sony XDCAM IMX, CableLabs®, and D10)
• MPEG-4 Part 2
• VC-1/WMV9

Audio codecs up to two tracks

• AAC (AAC-LC, AAC-HE, and AAC-HEv2)
• FLAC
• MPEG Layer 2
• MP3 (MPEG-1 Audio Layer 3)
• VORBIS
• WAV/PCM
• Windows Media Audio

Bring your own model

Azure AI Video Indexer enabled by Arc also supports bringing your own model. For more information, see the Bring Your Own Model (BYO) article.

Limitations

• The supported file size for indexing is up to 2 GB.
• Upgrading the extension:
  • Extension support applies to the latest version only.
  • We recommend setting the auto-upgrade property to true. The setting keeps the extension up to date.
  • If the auto upgrade setting is set to false, you should upgrade the version incrementally. Jumping between versions can cause indexing processes to fail.
• After extension installation or upgrade, expect the first index\translation process duration to be longer than normal. The longer duration is due to AI model image download. The duration varies depending on network speed.
• Only one Video Indexer extension can be deployed per Arc enabled Kubernetes cluster.
• The cluster's volume performance (based on storage class) has significant influence on the turnover duration of the indexing job especially since the frame extraction is writing all frames into the volume.
• Only extension access tokens are supported. You can obtain extension access tokens from API/CLI. For samples to get-access-token, see How to access the extension.
• Video error messages aren't stored due to memory limitations.

Azure Container Storage enabled by Arc

We recommend that you use Azure Container Storage enabled by Azure Arc for storage. For more information, see the following articles:

• What is Azure Container Storage enabled by Azure Arc?
• Prepare Linux for Edge Volumes

Related content

• Try the Azure AI Video Indexer enabled by Arc sample on GitHub
• Try the Azure AI Video Indexer enable by Arc Jumpstart
• Try deploying in the Azure portal using the Azure AI Video Indexer enabled by Arc quickstart