Azure AI Search Vector Index showing search.answers are in triplicate

Question

Azure AI Search Vector Index showing search.answers are in triplicate

Charles Lawson 0

Azure AI Search Vector Index showing search.answers are in triplicate. deleted and recreated index/indexer same results., the document index count is over 2000 but the blob storage is only 46 pdfs?

Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-21T23:35:15.4933333+00:00
Hi @Charles Lawson

Ensure that the documents being indexed are not duplicated in your source data. Indexing the same PDF multiple times can result in triplicate results.

Check your query configuration. Improper filtering or aggregation can lead to duplicates. Use distinct or unique filters in your query.

After deleting and recreating the index/indexer, confirm that the indexing process is set up correctly to avoid re-indexing the same documents multiple times.

The difference between the document index count and the number of PDFs in blob storage may indicate additional indexing artifacts or issues. Verify that the indexing process references the correct documents.
https://learn.microsoft.com/en-us/azure/search/vector-store#physical-structure-and-size
https://learn.microsoft.com/en-us/azure/search/search-howto-indexing-azure-blob-storage
Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-22T23:16:52.1133333+00:00

Hi @Charles Lawson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-23T22:04:00.2633333+00:00

Hi @Charles Lawson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-24T12:38:15.3+00:00
Hello @Charles Lawson ,could please enable deletion tracking and re-run the indexer.

Refer to this MSDOC for Change and delete detection using indexers for Azure Storage in Azure AI Search.

Azure AI Search Does Not Automatically Delete Indexed Documents When a Blob is Deleted However, the following methods ensure that indexed documents are removed: - This method only works for Azure Blob Storage (not ADLS Gen2 or Azure Files). - It requires enabling soft delete on the storage account. - Azure AI Search detects soft-deleted blobs and removes the corresponding index documents. Enabling Soft Delete in Data Sources While creating the connection in Data Sources, make sure to: - Select the Track deletions checkbox. - Choose Native blob soft delete for soft deletion. The following JSON configuration is supported starting from API version 2024-07-01. Ensure you update the version accordingly before applying the changes:

"dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName": "IsDeleted", "softDeleteMarkerValue": "true" },

You can use the following REST API request to create or update a data source:

Refer to this link to Update Data Source in Azure AI Search .

PUT https://[service-name].search.windows.net/datasources/blob-datasource?api-version=2024-07-01 { "name": "blob-datasource", "type": "azureblob", "credentials": { "connectionString": "<your storage connection string>" }, "container": { "name": "my-container" }, "dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName": "IsDeleted", "softDeleteMarkerValue": "true" } }

When a file is deleted in Azure Storage, you need to reset the Indexer and run it to reflect the changes in the search index.
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-24T17:24:28.8033333+00:00

Hello @Charles Lawson , Just checking in to see if the provided comment helped. If you have any further queries, do let us know.
Charles Lawson 0 Reputation points

2025-04-27T13:27:54.3266667+00:00

By choosing Native blob soft delete for soft deletion for the blob data source and deleting and creating a new index the search displayed single records. Thank you all for your help
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-28T03:47:21.4233333+00:00

Hello Charles Lawson , thank you for the confirmation. I am posting the above comment to answer your query. Please accept it and click yes. Thank you.

1 answer

Your answer

Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-21T23:35:15.4933333+00:00

Hi @Charles Lawson

Ensure that the documents being indexed are not duplicated in your source data. Indexing the same PDF multiple times can result in triplicate results.

Check your query configuration. Improper filtering or aggregation can lead to duplicates. Use distinct or unique filters in your query.

After deleting and recreating the index/indexer, confirm that the indexing process is set up correctly to avoid re-indexing the same documents multiple times.

The difference between the document index count and the number of PDFs in blob storage may indicate additional indexing artifacts or issues. Verify that the indexing process references the correct documents.
https://learn.microsoft.com/en-us/azure/search/vector-store#physical-structure-and-size
https://learn.microsoft.com/en-us/azure/search/search-howto-indexing-azure-blob-storage
Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-22T23:16:52.1133333+00:00

Hi @Charles Lawson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Laxman Reddy Revuri 4,000 Reputation points Microsoft External Staff

2025-04-23T22:04:00.2633333+00:00

Hi @Charles Lawson
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-24T12:38:15.3+00:00

Hello @Charles Lawson ,could please enable deletion tracking and re-run the indexer.

Refer to this MSDOC for Change and delete detection using indexers for Azure Storage in Azure AI Search.

Azure AI Search Does Not Automatically Delete Indexed Documents When a Blob is Deleted However, the following methods ensure that indexed documents are removed: - This method only works for Azure Blob Storage (not ADLS Gen2 or Azure Files). - It requires enabling soft delete on the storage account. - Azure AI Search detects soft-deleted blobs and removes the corresponding index documents. Enabling Soft Delete in Data Sources While creating the connection in Data Sources, make sure to: - Select the Track deletions checkbox. - Choose Native blob soft delete for soft deletion. The following JSON configuration is supported starting from API version 2024-07-01. Ensure you update the version accordingly before applying the changes:

"dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName": "IsDeleted", "softDeleteMarkerValue": "true" },

You can use the following REST API request to create or update a data source:

Refer to this link to Update Data Source in Azure AI Search .

PUT https://[service-name].search.windows.net/datasources/blob-datasource?api-version=2024-07-01 { "name": "blob-datasource", "type": "azureblob", "credentials": { "connectionString": "<your storage connection string>" }, "container": { "name": "my-container" }, "dataDeletionDetectionPolicy": { "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy", "softDeleteColumnName": "IsDeleted", "softDeleteMarkerValue": "true" } }

When a file is deleted in Azure Storage, you need to reset the Indexer and run it to reflect the changes in the search index.
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-24T17:24:28.8033333+00:00

Hello @Charles Lawson , Just checking in to see if the provided comment helped. If you have any further queries, do let us know.
Charles Lawson 0 Reputation points

2025-04-27T13:27:54.3266667+00:00

By choosing Native blob soft delete for soft deletion for the blob data source and deleting and creating a new index the search displayed single records. Thank you all for your help
Sampath 2,345 Reputation points Microsoft External Staff

2025-04-28T03:47:21.4233333+00:00

Hello Charles Lawson , thank you for the confirmation. I am posting the above comment to answer your query. Please accept it and click yes. Thank you.

Answer 1

Hello @Charles Lawson ,

Thank you for your update, and glad to hear the issue is resolved!

Just to summarize for clarity and for future readers:

Since your source was Blob Storage and the index was showing triplicate results, the root cause was likely due to stale or orphaned documents not getting cleaned up properly during re-indexing. Azure AI Search indexers, by default, do not detect deletions unless you enable .

By choosing Native Blob Soft Delete for your blob data source and recreating the index , you enabled Azure AI Search to automatically detect and remove blobs that were soft deleted. This ensured the search index reflected only the current active files, eliminating duplicate or triplicate entries.

Always enable Native Blob Soft Delete if your storage is Azure Blob Storage.
Ensure the storage account's soft delete feature is enabled with a sufficient retention period.
Configure the data source with Track deletions checked, and select Native blob soft delete.
Use the latest supported API version (e.g., 2024-07-01-preview) when configuring via REST API.

References:

Please don't forget to click "Accept the answer" and click "yes" wherever the information provided helps you, this can be beneficial to other community members.

If you have any more questions, feel free to click Comment

Sampath 2,345 Reputation points Microsoft External Staff

2025-04-29T15:59:07.83+00:00

Hello @Charles Lawson , please click "Accept the answer” and "Yes" for the answer . So it can benefit other community members reading this thread.

Share via

Azure AI Search Vector Index showing search.answers are in triplicate

1 answer

Your answer