Manage personal data in Azure Monitor Logs

Azure Monitor Logs is a data store where personal data is likely to be found. This article explains where Azure Monitor Logs stores personal data and how to manage this data.

Strategy for personal data handling

While it's up to you and your company to define a strategy for handling personal data, here are a few approaches, listed from most to least preferable from a technical point of view:

• Filter out, obfuscate, anonymize, or adjust collected data to exclude it from being considered "personal" using data collection transformations. This is by far the preferred approach, which saves you the need to create a costly and impactful data handling strategy.
• Normalize the data to reduce negative affects on the data platform and performance. For example, instead of logging an explicit User ID, create a lookup to correlate the username and their details to an internal ID that can then be logged elsewhere. That way, if a user asks you to delete their personal information, you can delete only the row in the lookup table that corresponds to the user.
• If you need to collect personal data:
  • Use the Delete Data API or Purge API and the Query API to export and delete any personal data associated with a user.
  • Use summary rules to remove or obfuscate personal data in a new table that can be shared more widely, and limit access to the table with the personal data by managing table-level read access.

Where to look for personal data in Azure Monitor Logs

Azure Monitor Logs prescribes a schema to your data, but allows you to override every field with custom values. You can also ingest custom schemas. Therefore, it's impossible to say exactly where personal data exists in your specific workspace. The following locations, however, are good starting points in your inventory.

• IP addresses: Log Analytics collects various IP information in multiple tables. For example, the following query shows all tables that collected IPv4 addresses in the last 24 hours:

  search * 
  | where * matches regex @'\b((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.|$)){4}\b' //RegEx originally provided on https://stackoverflow.com/questions/5284147/validating-ipv4-addresses-with-regexp
  | summarize count() by $table
  

• User IDs: You can find user usernames and user IDs in various solutions and tables. Look for a particular username or user ID across your entire dataset using the search command:

  search "<username or user ID>"
  

  Remember to look not only for human-readable usernames but also for GUIDs that can be traced back to a particular user.

• Device IDs: Like user IDs, device IDs are sometimes considered personal data. Use the approach described for user IDs to identify tables that hold personal data.

• Custom data: Azure Monitor Logs lets you collect custom data through custom logs, custom fields, the Logs Ingestion API, and as part of system event logs. Check all custom data for personal data.

• Solution-captured data: Because the solution mechanism is open-ended, we recommend reviewing all tables generated by solutions to ensure compliance.

Export, delete, or purge personal data

We strongly recommend you restructure your data collection policy to stop collecting, filter out, obfuscate or anonymize personal data, or otherwise modify such data until it's no longer considered personal using data collection transformations. In handling personal, data you incur costs in defining and automating a strategy, building an interface through which your customers interact with their data, and ongoing maintenance. It's also computationally costly for Log Analytics and Application Insights, and a large volume of concurrent Query, Delete Data, or Purge API calls can negatively affect all other interactions with Log Analytics functionality. However, if you have to collect personal data, follow the guidelines in this section.

View and export

Use the Log Analytics query API to send for view and export data requests.

You need to implement the logic for converting the data to an appropriate format for delivery to your users. Azure Functions is a great place to host such logic.

Delete

The Azure Monitor Logs Delete Data API lets you make asynchronous requests to remove data for a specific table in your Log Analytics workspace. Use the Delete Data operation sparingly to avoid potential risks, performance impact, and the potential to skew all-up aggregations, measurements, and other aspects of your Log Analytics data. See the Strategy for personal data handling section for alternative approaches to handling personal data.

If you need to comply with General Data Protection Regulation (GDPR) requirements, use the Purge API, which is less performant and only supports operations required for GDPR compliance.

Purge

Azure Monitor's Purge API lets you purge personal data, as required by GDPR. The Purge API is less performant than the Delete Data API. Azure Monitor recommends using the Delete Data API and only authorizes purge requests required for GDPR compliance.

To manage system resources, we limit purge requests to 50 requests an hour. Batch the execution of purge requests by sending a single command whose predicate includes all user identities that require purging. Use the in operator to specify multiple identities. Run the query before executing the purge request to verify the expected results.

Permissions required

Purge log data

• The Workspace Purge POST API takes an object specifying parameters of data to delete and returns a reference GUID.

• The Get Purge Status POST API returns an 'x-ms-status-location' header that includes a URL you can call to determine the status of your purge operation. For example:

  x-ms-status-location: https://management.azure.com/subscriptions/[SubscriptionId]/resourceGroups/[ResourceGroupName]/providers/Microsoft.OperationalInsights/workspaces/[WorkspaceName]/operations/purge-[PurgeOperationId]?api-version=2015-03-20
  

Next steps

• Learn more about security in Azure Monitor.
• Learn more about how Application Insights collects, processes, and secures data.