Error writing to OneLake datastore from Azure ML pipeline
I'm currently building an Azure ML pipeline in which I'd like to be able to send outputs to our OneLake lakehouse's Files area, rather than storing them in the blob storage associated with the Azure ML workspace.
I'm using output paths that look like this (in my Python-defined @pipeline function), to dynamically redirect output to certain folders on the datastore:
azureml://datastores/{datastore}/paths/{path}
If the value of {datastore} is workspaceblobstore, i.e., the default storage for the workspace, the pipeline runs fine. However, if the value of {datastore} is the name of our OneLake datastore, the job fails immediately (within seconds, like it barely got started) with the following error:
Failed to submit job due to Exception: Error converting value "OneLake" to type 'Microsoft.Aether.BlueBox.DataStoreClient.Contracts.DataStoreType'. Path 'dataStoreType', line 3, position 28..
A couple of extra bits of info:
- This happens regardless of whether I use the above form of the URI, or the longer one that includes the subscription ID, resource group, and workspace name.
- I can, however, launch Jupyter on the compute resource and use AzureMachineLearningFileSystem to read and write from OneLake just fine, using the long-form datastore URI.
- I can also browse the OneLake datastore from within the Azure ML datastore web UI, and everything looks fine.
- The OneLake datastore was created using identity-based access, and points to the Files area of the lakehouse.
- The system-assigned identity of the compute resource is a Contributor in the Fabric workspace containing the lakehouse.
Please let me know if I can provide more info, and thanks!