Polybase accessing s3 storage error - Content of the directory cannot be listed

Question

Polybase accessing s3 storage error - Content of the directory cannot be listed

Haijin Li 21

I have the following script for accessing on-prem S3-compatible storage.

CREATE DATABASE SCOPED CREDENTIAL s3
 WITH IDENTITY = 'S3 Access Key',
 SECRET = '****:****' ;
END
GO 

CREATE EXTERNAL DATA SOURCE ds_s3
WITH
(   LOCATION = 's3://s3server.test.com:9000/'
,   CREDENTIAL = s3
);
GO

-- query with error 
SELECT *
FROM
    OPENROWSET(
        BULK 'testbucket/test1/*.parquet',
        DATA_SOURCE = 'ds_s3',
        FORMAT='PARQUET'
    ) AS src
GO

-- Error msg
-- Content of directory on path '/testbucket/test1/*.parquet' cannot be listed.

Additional information

I am not sure the port number is needed for the data source. With the port number included, I got the error in 80 seconds. Without the port number, I got the same error immediately.
There is no log entries related to above errors at C:\Program Files\Microsoft SQL Server\MSSQL16.MSSQLSERVER\MSSQL\Log\Polybase
The credential (access key and secret key) works. I tested it with a script accessing those parquet files via https.

I also accessed the parquet files using AWS CLI successfully, after I supplied the access key using aws configure command. aws s3 ls --endpoint-url https://s3server.test.com s3://testbucket/test1 --recursive

AWS CLI does not take * wildcard so the command below output nothing. But I do not think it is the issue. aws s3 ls --endpoint-url https://s3server.test.com s3://testbucket/test1/*.parquet --recursive

Thank you.

Punit R 0 Reputation points

2025-05-02T18:54:34.21+00:00

I am running in to this exact issue. Did you find the solution to this ?

1 answer

Your answer

Punit R 0 Reputation points

2025-05-02T18:54:34.21+00:00

I am running in to this exact issue. Did you find the solution to this ?

Answer 1

Alejandro Echeverria 6

Take a look at the documentation: https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configure-s3-compatible?view=sql-server-ver16

There is a part that mentions your exact same problem "cannot be listed".

Haijin Li 21 Reputation points

2024-08-15T18:08:42.1633333+00:00
Alejandro,

I re-created the external data source with the virtual url option, but got the same error message.

CREATE EXTERNAL DATA SOURCE ds_s3 WITH ( LOCATION = 's3://s3server.test.com:9000/' , CREDENTIAL = s3 , CONNECTION_OPTIONS = '{"s3":{"url_style":"virtual_hosted"}}' ); GO

Also, according to "Basic Authentication" section at <https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configure-s3-compatible?view=sql-server-ver16>, one limitation of basic authentication approach is that "The credential name created must contain the bucket name unless this credential is for a new external data source." Could you help me clarify what this means? I tried naming a credential after a bucket name but got the same error message.

Regards,

Haijin Li 21

I did the following tests confirming that virtual hosted url is NOT configured on my local S3 storage system.

rem This statement worked.
aws s3 ls --endpoint-url https://s3server.test.com s3://testbucket/test1 --recursive


rem This statement did not work
aws s3 ls --endpoint-url https://testbucket.s3server.test.com s3://test1 --recursive
rem The error is  - Could not connect to the endpoint URL: "https://testbucket.s3server.test.com/test1?list-type=2&prefix=&encoding-type=url"
aws s3 ls --endpoint-url https://testbucket.s3server.test.com s3://test1 --recursive

In addition, using AWS CLI, I confirmed that the Access Key has the permissions of get-bucket-location, list-objects, and get-object APIs.

Any other possible reason for the error "Content of the directory cannot be listed"?

Thank you.

Share via

Polybase accessing s3 storage error - Content of the directory cannot be listed

1 answer

Your answer