get_directory_properties and get_file_properties use the blob REST API

[mathieulongtin]

• Package Name: azure-storage-file-datalake
• Package Version: 12.8.0
• Operating System: CentOS
• Python Version: 3.10

Describe the bug
The datalake Python API uses the blob REST API for get_XXX_properties() methods, as a result, it doesn't return the file owner. group, permissions or ACL.

To Reproduce
Here's what get_file_properties returns:

In [27]: ff
Out[27]: <azure.storage.filedatalake._data_lake_file_client.DataLakeFileClient at 0x7f57ee12d2d0>
In [28]: dict(ff.get_file_properties())
Out[28]:
{'name': 'hello/world',
 'etag': '"0x8DB0611ED1897B0"',
 'deleted': False,
 'metadata': {},
 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None},
 'last_modified': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
 'creation_time': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
 'size': 5,
 'deleted_time': None,
 'expiry_time': None,
 'remaining_retention_days': None,
 'content_settings': {...}

Expected behavior
I would expect the information from calling this, including owner, group, permissions and ACL:

In [29]: ff._client.path.get_properties(cls=lambda a,b,c: (a,b,dict(c)))
Out[29]:
(<azure.core.pipeline.PipelineResponse at 0x7f57e7544700>,
 None,
 {'Accept-Ranges': 'bytes',
  'Cache-Control': None,
  'Content-Disposition': None,
  'Content-Encoding': None,
  'Content-Language': None,
  'Content-Length': 5,
  'Content-Range': None,
  'Content-Type': 'application/octet-stream',
  'Content-MD5': 'XUFAKrxLKna5cZ2REBfFkg==',
  'Date': datetime.datetime(2023, 2, 3, 21, 25, 52, tzinfo=datetime.timezone.utc),
  'ETag': '"0x8DB0611ED1897B0"',
  'Last-Modified': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
  'x-ms-request-id': '17183129-701f-000b-6916-38b966000000',
  'x-ms-version': '2021-08-06',
  'x-ms-resource-type': 'file',
  'x-ms-properties': '',
  'x-ms-owner': '--redacted--',
  'x-ms-group': '--redacted--',
  'x-ms-permissions': 'rw-rw----',
  'x-ms-acl': None,
  'x-ms-lease-duration': None,
  'x-ms-lease-state': 'available',
  'x-ms-lease-status': 'unlocked'})

I note however that creation time is not in there.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[SaurabhSharma-MSFT]

@mathieulongtin We are redirecting this to services team to look into.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details

---

• Package Name: azure-storage-file-datalake
• Package Version: 12.8.0
• Operating System: CentOS
• Python Version: 3.10

Describe the bug
The datalake Python API uses the blob REST API for get_XXX_properties() methods, as a result, it doesn't return the file owner. group, permissions or ACL.

To Reproduce
Here's what get_file_properties returns:

In [27]: ff
Out[27]: <azure.storage.filedatalake._data_lake_file_client.DataLakeFileClient at 0x7f57ee12d2d0>
In [28]: dict(ff.get_file_properties())
Out[28]:
{'name': 'hello/world',
 'etag': '"0x8DB0611ED1897B0"',
 'deleted': False,
 'metadata': {},
 'lease': {'status': 'unlocked', 'state': 'available', 'duration': None},
 'last_modified': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
 'creation_time': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
 'size': 5,
 'deleted_time': None,
 'expiry_time': None,
 'remaining_retention_days': None,
 'content_settings': {...}

Expected behavior
I would expect the information from calling this, including owner, group, permissions and ACL:

In [29]: ff._client.path.get_properties(cls=lambda a,b,c: (a,b,dict(c)))
Out[29]:
(<azure.core.pipeline.PipelineResponse at 0x7f57e7544700>,
 None,
 {'Accept-Ranges': 'bytes',
  'Cache-Control': None,
  'Content-Disposition': None,
  'Content-Encoding': None,
  'Content-Language': None,
  'Content-Length': 5,
  'Content-Range': None,
  'Content-Type': 'application/octet-stream',
  'Content-MD5': 'XUFAKrxLKna5cZ2REBfFkg==',
  'Date': datetime.datetime(2023, 2, 3, 21, 25, 52, tzinfo=datetime.timezone.utc),
  'ETag': '"0x8DB0611ED1897B0"',
  'Last-Modified': datetime.datetime(2023, 2, 3, 18, 10, 26, tzinfo=datetime.timezone.utc),
  'x-ms-request-id': '17183129-701f-000b-6916-38b966000000',
  'x-ms-version': '2021-08-06',
  'x-ms-resource-type': 'file',
  'x-ms-properties': '',
  'x-ms-owner': '--redacted--',
  'x-ms-group': '--redacted--',
  'x-ms-permissions': 'rw-rw----',
  'x-ms-acl': None,
  'x-ms-lease-duration': None,
  'x-ms-lease-state': 'available',
  'x-ms-lease-status': 'unlocked'})

I note however that creation time is not in there.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Author:	mathieulongtin
Assignees:	jalauzon-msft, vincenttran-msft
Labels:	Storage, question, Service Attention, customer-reported, needs-team-attention
Milestone:	-

[jalauzon-msft]

Hi @mathieulongtin, thanks for raising the issue. You are correct that the Datalake SDK calls the Blob endpoint when fetching properties. This is by design.

However, it does seem to be gap that we are missing the owner, group, and permissions properties (actually some others as well) when calling the Datalake get_file_properties or get_directroy_properties. The reason for this is not because the Blob endpoint does not return these headers, it does for HNS enabled accounts, its actually that we don't parse them in the Blob SDK since they are specific to Datalake.

We do believe this is a gap and that these properties should be returned for Datalake and so we are discussing a solution to this, and I will get back to you when I know more. Thanks.

[mathieulongtin]

That the data lake Python API uses the blob's REST API for anything kind of baffles me.

[jalauzon-msft]

Hi @mathieulongtin, to update, we have designed a solution to address this gap and will be working on implementing it soon. You can expect to see this in an upcoming release within the next few months.

In the meantime, you can try using get_access_control in the Datalake SDK to get the file ACLs/etc. This API does call into the DFS endpoint and, while it's a separate API, it should give you what you need. Thanks.

[github-actions]

Hi @mathieulongtin. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[jalauzon-msft]

Hi @mathieulongtin, a fix for this has been merged into the feature branch for our upcoming release. #29491

This will go to a preview release within the next few weeks, followed by a GA release sometime after.

[github-actions]

Hi @mathieulongtin, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.