The torch.tensor with special type cannot be viewed properly in the Dataviewer.

[wjfu99]

I have gotten many conveniences from the Dataviewer, and it's a really useful tool that prompts me to traverse from Pycharm to VsCode. Thanks for all your contributions. But, recently, I found that the Dataviewer cannot handle the torch.tensor of particular types (bfloat16, qint8, quint8, etc.). It will be helpful to support them, especially since the bfloat16 type is ubiquitous in the LLMs domain.

Environment data

• VS Code version: 1.85.1
• Jupyter Extension version (available under the Extensions sidebar): v2023.11.1003402403
• Python Extension version (available under the Extensions sidebar): v2023.22.1
• OS (Windows | Mac | Linux distro) and version: Sonoma 14.1.1
• Python and/or Anaconda version: Python 3.9.17
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Remote

Expected behaviour

Display the torch.tensor when clicking the "view values in Data Viewer" button:

Actual behaviour

Jupyter extension raises an error:

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

1. Create a tensor of a special type, e.g., bfloat16, qint8.
2. Right-click to select "view values in Data Viewer".

import torch

tensor = torch.tensor([1.0, 2.0, 3.0], dtype=torch.bfloat16)
tensor = torch.quantize_per_tensor(torch.tensor([-1.0, 0.0, 1.0, 2.0]), 0.1, 10, torch.quint8)
tensor = torch.quantize_per_tensor(torch.tensor([-1.0, 0.0, 1.0, 2.0]), 0.1, 10, torch.qint8)

I tested all the tensor types listed in link, and found the followings are not supported:

• torch.bfloat16
• torch.quint8
• torch.qint8
• torch.qint32
• torch.quint4x2

By the way, the complex types are also not supported well, but I think they are not used frequently.

Logs

09:53:32.811 [info] Found debugAdapterPython on Debug Configuration to use
09:53:33.528 [info] Process Execution: ~/.conda/envs/llm/bin/python -c "import pandas;print(pandas.__version__)"
09:53:34.295 [error] SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at cb.getDataFrameInfo (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:265:133367)
    at Mu.ensureInitialized (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:61001)
    at Mu.getDataFrameInfo (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:59826)
    at LI.onVariablePanelShowDataViewerRequest (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:52073)
    at d.h (/mnt/data0/<username>/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2/out/vs/workbench/api/node/extensionHostProcess.js:141:140086)
09:53:34.295 [warn] DataScience Error SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at cb.getDataFrameInfo (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:265:133367)
    at Mu.ensureInitialized (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:61001)
    at Mu.getDataFrameInfo (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:59826)
    at LI.onVariablePanelShowDataViewerRequest (/mnt/data0/<username>/.vscode-server/extensions/ms-toolsai.jupyter-2023.11.1003402403-linux-x64/dist/extension.node.js:351:52073)
    at d.h (/mnt/data0/<username>/.vscode-server/bin/0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2/out/vs/workbench/api/node/extensionHostProcess.js:141:140086)

[amunger]

closing as the jupyter data viewer was deprecated #15486