Jupyter Notebooks native support through latest Python VS Code extension

[l0rd]

Is your enhancement related to a problem? Please describe.

Latest python extension includes jupyter support. I have tried it out but loading a jupyter notebook failed.

To try it out on che.osio -->

Describe the solution you'd like

This is the devfile I have been using:

devfile.yaml
---
apiVersion: 1.0.0
metadata:
  name: python-workspace
projects:
  - name: jupyter-hello-world1
    source:
      location: 'https://github.com/chasbecker/TextAnalysis.git'
      type: git
components:
  - type: chePlugin
    alias: ms-python/python/latest
    reference: https://gist.github.com/l0rd/7190c65773a23f393a839598b22f3c1a/raw/meta.yaml
    memoryLimit: 512Mi
  - type: cheEditor
    alias: che-theia
    reference: https://gist.github.com/l0rd/7190c65773a23f393a839598b22f3c1a/raw/theia.yaml
    memoryLimit: 512Mi
  - type: chePlugin
    alias: machine-exec
    id: eclipse/che-machine-exec-plugin/latest

vscode python meta.yaml

vscode-python-meta.yaml
---
apiVersion: v2
publisher: ms-python
name: python
version: 2020.4.76186
type: VS Code extension
displayName: Python
title: Python extension
description: Linting, Debugging (multi-threaded, remote), Intellisense, code formatting,
  refactoring, unit tests, snippets, and more.
icon: https://www.eclipse.org/che/images/logo-eclipseche.svg
repository: https://github.com/Microsoft/vscode-python
category: Language
firstPublicationDate: '2020-04-27'
spec:
  containers:
  - image: "quay.io/eclipse/che-sidecar-python:3.7.3-8f39348"
    name: vscode-python
    memoryLimit: '512Mi'
  extensions:
  - https://github.com/microsoft/vscode-python/releases/download/2020.2.62710/ms-python-release.vsix

Describe alternatives you've considered

I have also tried to add python in che-theia container:

FROM quay.io/eclipse/che-theia:7.12.0

USER root

RUN apk add --no-cache python3

# make some useful symlinks that are expected to exist
RUN cd /usr/bin \
	&& ln -s idle3 idle \
	&& ln -s pydoc3 pydoc \
	&& ln -s python3 python \
	&& ln -s python3-config python-config

USER theia

But it didn't help

Sub-tasks

• vscode.workspace.fs API fix #7269: complete support vscode.workspace.fs API eclipse-theia/theia#7908
• Adapt Che Theia to the new Theia FS API Adapt Che Theia to the new Theia FS API #17357
• Che Theia Web View doesn't work properly when a plugin is running in its own sidecar Che Theia Web View doesn't work properly when a plugin is running in its own sidecar #16870
• Adapt vscode.workspace.fs Plug-in API to Che-Theia's remote plugin system Provide sidecar filesystem access for VS Code extensions #19110

[ericwill]

I was working on upgrading the Python plugin today (it's almost a year out of date) as part of #16723. Thanks for doing some more investigation, once I get the Python extension upgraded (there are some bugs at the moment) it will be quite helpful.

[JPinkney]

Just wanted to note, we might be missing some plugin api: eclipse-theia/theia#3186 (comment) for the newest version of the python plugin. There seems to be a few other Jupyter issues on the theia repo as well: https://github.com/eclipse-theia/theia/issues?q=is%3Aissue+is%3Aopen+Jupyter

[ericwill]

Just wanted to note, we might be missing some plugin api: eclipse-theia/theia#3186 (comment) for the newest version of the python plugin. There seems to be a few other Jupyter issues on the theia repo as well: https://github.com/eclipse-theia/theia/issues?q=is%3Aissue+is%3Aopen+Jupyter

Yes, jupyter issues aside the latest python extension also complains about Python not being installed, even though it's present in the container and running python code still works. Lots of weirdness going on there.

[azatsarynnyy]

Yes, jupyter issues aside the latest python extension also complains about Python not being installed, even though it's present in the container and running python code still works.

It looks like a known issue microsoft/vscode-python#11870

[azatsarynnyy]

The missing parts of upstream Theia are tracked here:

• [python]Problems with starting ms-python VSCode extension eclipse-theia/theia#8022
• Support Notebooks eclipse-theia/theia#3186 (comment)

[azatsarynnyy]

Editors Team will work on adding CustomEditor API support (eclipse-theia/theia#6636)
as Python extension relies on it for rendering Jupyter Notebooks.

[azatsarynnyy]

Several updates:

1. After a more detailed investigation, it turned out that Python VS Code extension needs support for CustomEditor Plugin API only as an experimental feature that is not used by default. So, it's not a blocker for Jupyter Notebooks support.

2. There's WIP PR fix #7269: complete support vscode.workspace.fs API eclipse-theia/theia#7908 (FS Plugin API) in Theia upstream that fixes running Jupyter Notebooks in vanilla Theia. Che Theia isn't compatible with these changes and needs to be adapted Adapt Che Theia to the new Theia FS API #17357.
   I fixed the compilation errors in Che Theia (Adapt to new upstream FS API che-theia#798) and tried to build Che Theia based on Theia from upstream Theia WIP PR branch, but it doesn't fix this issue - ... is not a valid file name

So, we will postpone work on CustomEditor Plugin API and start investigating why new FS Plugin API fixes Jupyter Notebooks support in vanilla Theia but not in Che Theia.

[azatsarynnyy]

After adapting Che Theia to the latest Theia FS Plugin API eclipse-che/che-theia#823 there's some progress with the Jupyter Notebooks support 🎉

It looks like working well when the Python VS Code extension is running in the Theia container

But it doesn't when it's running in a separate sidecar. I think it requires adapting the WebView API to work remotely #16870
So, it will be the next step.

[azatsarynnyy]

I'm converting it into an epic since multiple steps are required.
See the sub-tasks in the description.

[azatsarynnyy]

After examination of the Python extension's sources, I've noticed some interesting details regarding the Plug-in FS API usage.
The extension works with the file system via vscode.workspace.fs API that actually knows nothing about the remote plug-ins running in the dedicated sidecars.

When I run the extension in the Che-Theia container, it's able to get the required files and renders the Notebooks correctly.

But when I run the extension in a separate sidecar, vscode.workspace.fs API still tries to look for the requested resource in Che-Theia container, where the requested resources don't exist of course.

So, the next step here is to adapt vscode.workspace.fs Plug-in API to Che-Theia's remote plugin system.
I've filed the issue for that work - #19110 and updated the sub-tasks.

[azatsarynnyy]

We've finished the adaptation of vscode.workspace.fs Plug-in API to Che-Theia's remote plugin system.
See eclipse-che/che-theia#1045
The next step is to update Python Che Plug-in and to include all the required external dependencies into the image.
cc @ericwill
Adding plugins area

[themr0c]

Release notes: https://issues.redhat.com/browse/CRW-1877